Commit Graph

5031 Commits

Author SHA1 Message Date
y75zhang
4172c3bdd5 bitbake: fetch/wget: checkstatus: drop shared connecton when catch Timeout error
* to avoid wrong http response in checkstatus function:
   in wget checkstatus() we are using 'HTTPConnectionCache' to share connections
   1. state_file1(exists on http server) use shared connection <shared1> to send request
   2. http_server recieved request of state_file1, but delayed by some reason to sent respone
   3. state_file1 checkstatus() failed by timeout and drop shared connection <shared1>
   4. state_file2(not exists on http server) get shared connection <shared1> and send request
   5. http_server finally send 200 response for state_file1
   6. state_file2 recived 200 response and thought it was exists on http_server

(Bitbake rev: bf6d0282ab88b4edc4b9e58184cd76cce965abbd)

Signed-off-by: y75zhang <yang-mark.zhang@nokia-sbell.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-04 22:56:41 +01:00
Peter Marko
742e96ad38 bitbake: fetch/clearcase: remove True option to getVar calls in clearcase module
Layer cleanup similar to
https://git.openembedded.org/openembedded-core/commit/?id=26c74fd10614582e177437608908eb43688ab510

It was probably not found before beacause of the extra "d" parameter.
That seem to be a bug as getVar does not support that.

(Bitbake rev: 720189b810995c5737853458b7eb3779ca0df37e)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-02 22:37:13 +01:00
Richard Purdie
6be592d2a7 bitbake: codeparser: Skip non-local functions for module dependencies
If modules do something like "from glob import glob" then we end up
checksumming the glob code. That leads to bugs as the code can change
between different python versions for example, leading to checksum
instability.

We should ignore functions not from the current file as implemented
by this change.

(Bitbake rev: 1e6f862864539d6f6a0bea3e4479e0dd40ff3091)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-01 13:53:23 +01:00
Richard Purdie
0d49931755 bitbake: codeparser/data: Ensure module function contents changing is accounted for
Currently, if a pylib function changes contents, the taskhash remains
unchanged since we assume the functions have stable output. This is
probably a poor assumption so take the code of the function into account
in the taskhashes. This avoids certain frustrating build failures we've
been seeing in automated testing.

To make this work we have to add an extra entry to the python code parsing
cache so that we can store the hashed function contents for efficiency as
in the python module case, that isn't used as the key to the cache.

The cache version changes since we're adding data to the cache.

(Bitbake rev: b2c3438ebe62793ebabe2c282534893908d520b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-01 13:53:23 +01:00
Robert Yang
ac40cb5ee2 bitbake: cache: Remove invalid symlink for bb_cache.dat
The bb_cache.dat might be an invalid symlink when error happens, then
os.path.exists(symlink) would return False for it, the invalid symlink
wouldn't be removed and os.symlink can't update it any more.

Use os.path.islink(symlink) can fix the problem.

(Bitbake rev: 1387d7b9ee3f270488f89b29f36f9f240e44accc)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-25 11:51:45 +01:00
Martin Jansa
5d88faa0f3 bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes
* avoids long trace when BB_HASHSERVE points to non-existent socket
  file, e.g.:
  BB_HASHSERVE = "unix:///OE/no-socket.sock"
  or when running the build before starting the bin/bitbake-hashserv.

* now it shows just warnings like it did in kirkstone
  many of them, e.g. 6 just for rebuilding zlib-native, but better than long trace

  for nonexistent socket file:
  WARNING: zlib-native-1.3.1-r0 do_create_spdx: Error contacting Hash Equivalence Server unix:///OE/no-socket.sock: [Errno 2] No such file or directory
  for existing file, but before starting bin/bitbake-hashserv:
  WARNING: zlib-native-1.3.1-r0 do_create_spdx: Error contacting Hash Equivalence Server unix:///OE/hashserv.sock: [Errno 111] Connection refused

ERROR: An uncaught exception occurred in runqueue###############################################################                                                                                                               | ETA:  0:00:00
Traceback (most recent call last):
  File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client(addr='unix:///OE/no-socket.sock', username=None, password=None):
             if typ == ADDR_TYPE_UNIX:
    >            c.connect_unix(*a)
             elif typ == ADDR_TYPE_WS:
  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in Client.connect_unix(path='/OE/no-socket.sock'):
             self.loop.run_until_complete(self.client.connect_unix(path))
    >        self.loop.run_until_complete(self.client.connect())

  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in _UnixSelectorEventLoop.run_until_complete(future=<Task finished name='Task-6' coro=<AsyncClient.connect() done, defined at /OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.
py:150> exception=FileNotFoundError(2, 'No such file or directory')>):

    >        return future.result()

  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in AsyncClient.connect():
             if self.socket is None:
    >            self.socket = await self._connect_sock()
                 await self.setup_connection()
  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock:
                     sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
    >                sock.connect(os.path.basename(path))
                 finally:
FileNotFoundError: [Errno 2] No such file or directory

ERROR: Running idle function
Traceback (most recent call last):
  File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client(addr='unix:///OE/no-socket.sock', username=None, password=None):
             if typ == ADDR_TYPE_UNIX:
    >            c.connect_unix(*a)
             elif typ == ADDR_TYPE_WS:
  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in Client.connect_unix(path='/OE/no-socket.sock'):
             self.loop.run_until_complete(self.client.connect_unix(path))
    >        self.loop.run_until_complete(self.client.connect())

  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in _UnixSelectorEventLoop.run_until_complete(future=<Task finished name='Task-6' coro=<AsyncClient.connect() done, defined at /OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.
py:150> exception=FileNotFoundError(2, 'No such file or directory')>):

    >        return future.result()

  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in AsyncClient.connect():
             if self.socket is None:
    >            self.socket = await self._connect_sock()
                 await self.setup_connection()
  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock:
                     sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
    >                sock.connect(os.path.basename(path))
                 finally:
FileNotFoundError: [Errno 2] No such file or directory

Summary: There were 2 ERROR messages, returning a non-zero exit code.

1605616 09:29:05.369352 Parse cache valid
1605616 09:30:14.500863 Registering idle function <function BBCooker.buildTargets.<locals>.buildTargetsIdle at 0x7f43988c09a0>
1605616 09:30:14.500927 Removing idle function <bound method Command.runAsyncCommand of <bb.command.Command object at 0x7f43a961c350>>
1605616 09:30:14.573274 Exception Traceback (most recent call last):
  File "/OE/build/oe-core/bitbake/lib/bb/server/process.py", line 435, in idle_thread_internal
    retval = function(self, data, False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/cooker.py", line 1487, in buildTargetsIdle
    retval = rq.execute_runqueue()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1651, in execute_runqueue
    return self._execute_runqueue()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1567, in _execute_runqueue
    if self.rqdata.prepare() == 0:
       ^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1290, in prepare
    unihashes = bb.parse.siggen.get_unihashes(ready)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/siggen.py", line 713, in get_unihashes
    with self.client() as client:
  File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/siggen.py", line 595, in client
    self._client = hashserv.create_client(self.server, **self.get_hashserv_creds())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 88, in create_client
    raise e
  File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client
    c.connect_unix(*a)
  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in connect_unix
    self.loop.run_until_complete(self.client.connect())
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in connect
    self.socket = await self._connect_sock()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock
    sock.connect(os.path.basename(path))
FileNotFoundError: [Errno 2] No such file or directory
 broke the idle_thread, exiting
1605616 09:30:14.673756 Exiting (socket: True)
1605616 09:30:14.683153 Exiting as we could obtain the lock
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/OE/build/oe-core/bitbake-cookerdaemon.log' mode='a+' encoding='UTF-8'>
sys:1: ResourceWarning: unclosed <socket.socket fd=17, family=1, type=1, proto=0>
ResourceWarning: Enable tracemalloc to get the object allocation traceback

(Bitbake rev: 550c86969e5a137ffef61b08a520a4855232fb1c)

Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-18 08:45:22 +01:00
Rudolf J Streif
d0b02cf801 bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility
Some distributions (namely Fedora Core 40) have started replacing
wget with wget2. There are some changes to wget2 that make it
incompatible with wget:

1. ftp/ftps is not supported anymore
2. progress 'dot' is not yet supported
3. Relative paths in -P and -O are not correctly dealt with

Item 1: Is already dealt with since Scarthgap by only adding the
option --passive-ftp when the URL specifies ftp/sftp. While that
won't help if ftp/sftp is actually required it at least does
not break http/https downloads.

Item 2: While not supported it at least does not break the operation.

Item 3: If there are relative path components in -P or -O then wget2
only deals with them correctly if there is one, and only one, relative
path component at the beginning of the path:

-P ./downloads     works
-P ../downloads    works
-P ../../downloads does not work
-P ./../downloads  does not work
-P /home/user/downloads/../downloads does not work

In cases where there are more than one relative path component at
the beginning of the path and/or one or more reltaive path
component somewhere in the middle or end of the path, wget2 aborts
with the message Internal error: Unexpected relative path: '<path>')

Such can happen if DL_DIR includes relative path components e.g.
DL_DIR = "${TOPDIR}/../../downloads".

This patch canonicalizes DL_DIR before it is passed to wget.

(Bitbake rev: 3e4208952b086adc510e78c1c5f9cf4550d79dc9)

Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-07 13:45:43 +01:00
Richard Purdie
fd250d236a bitbake: runqueue: Avoid save_unitaskhashes
The save comes with an IO overhead which can slow down the rehash loop in bitbake
a lot. We only needed to do this when recipes were doing unihash cache copying. Now
they aren't doing that, drop this IO pain point.

(Bitbake rev: dfc15ef99302dea22a051c9eb8398ffd5cf1fc20)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-07 13:45:43 +01:00
Richard Purdie
6208d986bc bitbake: siggen: Drop copy_unihashes function
The code in OE-Core using this has been replaced with something more fit
for purpose. Drop these function calls as they were never a great idea in the
first place and cause IO slowdown for runqueue needing to sync the cache.

(Bitbake rev: 2c8fa57778c4bd2a5c48a60b701ac57de4289cb2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-07 13:45:43 +01:00
Richard Purdie
b6b66ca07b bitbake: tests/fetch: Tweak test to match upstream repo url change
Upstream changed their urls, update our test to match.

(Bitbake rev: dc391b86540ec5e0a0f1d811c776a22d443b1c06)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-05 18:33:42 +01:00
Joshua Watt
c88bee1a5e bitbake: asyncrpc: Use client timeout for websocket open timeout
The default connection timeout for websockets is 10 seconds, so use the
provided client timeout instead (which defaults to 30 seconds).

(Bitbake rev: 23681775e5941e54ebead469addf708fca1e6beb)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-04 12:05:20 +01:00
Richard Purdie
f24ffc087b bitbake: tests/fetch: Tweak to work on Fedora40
On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.

Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.

(Bitbake rev: 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Richard Purdie
f81127b619 bitbake: fetch2/wget: Fix failure path for files that are empty or don't exist
When we intercepted the file download to a temp file, we broke the
exist/size checks which need to happen before the rename. Correct
the ordering.

For some reason, python 3.12 exposes this problem in the selftests
differently to previous versions.

(Bitbake rev: c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Joshua Watt
5a308474c2 bitbake: siggen: Batch unihash_exists checks
Similar to looking up unihashes, use the batch API when checking if a
unihash exists to speed up lookups

(Bitbake rev: 0ac521ff37b578f7487bca0eccc7dc9e5974991b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Joshua Watt
247d08ae07 bitbake: asyncrpc: Remove ClientPool
Batching support on the client side has proven to be a much more
effective way of dealing with server latency than multiple client
connections and is also much nicer on the server, so drop the client
pool support from asyncrpc and the hash server

(Bitbake rev: 6f80560f1c7010d09fe5448fdde616aef8468102)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Joshua Watt
f618d1dfd7 bitbake: siggen: Drop client pool support
Drops support for client pools, since batching support in the client
code has proven to be much more effective

(Bitbake rev: 85dafaf8e070459f7de7bfb37300d8b60a27002e)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Richard Purdie
2ff7af74bc bitbake: bitbake: Drop older python version compatibility code
cooker: We can call multiprocessing close() unconditionally and tweak a
comment give 3.8 is now the minimum version.

lib/bb: We can drop the logger addition code only needed before 3.6

asyncrpc/hashserv: Since the minimum version is 3.8, we can drop the
conditional code.

(Bitbake rev: 16f4386400f88ba50605307961c248bef09895c1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Richard Purdie
277e07d1cc bitbake: cooker: Improve handling errors during parsing when profiling
We've seeing profiling tracebacks when parse errors occur during
profiling. Try and avoid these but not processing invalid profiles.

(Bitbake rev: 171bd9dd575307fbd61b5179ad86131d76add067)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Joshua Watt
e16d690e77 bitbake: hashserv: server: Add support for SO_REUSEPORT
SO_REUSEPORT is a socket option that allows multiple servers to listen
on the same TCP port, and the kernel will automatically load balance the
connections between them. This is particularly helpful for the hash
server since it runs in a single thread. To take advantage of a
multi-core server, multiple servers can be started in parallel with this
option (up to 1 per CPU) and the kernel will load balance between them.

(Bitbake rev: d72d5a7decb489e2af0ebc43cfea0ca3e4353e9b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-30 07:38:10 +01:00
Joshua Watt
76a63bd031 bitbake: siggen: Enable batching of unihash queries
Uses the batching API of the client to reduce the effect of latency when
making multiple queries to the server

(Bitbake rev: a54734b4ac2ddb3bce004e576cf74d6ad6caf62a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-30 07:38:10 +01:00
Joshua Watt
d31c64296d bitbake: bb: Use namedtuple for Task data
Task dependency data is becoming unwieldy with the number of indices it
contains. Convert it to use a named tuple instead, which allows members
to be indexed by a named property or an index (which allows it to retain
backward compatibility).

(Bitbake rev: 26446cca4d22734c3f1b328a205c169dadb7e494)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-28 23:46:21 +01:00
Richard Purdie
e9400f091c bitbake: runqueue: Improve rehash get_unihash parallelism
Improve the rehash code to query unihashes in parallel since this is more
efficient on slower links.

(Bitbake rev: c1949d5350342eaaf6ab988d7bfba99496d55523)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-28 23:46:21 +01:00
Richard Purdie
6a0a2c4618 bitbake: runqueue: Process unihashes in parallel at init
Improve the runqueue init code to call unihash queries in parallel since
this is faster and more efficient, particularly on slower links with longer
round trip times.

The call to the function from cooker is unneeded since that function calls
prepare() and hence this functionality will already have run, so drop
that obsolete call.

(Bitbake rev: 721c97a115a7a4bf21955be79391bd6e0099f40e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-28 23:46:21 +01:00
Richard Purdie
d89b436835 bitbake: runqueue: Allow rehash loop to exit in case of interrupts
The initial hash serve loop exits in the case where interrupts are present
but probably checks a bit too often. Tweak that and also allow the slow
rehash loop to break on interrupt, improving bitbake Ctrl+C response.

(Bitbake rev: 4534365591fd17bcc2b684900863b67bc69519ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-28 23:46:21 +01:00
Richard Purdie
2824b5e667 bitbake: runqueue: Add timing warnings around slow loops
With hashserve enabled, there are two slow paths/loops, one at initial runqueue
generation and also during the rehash process when new outhashes are found.

Add timing information at the hashserve log level for when these loops
take longer than 30s or 60s overall. This will leave evidence in the logs when
things are running particularly slowly.

(Bitbake rev: 6c357ede08e0b2a93bdaad2c1d631994faf2b784)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-28 23:46:21 +01:00
Marlon Rodriguez Garcia
1d86845c41 bitbake: ui/buildinfohelper: Add exception treatment to fix missing target_file
Based on the discution on  https://lists.yoctoproject.org/g/toaster/message/6157
in some cases the value for Target_file could be missing and is needed to bypass
it to finish build.

(Bitbake rev: c60f6d20911632b41473f7c8577949be2f99ad80)

Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-28 09:43:51 +01:00
Richard Purdie
7812f104db bitbake: fetch/npmsw: The fetcher shouldn't have any knowledge of S
I don't know why there is hardcoded knowledge of S in the fetcher but there
shouldn't be and the OE unpack changes highlight this doing things it
shouldn't.

Drop the S reference and use rootdir which is the only place it should
be touching.

(Bitbake rev: 84f102954e10a3390fca9c26d5c3c639e952a2c9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-23 11:27:08 +01:00
Antonin Godard
1aa8276c64 bitbake: tests.codeparser: add tests for shell expansions
Tests quotes around `` and $() expansions, nested and multiple
expansions, and that escaped quotes are treated as characters by the
parser.

(Bitbake rev: d98130cb4d500c495bc692c56dde3e019f36320a)

Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-21 14:14:41 +01:00
Antonin Godard
dd98d156ca bitbake: codeparser: remove redundant list conversion
(Bitbake rev: 89712949de9476e4674864a8dcd6862fefe92eae)

Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-21 14:14:41 +01:00
Antonin Godard
03742d7cb3 bitbake: codeparser: support shell substitutions in quotes
The current shell substitution mechanism only works without quotes. For
example:

  var1=$(cmd1 ...)

Will work and add `cmd1` to the correspondind `run.do_*` file.

However, although quite common, this syntax is not supported:

  var1="$(cmd1 ...)"

This commit adds this feature by adding a step to process_words() to
check whether we are dealing with quotes first, and by iterating on
what's between them to detect new shell substitution candidates. These
candidates are tested and parsed like before in the next step. The
original `part` being part of the candidates means the syntax
var1=$(cmd1 ...) is still valid.

(Bitbake rev: f56e1a37b2ba1773ed308043d7eb073cc2e6c06e)

Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-21 14:14:41 +01:00
Richard Purdie
4758e1c43a bitbake: cooker: Ensure generateTaskDepTreeData fails for NoProvider
If an invalid provider is requested, error out early rather than trying
to build partial runqueue data structures as the taskdep UI will have
exited after seeing the bad provider.

(Bitbake rev: a478087998cb794cc4e31189b3ce07973d3949bc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-17 11:56:40 +01:00
Richard Purdie
92d8c7c553 bitbake: parse: Improve/fix cache invalidation via mtime
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.

We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.

This hopefully fixes [YOCTO #15318].

(Bitbake rev: d9e5d313c79500e3c70ab9c3239b6b2180194f67)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-16 23:25:55 +01:00
Richard Purdie
229951e1da bitbake: asyncrpc/client: Fix websockets minimum version for python 3.10
python 3.10 support is only available in websockets 10.0 and later:

08d8011132

Update the version for this case. This avoids failures on Ubuntu 22.04.

(Bitbake rev: 0e4767c4a880408750e1a6855270c5a4eef8383d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-16 10:52:43 +01:00
Richard Purdie
f4885e97a6 bitbake: bitbake: update to version 2.9.1
This allow the use of new siggen API

(Bitbake rev: e53503546990adeab67b6d044fcce59dc5a3f455)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:53:01 +01:00
joshua Watt
23c5058707 bitbake: asyncrpc: Check websockets version
Checks that the minimum version of the websockets module is present, and
if not raises an ImportError. This allows the user to get earlier
feedback if using websockets is going to succeed

(Bitbake rev: 330ea6914aad65dc8b34c986c44779820c392f03)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:44 +01:00
joshua Watt
aff8b07334 bitbake: cooker: Handle ImportError for websockets
Handles ImportError when creating a hash equivalence to ping the server.
This notifies user earlier with a more precise error if websockets can't
be used, and also prevents passing a known bad upstream value to the
local server

(Bitbake rev: aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:44 +01:00
joshua Watt
8364aa5baa bitbake: siggen/runqueue: Report which dependencies affect the taskhash
Report which task dependencies in BB_TASKDEPDATA are included in the
taskhash. This allows tasks to identify which tasks dependencies may
change without the task re-running. Knowing this information is
important for tasks that want to transfer information from dependencies
(such as SPDX)

(Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:43 +01:00
Kari Sivonen
9925db0c4f bitbake: fetch2/svn: Fix mirroring issue with svn
Add return false to supports_checksum for svn fetcher which fhis
fixes MIRROR usage for svn uris. Also add a testcase.

[YOCTO #15473]

(Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f)

Signed-off-by: Kari Sivonen <kari.sivonen@live.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:43 +01:00
Richard Purdie
ead0ff2210 bitbake: build: Handle conflict between cwd and cleandirs
If the cwd of the task is also a cleandirs, you would see warnings from bitbake
about being unable to obtain cwd during the task execution. Tweak the code
to detect this and avoid the warnings.

(Bitbake rev: 6c7fd60c10955b0f23f64b25b5b4e154eb22a8f8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:43 +01:00
Alexander Kanavin
551fdabc54 bitbake: fetch2/crate: add upstream latest version check function
This is actually rather easy: crate web API provides a json
with all the versions, for example:
https://crates.io/api/v1/crates/cargo-c/versions

(Bitbake rev: f6c2755db9a1f88c8534193b420fa31d135945e6)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:43 +01:00
joshua Watt
e4ddff1399 bitbake: cooker: Use hash client to ping upstream server
The cooker attempts to connect to the upstream hash equivalent server to
warn the user early if it is misconfigured. However, this was making the
assumption that it was a raw TCP connection and failed when attempting
to use a websocket upstream server. Fix this by creating an hash client
and using the ping API to check the server instead of using a raw
socket.

(Bitbake rev: 5e84c13a6c594ed34c341849806657ddda206714)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-02 16:07:21 +01:00
Sven Schwermer
25dcc55b74 bitbake: fetch2/gcp: Add missing runfetchcmd import
This adds the missing import. This bug was introduced with 1ab1d36c.

(Bitbake rev: 97ffe14311407f6e705ec24b70870ab32f0637b9)

Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-20 07:30:29 +01:00
Richard Purdie
ae3bca8492 bitbake: bitbake: Bump to version 2.9.0 development version postrelease
(Bitbake rev: 67a1aa8dbb3cb3a30fa7d697431ebb30323e4f28)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16 08:12:27 +01:00
Richard Purdie
3bbe0a45b4 bitbake: bitbake: Bump to version 2.8.0
(Bitbake rev: c86466d51e8ff14e57a734c1eec5bb651fdc73ef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16 07:38:22 +01:00
Joshua Watt
2ecd97fa59 bitbake: siggen: Capture SSL environment for hashserver
Now that the bitbake hash server supports SSL connections, we need to
capture a few environment variables which can affect the ability to
connect via SSL. Note that the variables are only put in place to affect
the environment while actually invoking the server

[RP: Tweak to use BB_ORIGENV as well]
[RP: Tweak to handle os.environ restore correctly]
(Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16 07:33:19 +01:00
Richard Purdie
6bd8367aa9 bitbake: BBHandler: Handle unclosed functions correctly
A function accidentally defined as:

somefunction() {
	:
 }

which is unclosed due to the space at the end, would currently silently
cause breakage. Have the parser throw and error for this.

[YOCTO #15470]

(Bitbake rev: a7dce72da6be626734486808f1b731247697e638)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
d133dc7e77 bitbake: asyncrpc: include parse_address from hashserv
Moving the code and related definitions from
hashserv/__init__.py to asyncrpc/client.py,
allowing this function to be used in other asyncrpc clients.

(Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Rob Woolley
c0cd7a6d3b bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftps
Fedora 40 introduces wget2 as a drop-in replacement for wget.  This
rewrite does not currently have support for FTP.  This causes
the wget fetcher to fail complaining about an unrecognized option.

Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem.  It also gives us
an opportunity to build the older wget as a host tool.

(Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a)

Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-11 08:47:07 +01:00
Joshua Watt
978206fed4 bitbake: siggen: Add support for hashserve credentials
Adds support for hashserver credentials to be specified in the
SignatureGenerator

(Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-23 10:19:00 +00:00
Derek Erdmann
825055e83e bitbake: fetch2/git: Install Git LFS in local repository config
Git uses a lock file to prevent concurrent modifications to the global
config, so if unpack tasks for different recipes try to run "git lfs
install" simultaneously the operation can fail:

    error: could not lock config file /home/build/.gitconfig: File exists exit status 255
    Run `git lfs install --force` to reset Git configuration.

Adding "--local" sets the smudge and clean filters in the local
repository's config instead of modifying the user's global config.

(Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7)

Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:46 +00:00