Commit Graph

6230 Commits

Author SHA1 Message Date
Richard Purdie
c08e2c2e20 bitbake: utils: Use internal fetcher function to avoid duplication
We currently have two lists of "proxy" or "fetcher" environment exports.
Make the one in utils match the one on the fetcher which has a more complete
list of variables now.

(Bitbake rev: f9165a798a307a6f0fee120d5c3de660d3a44ae8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Richard Purdie
9345140f09 bitbake: fetch2/wget: Drop unused import
This import is no longer used anywhere so can be removed.

(Bitbake rev: 956128e394581855bf0d03b32a975dc91c2a7e0c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Caner Altinbasak
24e29f2d08 bitbake: fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARS
These environment variables are needed by gclient and needed to be
passed into fetcher.

(Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Richard Purdie
584a913d75 bitbake: cooker: Fix memory resident cache invalidation issue
We've been seeing weird PRServ failures on the autobuilder. These had
one failure always followed by a second. Whilst I can't reproduce the first,
if I made that test fail, I could reproduce the second with memory resident
bitbake. This was with the tests:

prservice.BitbakePrTests.test_import_export_replace_db
and then
prservice.BitbakePrTests.test_pr_service_deb_arch_dep

which was giving a strange looking error:

NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm)
NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa)
ERROR: No such task: do_package_write_rpm
ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1'

where the issue is that selftest.inc written by the test framework
and containing PACKAGE_CLASSES = "package_deb" was being ignored.

The issue is the cached_statements{} within BBHandler() is not being
invalidated at the right time.

This patch changes the code to ensure base configuration is not parsed
until inotify updates have been processed.

(Bitbake rev: cada37c6b9e5862ca2c5a54ad6fd1e1f1939cd9c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24 15:38:24 +00:00
Ross Burton
9b0277c3c3 bitbake: lib/bb/siggen: fix debug() call
Bitbake f68682 changed the logger's debug() method to be compatible with
logging.debug(), but this caller was still using the old API where you
can pass an integer as the first argument:

WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]')

Instead, call bbdebug() which has the priority argument.

(Bitbake rev: 18d4f9e8387f7994cf6d46300e25dda1c3a593b2)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24 15:38:24 +00:00
Richard Purdie
77be687a37 bitbake: bitbake: Bump to version 2.3.1
So that OE-Core can depend on bb.event.check_for_interrupts(), bump our
verison number to a development series version.

(Bitbake rev: dea1b2f3fc31f28daed5da16e62da8895d6e5716)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23 12:12:18 +00:00
Richard Purdie
42c92cbc83 bitbake: event/cooker/runqueue: Add ability to interrupt longer running code
Bitbake is now able to understand when a UI wants it to stop the current
processing. There are some long running functions which currently have no
mechanism to interrupt them however.

This patch adds a call, bb.event.check_for_interrupts(d) which can be
placed in long running code and allows an internal state flag within
bitbake to be checked. If set, that flag will trigger an exit.

This means that Ctrl+C can be made to work in a variety of places where
it currently would not.

Long running event handlers in OE-Core can also then benefit from this
new approach with the addition of the function call as well.

(Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23 12:12:18 +00:00
Paulo Neves
97b258985a bitbake: fetch/git: Replace mkdtemp with TemporaryDirectory and avoid exception masking
Due to using mkdtemp instead of TemporaryDirectory we needed to
manually cleanup the directory in a try finally block.
With tempfile.TemporaryDirectory we can handle the cleanup with
a "with" statement and not need to manually clean up oursevels.

(Bitbake rev: 36a231d2a45e7c87172196538d18ded81a234774)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Paulo Neves
d085dcb449 bitbake: fetch/git: Removed unused variables in _contains_lfs
branchname was set but never used in the context of _contains_lfs
method.

(Bitbake rev: a64ec64c2faf1ac6ce995cdd4e311d8be6046155)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Paulo Neves
ffcc20770d bitbake: tests/fetch: Add real git lfs tests and decorator
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.

The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.

lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.

lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.

lfs=0 actually disables git-lfs functionality even if supported.

(Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Paulo Neves
41c86ad30e bitbake: tests/fetch: git-lfs restore _find_git_lfs
Not restoring the mocked _find_git_lfs leads to other tests
failing.

(Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Richard Purdie
76db0baba2 bitbake: server/process: Improve idle thread exception handling
If the inotifier code has an exception, bitbake currently hangs. Catch any
exception and exit if seen. Also check the idle thread is alive and exit
if it disappears. This should stop bitbake hanging if such a situation arises
in future such as this example:

3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
    self.cooker.process_inotify_updates()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
    n.read_events()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
    if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)

(Bitbake rev: 358b5b02d5de1ab0f98104c4ec4953e46999b9a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-20 15:18:53 +00:00
Richard Purdie
7244bf2457 bitbake: cooker: Ensure lock is held with changing notifier
We've seen a couple of cases which bitbake hangs due to an inotifer exception
such as:

3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
    self.cooker.process_inotify_updates()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
    n.read_events()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
    if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)

Ensure we don't destory the inotifier when the idle thread is reading is
by holding the lock during setup/teardown.

(Bitbake rev: 8fc5c50c2e23017833f93bcd514d708a14fa4266)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-20 15:18:53 +00:00
Richard Purdie
a7e3c0f046 bitbake: runqueue: Drop SystemExit usage
Using bb.fatal for a fatal error message is the best practise, switch
the code to match other call sites.

(Bitbake rev: c27e48fa81c2327a4a355a028884ab457cde3ae7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-19 07:48:28 +00:00
Richard Purdie
65de367cfe bitbake: cookerdata: Drop dubious exception handling code
This code appears to be dangerous, it swallows exceptions, turning them into
"handled" versions which then show no errors to the user. This is a pretty
poor user experience and I can't see why this code should be swallowing
such things. Drop the worst bits of code.

(Bitbake rev: 13279044f16f2cf2502ebf39d277415f99bb6c18)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-19 07:48:28 +00:00
Richard Purdie
42f4f740b1 bitbake: cookerdata: Improve early exception handling
Martin Jansa reported that if you put a syntax error into an imported
module such as qa.py in OE, no error is shown.

Part of the issue appears to be that the catch_parse_error() decorator only
catches certain exceptions and SyntaxError isn't one of them. As far as I can
tell we should remove all the special cases and use the more advanced code
in all cases, not just expansion errors.

I confirmed this now prints a proper error message for a qa.py syntax error.

(Bitbake rev: 2365d891847f8e73d1c4661ddfdab8818ff619dc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-19 07:48:28 +00:00
Pedro Baptista
445caa139c bitbake: action.py: add topdir in bblayers_conf path for remove-layer
By including the full path to bblayers.conf the remove-layer
command can be executed from any location, not only from the
build directory.

(Bitbake rev: 25cb4e17e8a4c0310df018b9df184b0155b267eb)

Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-17 18:02:04 +00:00
Pedro Baptista
5630cb5ccd bitbake: action.py: add topdir in bblayers_conf path for add-layer
By including the full path to bblayers.conf the add-layer
command can be executed from any location, not only from the
build directory.

(Bitbake rev: 77aa230bf2a053fc0941723a6abbc798ebe53a19)

Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-17 18:02:04 +00:00
Ross Burton
77f91746dc bitbake: fetch2/wget: clean up netrc usage
Assigning a return value which is potentially None to a tuple and
catching TypeError is pretty ugly.  Rewrite the code to explicitly check
the value for clarity.

(Bitbake rev: f4ebb27616ac2df27c29a6052b1526a4c48db607)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-17 18:02:04 +00:00
Richard Purdie
d976eb3d52 bitbake: data: Evaluate the value of export/unexport/network flags
Currently the export/unexport/network flags are acted on only by presence
or lack of in the data store. This is deliberate due to performance reasons
since a given recipe shell task might export ~80-90 variables and this means
expanding flags for every one of them.

This does catch users unaware since values which expand to nothing don't work
e.g. ${@""} and setting values to "0" wouldn't work either.

The downside to this patch is slow down in parsing speed of around 4%.

The network flag is easier to change and doesn't affect performance, it was
made to operate the same as export/unexport for consitency reasons.

(Bitbake rev: 3d96c07f9fd4ba1a84826811d14bb4e98ad58846)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-17 18:02:04 +00:00
Richard Purdie
ca4721c64a bitbake: cookerdata: Remove incorrect SystemExit usage
Calling SystemExit doesn't work well with server/client usage since the string
isn't printed to the right place. Use bb.fatal() instead which prints the right
log output and raises and handled exception which then shows correctly on the
UI.

(Bitbake rev: aaefb43b41a0d9b16a59643136268eb6e5d48cd2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-17 15:10:02 +00:00
George Kelly
aa6cd06a9f bitbake: fetch2: Add NODE_EXTRA_CA_CERTS to export list
If you are behind a corporate proxy, the npm fetcher uses
the proxy IP address already passed in the list of exports.
However, it will error if the proxy injects its own root
CA certificate. Thus, the NODE_EXTRA_CA_CERTS environment
variable must be passed so the user can include their
company's root CA as a trusted CA inside node's
certificate store.

(Bitbake rev: 5daecc25a59ce5bf6c31f5085e6caa7cb543517e)

Signed-off-by: George Kelly <george.kelly1097@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-09 10:00:31 +00:00
Etienne Cordonnier
edb60ef6fd bitbake: siggen: Fix inefficient string concatenation
As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython
has an optimization for statements in the form "a = a + b" or "a += b".
It seems that this line does not get optimized, because it has a form a = a + b + c:
data = data + "./" + f.split("/./")[1]

For that reason, it does a copy of data for each iteration, potentially copying megabytes
of data for each iteration.

Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds
instead of 45 seconds on my test setup where SRC_URI points to a big directory.

Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython:
"do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b"

However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup:
data = ''.join((data, "./", f.split("/./")[1]))

I have changed the other lines to also use += for consistency only, however those were in the form a = a + b
and were optimized already.

Co-authored-by: JJ Robertson <jrobertson@snap.com>
(Bitbake rev: 195750f2ca355e29d51219c58ecb2c1d83692717)

Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 17:03:15 +00:00
Pedro Baptista
940cee8422 bitbake: bblayers/query: Adjust show-layers output layout
Increase the `ljust` value for each column in show-layers
output. This improves readability  when long layer paths are
included

(Bitbake rev: a56cd53e0e67a368acace9dab184bd94b9b67b1b)

Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 17:03:15 +00:00
Pedro Baptista
9724df2861 bitbake: bblayers/query: Replace layer directory name with layer name for show-layers
Replace the layer directory name with the name from BBFILE_COLLECTIONS
which is useful when assigning a LAYERDEPENDS value

(Bitbake rev: 517cc3f5aee2a7d6362ce0202d4ee709a5378096)

Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 17:03:15 +00:00
Ross Burton
b1ec98f71f bitbake: bb/utils: include SSL certificate paths in export_proxies
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).

Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.

In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.

Also remove the return value as nothing ever checked it.

[ YOCTO #15000 ]

(Bitbake rev: c19035e8e71c419c5688a86bfc9c946c96f638e8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-26 21:50:31 +00:00
Richard Purdie
557aab2158 bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache location
Currently the codeparser cache is set from CACHE, which is typically in
bitbake.conf which means we can't read/write any cache until it is found/read.
We may well have python expressions to parse before that happens.
The net result is suboptimal functioning of the codeparser cache since it will
often be invalidated by data that is never written.

This patch changes the codeparser and filechecksum caches to use BB_CACHE as
their setting and defaults it to ${TOPDIR}/cache.

The patch doesn't change where the "persistent" data such as prserver and
hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing
cache resists (still currently CACHE). I've left those for a later patch.

The patch does ensure data parsed by the core datastore parsing calls is
written back since this is now much more useful after this change.

(Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-26 21:50:31 +00:00
Tim Orling
95f4795660 bitbake: toaster: update fixtures
* Update to latest langdale 4.1.2 and kirkstone 4.0.6
* Re-instate dunfell and update to 3.1.22
  - drop comments about bitbake crash

(Bitbake rev: f90c45df5cf4640e0714fde13e311db3327f9fd4)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-26 21:50:31 +00:00
Tim Orling
4d1675beac bitbake: bitbake: fix deprecated threading.Thread.setDaemon
Deprecated in Python 3.10:
https://docs.python.org/3/whatsnew/3.10.html#deprecated
https://github.com/python/cpython/pull/25174

Fixes warnings like:

...bitbake/lib/bb/ui/uievent.py:68: DeprecationWarning: setDaemon() is
deprecated, set the daemon attribute instead
  self.t.setDaemon(True)

(Bitbake rev: 323f6ce27a1bfd7159e72f29684674ff495dedee)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Richard Purdie
ea12b673cf bitbake: data_smart: Small optimisation to _findVar()
Some users of _findVar don't need the override data and even
getVarFlag doesn't need it in some common cases (parsing=True).

Rearrange the code as the current overridedata approach doesn't need
to be in the _findVar code anyway. This removes some search overhead
from a critical path.

(Bitbake rev: fcb64e1138a20eb19560af3fc5d1fa748cc9cf34)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Richard Purdie
663c259ffa bitbake: data: Add support for new BB_HASH_CODEPARSER_VALS for cache optimisation
Currently the codeparser cache ends up being extended for every parse run
since there are values in the functions such as the result of os.getpid()
from LOGFIFO in OE-Core.

Digging into that issue, there are also lots of similar but different
functions being parsed where the change might just be a path to WORKDIR,
a change in PN or PV or something like DATE/TIME.

There is no reason we have to use these changing values when computing the
dependenies of the functions. Even with a small tweak like:

BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn"

the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run
to around 1.3MB and remaining static for oe-core and meta-oe. In my local
build, admittedly heavily experimented with, the cache had grown to 120MB.

The benefits of doing this are:

 * faster load time for bitbake since the cache is smaller to read from disk
   and load into memory
 * being able to skip saving the cache upon shutdown
 * lower memory footprint for bitbake
 * faster codeparser data lookups (since there is less data to search)

We only use these special values when passing code fragments to the codeparser
to parse so the real variable values should otherwise be used in the hash data.

The overall effect of this change, combined with others to avoid saving unchanged
cache files can be ~2s on a ~16s parse on my local system and results in a more
responsive feeling bitbake. It also allows parsing performance to be investigated
more consistently.

(Bitbake rev: f24bbaaddb36f479a59a958e7fc90ef454c19473)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Richard Purdie
72072db004 bitbake: cooker: Fix parsing race around cache handling
When draining the result queue from the parsing processes, cache objects
can be created even if they are then immediately destroyed. The reset
in the sync code needs to happen after any objects have been created.

Change the ordering to fix this. This ordering has caused various
cache errors, particularly when interrupting parsing with Ctrl+C.

(Bitbake rev: f45a94e6720dacf7f51ac147c115a6f608769093)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Richard Purdie
4a94cf21c5 bitbake: cache: Only write files if we have data
By writing the cache files only if there is data to write, we can save a
bit of time.

(Bitbake rev: abeff1f80bb1c690b92d535d472dff9df7a56067)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Richard Purdie
ae1da8b75c bitbake: cooker: Fix siggen recipe cache race issue
We need to reset the cache before the start() call which starts the
parsing processs, tweak the code to ensure this is the case.

(Bitbake rev: 4712d6237df5d2188238294fbdb9f107b068a3a7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Richard Purdie
18d2c489f0 bitbake: server/process: Fix lockfile contents check bug
We need to check against the first line of the file, fix the typo.

(Bitbake rev: 4abc598fb01d426394f4222dfc752e620a8e1b7b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Pavel Zhukov
f962c586fc bitbake: fetch2/tests: Add parameter to recipe_uri
While the parameter is not required it allows testing of possible
regression in fetcher code when parameter specified and mirrors are
used.

(Bitbake rev: 1a21918049091a6d77426dbf8868ffdc14ba1003)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-20 17:49:26 +00:00
Pavel Zhukov
2ac1235b9b bitbake: fetch2/tests: Add test for Mercurial
Basic test for mercurial functionality including PREMIRRORONLY with
disabled network

(Bitbake rev: c0e349a5fc92e9c90afc53800c7e83495393ff79)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-18 16:42:27 +00:00
Pavel Zhukov
e82805663f bitbake: fetch2: kill parameters in uri_find_decoded
Fixes:
  File "/bitbake/./lib/bb/fetch2/__init__.py", line 446, in uri_replace
    if uri_decoded[loc][k] != uri_find_decoded[loc][k]:
KeyError: 'module'

(Bitbake rev: 80421abd5bf3f1f751219c050fa93970cf1f2088)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-18 16:42:27 +00:00
Richard Purdie
11698e027d bitbake: server/process: Improve lockfile handling at exit
If memory resident bitbake is active and the build directory is renamed upon
build completion, several bad things can happen:

* the old build directory could be re-created to contain a lockfile
  leaving an empty directory behind
* a lockfile for a new build could be found and attempt to be locked

This patch avoids creating an empty directory (not perfectly, but should
work in the majority of cases - an empty directory is cosmetic).

It also now compares the lock file contents to it's own pid and
just exits if it doesn't match, it is clearly then belonging to some
new process.

This will be combined with bitbake shutdown calls on the autobuilder to
ensure "saved" build directories, or build directories being deleted by
clobberdir don't do strange things.

(Bitbake rev: b986eac18b6a8bf633f5ef15f32f68de4c86173b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-14 14:33:00 +00:00
Yoann Congal
e1a661e3d2 bitbake: persist_data: Handle sqlite error when cachefile path is too long
Sqlite can't open the cachefile when its path is too long (>= 505 char by
my tests).

We do have a path length check in sanity.bbclass but this code is called
before sanity checks.

Treat the error in the exception instead of checking before hand in case
sqlite eventually fixes this.

Fixes [YOCTO #12374]

(Bitbake rev: bf681d173263cd42ffc143655f61abe0573fd83c)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-13 17:34:08 +00:00
Richard Purdie
e26e0b92e8 bitbake: server/process: Move heartbeat to idle thread
Rather than risk the heartbeat event code locking up the server control
socket, handle it in the 'idle' thread with the other work. The aim
is to remove it as a possible issue with some ongoing hangs.

(Bitbake rev: 0f9a0c7853b181817bf01863a26da21412376294)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 23:16:31 +00:00
Yoann Congal
29cfebf4e0 bitbake: main: Move --buildfile help at the end of "Execution" group
Often, '--buildfile/-b' is used to mean "Do not execute anything but
this recipe", so it make sense to have it in the "Execution control
options" group.

Fixes [YOCTO #12018].

(Original suggestion by Ross Burton)

(Bitbake rev: 412a6d8701a92efe4dd6db8ec713c0013f7db64d)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 23:14:48 +00:00
Richard Purdie
51f7321141 bitbake: cooker: Fix exit handling issues
The shutdown() call should definitely not be waiting for idle, since we
expect execution and events to continue even after setting either shutdown
state. This was causing the UI to appear to hang at Ctrl+C interrupts.

Also ensure that if we reset cooker, we stop any active parser processes
since these currently appear to be being left behind.

(Bitbake rev: 4f73c2eb12ee074f3b7d4717380649f6ca8f3def)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 23:14:48 +00:00
Richard Purdie
3a439f94ab bitbake: knotty: Improve shutdown handling
There are three levels of shutdown, the initial "wait for current tasks",
then "stop current tasks" and "exit now". Change the code so that we don't
instantly exit after "stop current tasks" but allow the events to come
through from the server first. The new shutdown level allows that to then
be broken out of too.

(Bitbake rev: af38345b91cfc8a6bb5c38f5753255e7635bfe2b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 23:14:48 +00:00
Richard Purdie
92c0dba456 bitbake: tinfoil: Don't wait for events indefinitely
If for some reason the bitbake server dies, we shouldn't loop indefinitely
waiting for events within tinfoil. Add a ping test and exit if things
have somehow failed.

(Bitbake rev: 32da768ae67cfe0b2d154ce4667a7cfdfa5bf4b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 23:14:48 +00:00
Richard Purdie
165e8b563d bitbake: process/cooker/command: Fix currentAsyncCommand locking/races
currentAsyncCommand currently doesn't have any locking and we have
a conflict in "idle" conditions since the idle functions count needs
to be zero *and* there needs to be no active command.

Move the changes/checks of currentAsyncCommand to within the lock
and then we can add it to the condition for idle, simplifying some
of the code.

(Bitbake rev: b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 10:58:36 +00:00
Richard Purdie
4e4f040a73 bitbake: cooker/command: Drop async command handler indirection via cooker
Indirecting the async command handler via cooker is confusing and no
longer needed. Drop it to make things slightly clearer.

(Bitbake rev: 4a41a7d0594e6a84a67b9de70a505858aebcd84a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 10:58:36 +00:00
Yoann Congal
654380885a bitbake: Group and reorder options in bitbake help
Fixes [YOCTO #12018].

Also, I have included some small fixes:
* added a '.' at the end of sentence where it was missing
* split some long lines

(Bitbake rev: 9257c48c3e36daaecb5e15da22d0bed24865f02c)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-09 14:21:01 +00:00
Richard Purdie
7f556c6861 bitbake: cooker: Clean up inotify idle handler
We no longer need to abstract the inotify callback handler, remove the
abstraction and simplify/clean up the code.

(Bitbake rev: af4ccab8acc49e91bf7647f209d69f4858618466)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-06 17:40:01 +00:00
Richard Purdie
9043034da5 bitbake: cooker: Rework the parsing results submission
The loop submitting the parser results was not efficient on many core systems
as may of the parsers could block for a long time waiting to send the results.
While a result was pending, the code wouldn't parse further jobs.

Change the code to parse further jobs even if the results can't be submitted
and lower the result submission timeout to keep the parsers busy.

(Bitbake rev: 9ece4a2e406509bd89dbce8dac80a02e600b0ecf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-06 17:40:01 +00:00