We've been moving to try and separate several pieces of runqueue. Allow
start_worker to operate separately to rqexe since they don't need to be
tied. This allows rqexe to be available to print_diff for future
improvements.
(Bitbake rev: 834e452243ff2eea6e8e2e7f4935b5233ffb4b00)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added a new button on the base template to access a new template.
Added a model register the information on the builds and generate access links
Added a form to include the option to load specific files
Added jquery and ajax functions to block screen and redirect to build page when import eventlogs is trigger
Added a new button on landing page linked to import build page, and set min-height of buttons in landing page for uniformity
Removed test assertion to check command line build in content, because new button contains text
Updated toaster_eventreplay to use library
Fix test in test_layerdetails_page
Rebased from master
This feature uses the value from the variable BB_DEFAULT_EVENTLOG to read the files created by bitbake
Exclude listing of files that don't contain the allvariables definitions used to replay builds
This part of the feature should be revisited. Over a long period of time, the BB_DEFAULT_EVENTLOG
will exponentially increase the size of the log file and cause bottlenecks when importing.
(Bitbake rev: ab96cafe03d8bab33c1de09602cc62bd6974f157)
Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases ${ may not be expanded in the WORKDIR (one of the cases is
undefined variable) and causes cryptic failures [1]. Guard this by
erroring out if directory name contains ${
Fixes: [Yocto #15255]
[1]
ERROR: x-native-1.0+${SRCPV}-r0 do_deploy_source_date_epoch: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_hardcode_path(d)
0003:
File: '/home/mischief/src/poky/meta/classes-global/sstate.bbclass', lineno: 654, function: sstate_hardcode_path
0650: bb.note("Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd))
0651: subprocess.check_output(sstate_hardcode_cmd, shell=True, cwd=sstate_builddir)
0652:
0653: # If the fixmefn is empty, remove it..
*** 0654: if os.stat(fixmefn).st_size == 0:
0655: os.remove(fixmefn)
0656: else:
0657: bb.note("Replacing absolute paths in fixmepath file: '%s'" % (sstate_filelist_relative_cmd))
0658: subprocess.check_output(sstate_filelist_relative_cmd, shell=True)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/mischief/src/poky/build/tmp/work/core2-64-poky-linux/x-native/1.0+${SRCPV}-r0/sstate-build-deploy_source_date_epoch/fixmepath'
(Bitbake rev: e91c256ec076e70cf8a18e369fe7862e50618c48)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way the code currently handles dependencies between setscene tasks is fairly
poor, basically by deleting chunks of dependencies and adding reversed dependency
relationships.
This was once the best way to handle things but now a lot of the surrounding code
has changed and this approach is suboptimal and can be improved.
This change firstly adds debug logging for "hard" setscene task dependencies since
previously the codepaths were missing from logs making them very hard to read.
The changes to the setscene dependency graph are removed entirely this these altered
graphs were a significant source of problems. Instead, if a hard dependency is run
into, we mark the hard dependency as buildable and defer the task until the hard
dependencies are met.
The code now also skips the check_dependencies() code for hard dependencies since
previously that code was having to list all possible hard dependencies. We don't
need to do that as we can safely assume hard dependencies are required.
With these changes to runqueue's behaviour, we stand some chance of being able to
fix other bugs in OE-Core related to useradd for example.
(Bitbake rev: 367789b53c1c22ec26e0f4836cdf2bdd9c7d84fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The GCP fetcher was calling bb.fetch2.check_network_access with
"gsutil stat" as the command, but then never actually ran that
command to check if the file exists. In cases where the file did
not exist in a gs:// premirror, this would lead to an unhandled
exception from do_fetch when the GCP python API tried to perform
the download.
This change resolves that issue by adding a runfetchcmd to call
gsutil.
(Bitbake rev: 1ab1d36c0af6fc58a974106b61ff4d37da6cb229)
Signed-off-by: Charlie Johnston <charlie.johnston@loftorbital.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Found a duplicate test, added _2 suffix to one, 74 tests now pass up from 73.
(Bitbake rev: ae2a19dadb4f3065b8731a61f45f29e6a70af402)
Signed-off-by: Richard Haar <rh@richhaar.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the event log is enabled and parsing the metadata triggers log messages,
the event code and deadlock. Iterating the variables inside the event handling
code causes this. SOURCE_DATE_EPOCH triggers a python function which calls
bb.debug() and can trigger a lockup as one example.
Move the code around and add it to the BuildStarted events explictly. This
does mean runs without builds no longer get variables added to the eventlog
however we can look into a more targetted version of data if/as/where neded.
(Bitbake rev: 4135a617ae16d509362b5bf56378139cdc0876d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ncurses UI wasn't working due to a missing function call. Add it
to avoid a traceback when starting builds.
(Bitbake rev: db8f36b69a68de2179e5685cf24a42ec10d68257)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Docstring for git fetcher is missing destsuffix and subpath parameters,
so add them
(Bitbake rev: 7cea6c2efefbb0f789f654c04a61d9ba770c148b)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When downloading a Git repository containing an LFS, the Git hooks
are not always initialized correctly to perform the download. This
change updates the Git downloader to run the "git lfs install" command
in order to "smudge" the LFS files. The tests have also been updated
to do checks for situations in which git lfs is not installed, as
the application was required to be installed for any of the tests
to run previously.
The Git LFS functionality was working to some degree previously,
but this change also updates the fetcher to allow LFS downloads for
nobranch SRC_URIs.
(Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217)
Signed-off-by: Desone Burns II <dburns@seegrid.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In _parse_recipe(), bb_data was initially a datastore object, but then
after the call to bb.parse.handle() it was all of a sudden turned into a
dict of variant:datastore pairs. At the same time, parseRecipeVariants()
used bb_data for datastore objects, while parseRecipe() used it for the
dicts of variant:datastore pairs.
Change these functions to consistently use "bb_data" for datastore
objects and use "datastores" for the dicts of variant:datastore pairs.
(Bitbake rev: ed0f94c037476efd465db87726b2480406076beb)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running `devtool search gcc` would result in errors like this for
virtual recipes:
ERROR: When reparsing
.../meta/recipes-devtools/gcc/libgcc-initial_13.2.bb:do_populate_sysroot,
the basehash value changed from
b1cd809ed98cef9db0fb1b17d34c4083e739c336f9d5619b89715b0294d81af5 to
44c2f92781dc4a20e98b7bb4724e204e64b101905fa75e71241a574b725997dc. The
metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Snone
ERROR: Then:
ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Sprintdiff
The reason was the newly introduced :layer-<layername> override, which
is used, e.g., in meta/classes-global/insane.bbclass to add the
patch-status QA test only for the meta layer:
ERROR_QA:append:layer-core = " patch-status"
When tinfoil parsed the recipes using the parseRecipeFile() function, it
failed to properly identify the layername for virtual recipes, which
resulted in the error above.
The correct thing to do is to make parseRecipeFile() call
bb.cache.virtualfn2realfn() to convert the virtual filename into a real
filename and virtual class.
(Bitbake rev: da2aed134412f5de04d7b540f92d735983ad0108)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we expand the variable AUTOREV in OE-Core, it triggers side effects in the
fetcher. The situation isn't ideal and needs improvement but this breaks
and is blocking enabling BB_DEFAULT_EVENTLOG.
Hack around the issue for now so we unblock things until we can work out
a better plan for how to improve AUTOREV support.
(Bitbake rev: cb9b6530f3d12c56a8b48847af2e7461924205d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently it is only possible to specify an eventlog on the bitbake
commandline. Add a variable that can be used in bitbake.conf so that
we can log data by default more easily.
(Bitbake rev: ee174b231897a53cdde0f68769518342e53210cf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The event at the end of the function doesn't really belong there,
move it out so the function becomes cleaner. It also avoids having
to pass a number of parameters around, another sign this is the correct
thing to do. This continues refactoring started a while ago to improve
and modularise the code.
(Bitbake rev: f712b927a84e8b8deaa8c907e8f9bcdec681f2aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Conceptually, this belongs in sqdata so this is the next step in refactoring
this code, continuing what was started a while ago.
(Bitbake rev: 2cf267cb657cd1864db43955cc010abba7a64093)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running an asyncrpc server as a subprocess, it is often desired to
run it with a lower logging level since the normal logging of clients
connecting and disconnecting is not desired.
As such, add an option to set the logging level of the server when
running as a subprocess and set the level to WARNING when starting a
local hashserver or PRserver
(Bitbake rev: 61dac7b99ad6d2a858f85d8ed1b5524d558be6c8)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an API to explicitly disconnect a client. This can be useful for
testing the auto-reconnect behavior of clients
(Bitbake rev: cb2fec7eaca20608faf4b8ff11ec8590ac7f2229)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Invalidate the loop in the client close() call so that it is not closed
twice (which is an error in the asyncio code)
(Bitbake rev: ef22f8ee82c242383625f078baafb629e45dad7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for Invocation Errors (that is, errors raised by the actual
RPC call instead of at the protocol level) to propagate across the
connection. If a server RPC call raises an InvokeError, it will be sent
across the connection and then raised on the client side also. The
connection is still terminated on this error.
(Bitbake rev: 50ee68175e7cf20a32bfbb176db2c47d7859da04)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an SQLAlchemy backend to the server. While this database backend is
slower than the more direct sqlite backend, it easily supports just
about any SQL server, which is useful for large scale deployments.
(Bitbake rev: e0b73466dd7478c77c82f46879246c1b68b228c0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a logging adaptor to the asyncrpc clients that prefixes log
messages with the client remote address to aid in debugging
(Bitbake rev: f4d64ce73c2449c008ff5d9b32376a2893ef7195)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
(Bitbake rev: d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support to the hash equivalence client and server to communicate
over websockets. Since websockets are message orientated instead of
stream orientated, and new connection class is needed to handle them.
Note that websocket support does require the 3rd party websockets python
module be installed on the host, but it should not be required unless
websockets are actually being used.
(Bitbake rev: 56dd2fdbfb6350a9eef43a12aa529c8637887a7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrites the asyncrpc client and server code to make it possible to have
other transport backends that are not stream based (e.g. websockets
which are message based). The connection handling classes are now shared
between both the client and server to make it easier to implement new
transport mechanisms
(Bitbake rev: 2aaeae53696e4c2f13a169830c3b7089cbad6eca)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running printdiff, messages like:
"Can't find a task we're supposed to have written out?"
might be shown. This happens when hashequivalence is enabled and is due
to the use of "hash" when "unihash" should be used (which defaults back
to hash). Changing this fixed various errors oe-selftest highlighted.
Also print the task ID when erroring about a missing task as it aids
debugging.
(Bitbake rev: e7783bbe0559abb91f5038cf83575c0f25138af2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If --runall is used with setscene tasks, it will still skip them if it
can. By marking the tasks as targets, this avoids that skipping
and means --runall deploy_source_date_epoch works as expected for
example.
(Bitbake rev: 8c6cd1e8dd224bc1cc92736cbb7e113fd1fb5aae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The IPC mechanism between runqueue.py and bitbake-worker is currently
not scalable:
The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker
has no information about the size of the message. Therefore, the bitbake-worker
is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")"
for each chunk received.
This does not scale, because queue.find has a linear complexity relative to the size of the queue,
and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI.
The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each
iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup
35 seconds before this fix, and 1.5 seconds after this fix).
This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be
received, and does not need to constantly search the whole queue for </tag>.
(Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pre-fetching of LFS blobs now also works if a destsuffix parameter is
supplied.
This also fixes issues with Yocto Kirkstone (and newer) builds, where
fetching the LFS blobs is retried unsuccessfully during do_unpack, as
network access is blocked for any task other than do_fetch.
(Bitbake rev: e411dc07d032be6811d0393c50a06fc28e669b24)
Signed-off-by: Ragesh Nair <ragesh.nair@duagon.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is printed by 'bitbake -S printdiff' and more accurately reflects what
the code does.
Most of the time the most recent item should be what the user wants to see when debugging sstate
misses, but there could also be printdiff-all (print differences with all
matching sstate objects), or printdiff-N (N most recent, and not just the latest).
(Bitbake rev: 315f48f965a54da5cec92908d91aa61c2d450add)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create repository with few branches and test if fetcher can work
with such repository as PREMIRROR
(Bitbake rev: a1737610e5d5b61e126ec3632d7f27b337a87818)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds an API to bb.fetch2 to enable users to plug in an unpack
tracer that can trace each source file back to its corresponding
upstream source url, even when multiple upstream sources are combined
together in the same unpack directory. This may be required for software
composition analysis, license compliance, and detailed SBoM generation.
This patch provides only the needed hooks in bb.fetch2 code and a dummy
abstract class defining the API; users may load their own unpack tracer
class by setting the BB_UNPACK_TRACER_CLASS config parameter.
(Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f)
Signed-off-by: Alberto Pianon <alberto@pianon.eu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The infrastructure changes highlighed an issue where the top level directory
mtime stamp isn't included in the tarballs. This isn't an issue the test is
meant to check for so ignore it.
(Bitbake rev: 5a1e5b8dc55d6bb4d93bac3492f8c43ff957e712)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the
signatures using tinfoil
(Bitbake rev: 05c15162de90c41dad67e37a95ec9fdb440a7864)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Visual studio code is injecting a git credential helper
into every dev container. This helper is forwarding the git credentials
from the host into the container.
In order for this helper to work the REMOTE_CONTAINERS_IPC
needs to be available in the environment for all git operations
that require authentication.
(Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963)
Signed-off-by: Matthias Schnelte <develop@schnelte.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user may have invoked ./bin/bitbake using a different Python
interpreter than whatever python3 is on $PATH (for example, explicitly
using a different version). However, as the server and workers are
spawned directly they'll use the hashbang and thus a different Python.
We also ensure that argv[0] is set to sys.executable instead of
'bitbake-server' or 'bitbake-worker', so that sys.executable is set to
the right value inside the child. Without this the server won't be
able to start any workers.
(Bitbake rev: b44d5d2a53d3082c8ce94e09c0cf833e33e25aec)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If Tinfoil is initialized with setup_logging = False and
Tinfoil.prepare() is called with config_only = False, then it fails
because self.localhandlers is only initialized when
setup_logging = True.
This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`:
Traceback (most recent call last):
File ".../bitbake/bin/bitbake-getvar", line 41, in <module>
tinfoil.prepare(quiet=2)
File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare
for handler in self.localhandlers:
AttributeError: 'Tinfoil' object has no attribute 'localhandlers'.
Did you mean: 'oldhandlers'?
(Bitbake rev: 616101ddb630e2c9975022068b52a87c4cf647f6)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code has changed and the debug message didn't work. Fix it. The output
is still incredibly useful.
(Bitbake rev: f1fa4bb3066e2bbaff0b69088ba5c6c6c597b93d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
declaring queue=b"" creates an object of types bytes().
bytes() is an immutable object, and therefore doing "self.queue = self.queue + r"
creates a new object containing "self.queue" concatenated with "r".
On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker,
so those copies significantly slow down the initialization of the bitbake-worker.
Rather use bytearray() which a mutable type, and use extend() to avoid copies.
In my test setup, byterray.extend() is 10.000 times faster than copying the queue,
for a queue size of 180MB.
(Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A recipe variable handles its dependencies even on the "contains"
variables within the "inline Python expressions" like bb.utils.filter().
And it also handles those in the append operator correctly, but the
problem is that it does not so in the remove operator.
Fix it by adding the missing dependencies every time the remove
operator has been handled.
Also add a test case to check if the override operators handle
dependencies correctly.
(Bitbake rev: b90520eedb1dbc7f6a3928d089fe74fafb864eb5)
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
