Adds an API to explicitly disconnect a client. This can be useful for
testing the auto-reconnect behavior of clients
(Bitbake rev: cb2fec7eaca20608faf4b8ff11ec8590ac7f2229)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Invalidate the loop in the client close() call so that it is not closed
twice (which is an error in the asyncio code)
(Bitbake rev: ef22f8ee82c242383625f078baafb629e45dad7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for Invocation Errors (that is, errors raised by the actual
RPC call instead of at the protocol level) to propagate across the
connection. If a server RPC call raises an InvokeError, it will be sent
across the connection and then raised on the client side also. The
connection is still terminated on this error.
(Bitbake rev: 50ee68175e7cf20a32bfbb176db2c47d7859da04)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an SQLAlchemy backend to the server. While this database backend is
slower than the more direct sqlite backend, it easily supports just
about any SQL server, which is useful for large scale deployments.
(Bitbake rev: e0b73466dd7478c77c82f46879246c1b68b228c0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a logging adaptor to the asyncrpc clients that prefixes log
messages with the client remote address to aid in debugging
(Bitbake rev: f4d64ce73c2449c008ff5d9b32376a2893ef7195)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
(Bitbake rev: d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support to the hash equivalence client and server to communicate
over websockets. Since websockets are message orientated instead of
stream orientated, and new connection class is needed to handle them.
Note that websocket support does require the 3rd party websockets python
module be installed on the host, but it should not be required unless
websockets are actually being used.
(Bitbake rev: 56dd2fdbfb6350a9eef43a12aa529c8637887a7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrites the asyncrpc client and server code to make it possible to have
other transport backends that are not stream based (e.g. websockets
which are message based). The connection handling classes are now shared
between both the client and server to make it easier to implement new
transport mechanisms
(Bitbake rev: 2aaeae53696e4c2f13a169830c3b7089cbad6eca)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running printdiff, messages like:
"Can't find a task we're supposed to have written out?"
might be shown. This happens when hashequivalence is enabled and is due
to the use of "hash" when "unihash" should be used (which defaults back
to hash). Changing this fixed various errors oe-selftest highlighted.
Also print the task ID when erroring about a missing task as it aids
debugging.
(Bitbake rev: e7783bbe0559abb91f5038cf83575c0f25138af2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If --runall is used with setscene tasks, it will still skip them if it
can. By marking the tasks as targets, this avoids that skipping
and means --runall deploy_source_date_epoch works as expected for
example.
(Bitbake rev: 8c6cd1e8dd224bc1cc92736cbb7e113fd1fb5aae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The IPC mechanism between runqueue.py and bitbake-worker is currently
not scalable:
The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker
has no information about the size of the message. Therefore, the bitbake-worker
is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")"
for each chunk received.
This does not scale, because queue.find has a linear complexity relative to the size of the queue,
and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI.
The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each
iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup
35 seconds before this fix, and 1.5 seconds after this fix).
This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be
received, and does not need to constantly search the whole queue for </tag>.
(Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pre-fetching of LFS blobs now also works if a destsuffix parameter is
supplied.
This also fixes issues with Yocto Kirkstone (and newer) builds, where
fetching the LFS blobs is retried unsuccessfully during do_unpack, as
network access is blocked for any task other than do_fetch.
(Bitbake rev: e411dc07d032be6811d0393c50a06fc28e669b24)
Signed-off-by: Ragesh Nair <ragesh.nair@duagon.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is printed by 'bitbake -S printdiff' and more accurately reflects what
the code does.
Most of the time the most recent item should be what the user wants to see when debugging sstate
misses, but there could also be printdiff-all (print differences with all
matching sstate objects), or printdiff-N (N most recent, and not just the latest).
(Bitbake rev: 315f48f965a54da5cec92908d91aa61c2d450add)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create repository with few branches and test if fetcher can work
with such repository as PREMIRROR
(Bitbake rev: a1737610e5d5b61e126ec3632d7f27b337a87818)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds an API to bb.fetch2 to enable users to plug in an unpack
tracer that can trace each source file back to its corresponding
upstream source url, even when multiple upstream sources are combined
together in the same unpack directory. This may be required for software
composition analysis, license compliance, and detailed SBoM generation.
This patch provides only the needed hooks in bb.fetch2 code and a dummy
abstract class defining the API; users may load their own unpack tracer
class by setting the BB_UNPACK_TRACER_CLASS config parameter.
(Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f)
Signed-off-by: Alberto Pianon <alberto@pianon.eu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The infrastructure changes highlighed an issue where the top level directory
mtime stamp isn't included in the tarballs. This isn't an issue the test is
meant to check for so ignore it.
(Bitbake rev: 5a1e5b8dc55d6bb4d93bac3492f8c43ff957e712)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the
signatures using tinfoil
(Bitbake rev: 05c15162de90c41dad67e37a95ec9fdb440a7864)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Visual studio code is injecting a git credential helper
into every dev container. This helper is forwarding the git credentials
from the host into the container.
In order for this helper to work the REMOTE_CONTAINERS_IPC
needs to be available in the environment for all git operations
that require authentication.
(Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963)
Signed-off-by: Matthias Schnelte <develop@schnelte.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user may have invoked ./bin/bitbake using a different Python
interpreter than whatever python3 is on $PATH (for example, explicitly
using a different version). However, as the server and workers are
spawned directly they'll use the hashbang and thus a different Python.
We also ensure that argv[0] is set to sys.executable instead of
'bitbake-server' or 'bitbake-worker', so that sys.executable is set to
the right value inside the child. Without this the server won't be
able to start any workers.
(Bitbake rev: b44d5d2a53d3082c8ce94e09c0cf833e33e25aec)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If Tinfoil is initialized with setup_logging = False and
Tinfoil.prepare() is called with config_only = False, then it fails
because self.localhandlers is only initialized when
setup_logging = True.
This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`:
Traceback (most recent call last):
File ".../bitbake/bin/bitbake-getvar", line 41, in <module>
tinfoil.prepare(quiet=2)
File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare
for handler in self.localhandlers:
AttributeError: 'Tinfoil' object has no attribute 'localhandlers'.
Did you mean: 'oldhandlers'?
(Bitbake rev: 616101ddb630e2c9975022068b52a87c4cf647f6)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code has changed and the debug message didn't work. Fix it. The output
is still incredibly useful.
(Bitbake rev: f1fa4bb3066e2bbaff0b69088ba5c6c6c597b93d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
declaring queue=b"" creates an object of types bytes().
bytes() is an immutable object, and therefore doing "self.queue = self.queue + r"
creates a new object containing "self.queue" concatenated with "r".
On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker,
so those copies significantly slow down the initialization of the bitbake-worker.
Rather use bytearray() which a mutable type, and use extend() to avoid copies.
In my test setup, byterray.extend() is 10.000 times faster than copying the queue,
for a queue size of 180MB.
(Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A recipe variable handles its dependencies even on the "contains"
variables within the "inline Python expressions" like bb.utils.filter().
And it also handles those in the append operator correctly, but the
problem is that it does not so in the remove operator.
Fix it by adding the missing dependencies every time the remove
operator has been handled.
Also add a test case to check if the override operators handle
dependencies correctly.
(Bitbake rev: b90520eedb1dbc7f6a3928d089fe74fafb864eb5)
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since the flush calls have significant effects for bitbake timeout issues,
drop the remaining ones from cooker. These aren't in as critical paths
as the other issues but it makes sense to clean up.
(Bitbake rev: dd682363341bae3b060e284d73f000813964dc05)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been chasing bitbake timeouts for a while and it was unclear where things
were blocking on IO. It appears the flush() call in server logging can cause
pauses up to minutes long on systems with slow (spinning) disks that are heavily
loaded with IO.
Since the flush() was added to aid debugging of other timing issues, we shouldn't
need it now and it can be disabled. Leave a comment as a reminder of the pain this
can cause.
(Bitbake rev: afbc169e1490a86d6250969f780062c426eb4682)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the flush in serverlog() removed and a memory resident bitbake with a
60s timeout, the following could fail in strange ways:
rm bitbake-cookerdaemon.log
bitbake-layers add-layer ../meta-virtualization/
bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake -m
specifically that it might error adding meta-oe with an error related to meta-virt.
This clearly shows that whilst bblayers.conf was modified, bitbake was not
recognising that. This would fit with the random autobuilder issues seen when
the serverlog flush() call was removed.
The issue appears to be that you have no way to "sync()" the inotify events with
the command stream coming over the socket. There is no way to know if there are
changes in the IO queue which bitbake needs to wait for before proceeding with
the next command.
I did experiment with os.sync() and fsync on the inotify fd, however nothing
addressed the issue. Since it is extremely important we have accurate cache data,
the only realistic thing to do is to switch to stat() calls and check mtime.
For bitbake commands, this is straightforward since we can revalidate the cache
upon new connections/commands. For tinfoil this is problematic and we need to
introduce and explict command "revalidateCaches" that the code can use to force
bitbake to re-check it's cache validity. I've exposed this through tinfoil with
a new "modified_files" function.
So, this patch:
a) drops inotify support within bitbake's cooker/server and switch to using mtime
b) requires a new function call in tinfoil when metadata has been modified
(Bitbake rev: da3ec3801bdb80180b3f1ac24edb27a698415ff7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using path prefixes to check if the git directory is a descendant of the
clone directory can be easily confused with symlinkes and bind mounts,
causing directories to be deleted unnecessarily. Instead, use
bb.utils.path_is_descendant() which is immune to the these sorts of
problems.
(Bitbake rev: b4d7a0546630620480b7fee159b84c3506e941a2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a utility that checks if one path is an descendant of another. This
check uses os.path.samestat() to make it immune to symlinks and bind
mounts.
(Bitbake rev: c3ae45946886ee2049939dd5a205790657a7de32)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is helpful to have timestamps on the ping failures so that they
can be matched against the bitbake logs. It is also useful to understand
how long the server takes for form a reply verses when it is sent.
(Bitbake rev: 65969a7a8f5ae22c230431d2db080eb187a27708)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* it might be a bit confusing as it shows number of threads before making
the decision to start more tasks and also it can show only a few tasks
running, but not because of pressure when there just aren't many tasks
left or wait for their dependencies to be finished first
* example output:
NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads
NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads
(Bitbake rev: b0b114f31f20c5fcde31e6c308937ad4102dfe0a)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no reason to have the else clause in this try block, as it can
be moved into the try block, which is clearer.
(Bitbake rev: 5625849e9327fc71a38eea00d4506f80abc11bc6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We noticed some users were seeing very slow ping response times which
caused 'server timeout' issues. There were some function calls in runCommand
which could be slow such as the inotify callback processing.
Mark up the ping command such that it doesn't need configuration information,
it is allowed on a readonly server and specifically skip the inofity processing
too since ping would never need that.
This will hopefully resolve various ping timeout issues that were being reported.
(Bitbake rev: 0fc821a22f2b49cbd336d9658d92942c0d733be1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value "Image Name" in bitbake events was missing for certain builds. Update 'buildinfohelper' to extract the
image name elsewhere in this circumstance and not crash.
[YOCTO #13191]
(Bitbake rev: 703792c48c818025163de9c2f35f6ac815500607)
Signed-off-by: Kieran McNulty <Kieran.McNulty@windriver.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the clone target directory exists and is a valid git repo, but the
git directory is not a child, it needs to be erased and re-cloned. One
example of how this can happen is if a clone creates the directory, but
then fails to actual clone and make it a git repository. This left-over
directory can be particularly problematic if the download directory is a
descent of some top layer git repo (e.g. the default with poky), as the
commands that operate on the remote later will then mangle the layers
git repository instead of the download git repo.
(Bitbake rev: 2117db3146ce38bb4a6e2df40b6cd2ab11b514d5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Explicitly pass down the lfs parameter from the source repo URI to
submodules. Add smudge skip to final checkout phase to make sure we
don't access the network here. Everything should have been fetched and
setup from the lfs logic in the git fetcher at this point.
(Bitbake rev: 1f8f21fe024b391d3a6670c64b839db0eee083c3)
Signed-off-by: Randolph Sapp <rs@ti.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Error occured while running bitbake on cephfs:
WARNING: The free inode of path is running low (-0.001K left)
ERROR: Immediately halt since the disk space monitor action is "HALT"!
(Bitbake rev: 95088b447f563c5e1d9630e6acb32787b5ebed9c)
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds Python wrappers around the xattr API from libc and the ACL API from
libacl.
(Bitbake rev: 538011256964d0253f8e3ab7ff1d6fd62c7c2f89)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* with latest bitbake I'm seeing very low number of bitbake tasks
executed in parallel, probably due to pressure regulation
show the values this is based on in the note
* also simplify a bit by counting the pressure and exceeds signs
only once
(Bitbake rev: 21c17968f801e406ef7f328656587fadd9ef7f5d)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add new functions to return some of the get_srcrev data in new and different
ways. We need two different forms of the data, one is a string to inject into
PKGV, the other is the full revisions as a string to include in hash computations
so that the hash changes when the input revisions change.
This allows us to clean up and simplify the code in OE-Core and move the
version information from PV to PKGV.
(Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
file checksums are part of the data checksummed
to generate the task hash. The list of file checksums
was not ordered.
In this commit we make sure the task hash checksum takes
a list of checksum data that is ordered by unique file name
thus guaranteeing reproducibility.
(Bitbake rev: 5293a1b36eeb89f57577cb709ec7f293909039a1)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The debug in the comments was out of date. It is still useful so update the
code sample to the new code needed.
(Bitbake rev: fa2724069ea7028939d816cb5ccd0e7c1bed09a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current calculation is not correct because if tdiff is less than
1.0, it's not taken into consideration when calculating the current
pressure.
Also, make it clear that the 1.0s is the psi accumulation cycle,
which might be changed in the future. We have this cycle because it
could largely avoid the 0 result issue, that is, if the interval
between checks are too small, the result might be 0. With this
accumulation logic, which has been there but let's make it clear,
this 0 result problem could be mitigated.
(Bitbake rev: 95fa8fb5fb4d5a72e79b11d69792613bfd494e72)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fetcher allows BitBake to fetch from a Google Cloud Storage
bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and
expects it to be installed, configured, and authenticated prior
to use.
If accepted, this patch should merge in with the corresponding oe-core
patch titled "Add GCP fetcher to list of supported protocols".
Some comments on the patch:
There is also documentation for the fetcher added to the User
Manual. I'm still not completely sure about the recommends_checksum()
being set to True. As I've noted in the mailing list, it will throw
warnings if the fetcher is used in recipes without specifying a
checksum. Please let me know if this is intended behavior or if it
should be modified.
Here is how this fetcher conforms to the fetcher expectations
described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no
network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher
only downloads to the DL_DIR in the same way that other fetchers,
namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this
context
e) Yes, the fetcher output is deterministic because it is downloading
tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy
variable to test whether the Python API respected the proxy. It
appears that it did as I could see traffic passing through the
proxy. I also did some searching online and found posts indicating
that the Google Cloud Python APIs supported the classic Linux proxy
variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading
it if it does.
h) Not applicable, BitBake already knows which version it wants and
the version infomation is encoded in the filename. The fetcher has no
concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests
for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any
tools during parse time. Please correct me if I'm wrong.
(Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b)
Signed-off-by: Emil Ekmečić <eekmecic@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
