Adds a subcommand to clean unused outhash entries from the server based
on age
(Bitbake rev: 29c310be2c00d0e192923a4b90c60771ef0c620c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an API to remove unused entries in the outhash database based on
age and if they are referenced by any unihash
(Bitbake rev: a169ac523d166c6cbba918b152a76782176c3e88)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extends the get_outhash API with a flag indicating whether to include
the unihash in the output. This is means that the query doesn't require
the unihash entry to be present to return a result
(Bitbake rev: b8d6abfeb4a0765727a62b3d8d83276335c7c7d6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a subcommand to invoke the remove API on the server
(Bitbake rev: a778b99e6a2a1a1e1b7eb26d48313fadcd34de54)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a `remove` API to the client and server that can be used to remove
hash equivalence entries that match a particular critera
(Bitbake rev: 861d068b3a9fb5e91a01dbec54996a5a6f93ef29)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The infrastructure changes highlighed an issue where the top level directory
mtime stamp isn't included in the tarballs. This isn't an issue the test is
meant to check for so ignore it.
(Bitbake rev: 5a1e5b8dc55d6bb4d93bac3492f8c43ff957e712)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updated selenium version to latest 4.13.0, changed selenum specific version syntax elements to accomplish test success
(Bitbake rev: 868c88a249ef4b9fe5a891e76e25e054e4fcd994)
Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than outputting the string "None" for undefined variables, output
only a linefeed (the same as for variables that are defined to the empty
string).
(Bitbake rev: f3ba9c3726ec7b38b557100d8a2d4b6a1446a968)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an undefined variable or variable flag is specified, bitbake-getvar
will now fail with an error message indicating this.
The error can be supressed with --ignore-undefined, which matches the
previous behavior.
This also changes the errors related to specifying --flag or --unexpand
without --value so that they are sent to stderr rather than stdout.
(Bitbake rev: 136b8dda4e8b6f4d7e45a552c2d2e278b3ae1b7d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the
signatures using tinfoil
(Bitbake rev: 05c15162de90c41dad67e37a95ec9fdb440a7864)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Visual studio code is injecting a git credential helper
into every dev container. This helper is forwarding the git credentials
from the host into the container.
In order for this helper to work the REMOTE_CONTAINERS_IPC
needs to be available in the environment for all git operations
that require authentication.
(Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963)
Signed-off-by: Matthias Schnelte <develop@schnelte.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user may have invoked ./bin/bitbake using a different Python
interpreter than whatever python3 is on $PATH (for example, explicitly
using a different version). However, as the server and workers are
spawned directly they'll use the hashbang and thus a different Python.
We also ensure that argv[0] is set to sys.executable instead of
'bitbake-server' or 'bitbake-worker', so that sys.executable is set to
the right value inside the child. Without this the server won't be
able to start any workers.
(Bitbake rev: b44d5d2a53d3082c8ce94e09c0cf833e33e25aec)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It does not make any sense to get log output from bitbake-getvar when
the --value option is used as the log output is sent to stdout and thus
interferes with the output of the variable's value.
(Bitbake rev: 6b7883533af9c14d80a9f1ae5142644f155b5dee)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Initializing Tinfoil with setup_logging = False only has an effect when
recipe parsing is not needed. To make it work regardless of if --recipe
is used, manipulate the quiet parameter to Tinfoil.prepare() instead.
(Bitbake rev: 71ee69a20f21f3d37f4f060a7d8e87d9f1dc6aa1)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If Tinfoil is initialized with setup_logging = False and
Tinfoil.prepare() is called with config_only = False, then it fails
because self.localhandlers is only initialized when
setup_logging = True.
This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`:
Traceback (most recent call last):
File ".../bitbake/bin/bitbake-getvar", line 41, in <module>
tinfoil.prepare(quiet=2)
File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare
for handler in self.localhandlers:
AttributeError: 'Tinfoil' object has no attribute 'localhandlers'.
Did you mean: 'oldhandlers'?
(Bitbake rev: 616101ddb630e2c9975022068b52a87c4cf647f6)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code has changed and the debug message didn't work. Fix it. The output
is still incredibly useful.
(Bitbake rev: f1fa4bb3066e2bbaff0b69088ba5c6c6c597b93d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
declaring queue=b"" creates an object of types bytes().
bytes() is an immutable object, and therefore doing "self.queue = self.queue + r"
creates a new object containing "self.queue" concatenated with "r".
On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker,
so those copies significantly slow down the initialization of the bitbake-worker.
Rather use bytearray() which a mutable type, and use extend() to avoid copies.
In my test setup, byterray.extend() is 10.000 times faster than copying the queue,
for a queue size of 180MB.
(Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A recipe variable handles its dependencies even on the "contains"
variables within the "inline Python expressions" like bb.utils.filter().
And it also handles those in the append operator correctly, but the
problem is that it does not so in the remove operator.
Fix it by adding the missing dependencies every time the remove
operator has been handled.
Also add a test case to check if the override operators handle
dependencies correctly.
(Bitbake rev: b90520eedb1dbc7f6a3928d089fe74fafb864eb5)
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since the flush calls have significant effects for bitbake timeout issues,
drop the remaining ones from cooker. These aren't in as critical paths
as the other issues but it makes sense to clean up.
(Bitbake rev: dd682363341bae3b060e284d73f000813964dc05)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been chasing bitbake timeouts for a while and it was unclear where things
were blocking on IO. It appears the flush() call in server logging can cause
pauses up to minutes long on systems with slow (spinning) disks that are heavily
loaded with IO.
Since the flush() was added to aid debugging of other timing issues, we shouldn't
need it now and it can be disabled. Leave a comment as a reminder of the pain this
can cause.
(Bitbake rev: afbc169e1490a86d6250969f780062c426eb4682)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the flush in serverlog() removed and a memory resident bitbake with a
60s timeout, the following could fail in strange ways:
rm bitbake-cookerdaemon.log
bitbake-layers add-layer ../meta-virtualization/
bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake -m
specifically that it might error adding meta-oe with an error related to meta-virt.
This clearly shows that whilst bblayers.conf was modified, bitbake was not
recognising that. This would fit with the random autobuilder issues seen when
the serverlog flush() call was removed.
The issue appears to be that you have no way to "sync()" the inotify events with
the command stream coming over the socket. There is no way to know if there are
changes in the IO queue which bitbake needs to wait for before proceeding with
the next command.
I did experiment with os.sync() and fsync on the inotify fd, however nothing
addressed the issue. Since it is extremely important we have accurate cache data,
the only realistic thing to do is to switch to stat() calls and check mtime.
For bitbake commands, this is straightforward since we can revalidate the cache
upon new connections/commands. For tinfoil this is problematic and we need to
introduce and explict command "revalidateCaches" that the code can use to force
bitbake to re-check it's cache validity. I've exposed this through tinfoil with
a new "modified_files" function.
So, this patch:
a) drops inotify support within bitbake's cooker/server and switch to using mtime
b) requires a new function call in tinfoil when metadata has been modified
(Bitbake rev: da3ec3801bdb80180b3f1ac24edb27a698415ff7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using path prefixes to check if the git directory is a descendant of the
clone directory can be easily confused with symlinkes and bind mounts,
causing directories to be deleted unnecessarily. Instead, use
bb.utils.path_is_descendant() which is immune to the these sorts of
problems.
(Bitbake rev: b4d7a0546630620480b7fee159b84c3506e941a2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a utility that checks if one path is an descendant of another. This
check uses os.path.samestat() to make it immune to symlinks and bind
mounts.
(Bitbake rev: c3ae45946886ee2049939dd5a205790657a7de32)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is helpful to have timestamps on the ping failures so that they
can be matched against the bitbake logs. It is also useful to understand
how long the server takes for form a reply verses when it is sent.
(Bitbake rev: 65969a7a8f5ae22c230431d2db080eb187a27708)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* it might be a bit confusing as it shows number of threads before making
the decision to start more tasks and also it can show only a few tasks
running, but not because of pressure when there just aren't many tasks
left or wait for their dependencies to be finished first
* example output:
NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads
NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads
(Bitbake rev: b0b114f31f20c5fcde31e6c308937ad4102dfe0a)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable is implemented in OE-Core, and therefore
only documented in the Yocto Project manual.
(Bitbake rev: 1019998e2f8682c8f2f13320fdb0de1a9a23e854)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using tags with the git fetcher may cause surprising behaviour. There
are reasons for this, document them.
(Bitbake rev: 56224da378ab63526d44fd7a70bcfd2cffe245cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no reason to have the else clause in this try block, as it can
be moved into the try block, which is clearer.
(Bitbake rev: 5625849e9327fc71a38eea00d4506f80abc11bc6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We noticed some users were seeing very slow ping response times which
caused 'server timeout' issues. There were some function calls in runCommand
which could be slow such as the inotify callback processing.
Mark up the ping command such that it doesn't need configuration information,
it is allowed on a readonly server and specifically skip the inofity processing
too since ping would never need that.
This will hopefully resolve various ping timeout issues that were being reported.
(Bitbake rev: 0fc821a22f2b49cbd336d9658d92942c0d733be1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have a contributor guide combining various wiki pages, point at
that.
(Bitbake rev: fb19d647697d56e7554722abb5f4903c774d4213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value "Image Name" in bitbake events was missing for certain builds. Update 'buildinfohelper' to extract the
image name elsewhere in this circumstance and not crash.
[YOCTO #13191]
(Bitbake rev: 703792c48c818025163de9c2f35f6ac815500607)
Signed-off-by: Kieran McNulty <Kieran.McNulty@windriver.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you import a build directory, Toaster still adds openembedded-core,
meta-poky and meta-yocto-bsp to the newly created project. Toaster
should only be including in the project the layers that it imported.
[YOCTO #13764]
(Bitbake rev: e73c4d7685a3bd6b806a8f1a3600a3a86266f0b6)
Signed-off-by: Kieran McNulty <Kieran.McNulty@windriver.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the clone target directory exists and is a valid git repo, but the
git directory is not a child, it needs to be erased and re-cloned. One
example of how this can happen is if a clone creates the directory, but
then fails to actual clone and make it a git repository. This left-over
directory can be particularly problematic if the download directory is a
descent of some top layer git repo (e.g. the default with poky), as the
commands that operate on the remote later will then mangle the layers
git repository instead of the download git repo.
(Bitbake rev: 2117db3146ce38bb4a6e2df40b6cd2ab11b514d5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Explicitly pass down the lfs parameter from the source repo URI to
submodules. Add smudge skip to final checkout phase to make sure we
don't access the network here. Everything should have been fetched and
setup from the lfs logic in the git fetcher at this point.
(Bitbake rev: 1f8f21fe024b391d3a6670c64b839db0eee083c3)
Signed-off-by: Randolph Sapp <rs@ti.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Error occured while running bitbake on cephfs:
WARNING: The free inode of path is running low (-0.001K left)
ERROR: Immediately halt since the disk space monitor action is "HALT"!
(Bitbake rev: 95088b447f563c5e1d9630e6acb32787b5ebed9c)
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds Python wrappers around the xattr API from libc and the ACL API from
libacl.
(Bitbake rev: 538011256964d0253f8e3ab7ff1d6fd62c7c2f89)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* with latest bitbake I'm seeing very low number of bitbake tasks
executed in parallel, probably due to pressure regulation
show the values this is based on in the note
* also simplify a bit by counting the pressure and exceeds signs
only once
(Bitbake rev: 21c17968f801e406ef7f328656587fadd9ef7f5d)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add new functions to return some of the get_srcrev data in new and different
ways. We need two different forms of the data, one is a string to inject into
PKGV, the other is the full revisions as a string to include in hash computations
so that the hash changes when the input revisions change.
This allows us to clean up and simplify the code in OE-Core and move the
version information from PV to PKGV.
(Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
file checksums are part of the data checksummed
to generate the task hash. The list of file checksums
was not ordered.
In this commit we make sure the task hash checksum takes
a list of checksum data that is ordered by unique file name
thus guaranteeing reproducibility.
(Bitbake rev: 5293a1b36eeb89f57577cb709ec7f293909039a1)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a --mc argument to `bitbake-layers show-appends` so that users can
filter appends for a specific multiconfig (instead of always showing the
default configuration)
(Bitbake rev: f4dcbf114554c829467623b5587314d16ebdf3eb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The debug in the comments was out of date. It is still useful so update the
code sample to the new code needed.
(Bitbake rev: fa2724069ea7028939d816cb5ccd0e7c1bed09a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current calculation is not correct because if tdiff is less than
1.0, it's not taken into consideration when calculating the current
pressure.
Also, make it clear that the 1.0s is the psi accumulation cycle,
which might be changed in the future. We have this cycle because it
could largely avoid the 0 result issue, that is, if the interval
between checks are too small, the result might be 0. With this
accumulation logic, which has been there but let's make it clear,
this 0 result problem could be mitigated.
(Bitbake rev: 95fa8fb5fb4d5a72e79b11d69792613bfd494e72)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fetcher allows BitBake to fetch from a Google Cloud Storage
bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and
expects it to be installed, configured, and authenticated prior
to use.
If accepted, this patch should merge in with the corresponding oe-core
patch titled "Add GCP fetcher to list of supported protocols".
Some comments on the patch:
There is also documentation for the fetcher added to the User
Manual. I'm still not completely sure about the recommends_checksum()
being set to True. As I've noted in the mailing list, it will throw
warnings if the fetcher is used in recipes without specifying a
checksum. Please let me know if this is intended behavior or if it
should be modified.
Here is how this fetcher conforms to the fetcher expectations
described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no
network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher
only downloads to the DL_DIR in the same way that other fetchers,
namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this
context
e) Yes, the fetcher output is deterministic because it is downloading
tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy
variable to test whether the Python API respected the proxy. It
appears that it did as I could see traffic passing through the
proxy. I also did some searching online and found posts indicating
that the Google Cloud Python APIs supported the classic Linux proxy
variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading
it if it does.
h) Not applicable, BitBake already knows which version it wants and
the version infomation is encoded in the filename. The fetcher has no
concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests
for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any
tools during parse time. Please correct me if I'm wrong.
(Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b)
Signed-off-by: Emil Ekmečić <eekmecic@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
