Drops support for client pools, since batching support in the client
code has proven to be much more effective
(Bitbake rev: 85dafaf8e070459f7de7bfb37300d8b60a27002e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: We can call multiprocessing close() unconditionally and tweak a
comment give 3.8 is now the minimum version.
lib/bb: We can drop the logger addition code only needed before 3.6
asyncrpc/hashserv: Since the minimum version is 3.8, we can drop the
conditional code.
(Bitbake rev: 16f4386400f88ba50605307961c248bef09895c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seeing profiling tracebacks when parse errors occur during
profiling. Try and avoid these but not processing invalid profiles.
(Bitbake rev: 171bd9dd575307fbd61b5179ad86131d76add067)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It makes sense to switch to a more recent version and keep up to date
with upstream changes and things like new python version support.
(Bitbake rev: f5462156036e71911c66d07dbf3303cde862785b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SO_REUSEPORT is a socket option that allows multiple servers to listen
on the same TCP port, and the kernel will automatically load balance the
connections between them. This is particularly helpful for the hash
server since it runs in a single thread. To take advantage of a
multi-core server, multiple servers can be started in parallel with this
option (up to 1 per CPU) and the kernel will load balance between them.
(Bitbake rev: d72d5a7decb489e2af0ebc43cfea0ca3e4353e9b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Uses the batching API of the client to reduce the effect of latency when
making multiple queries to the server
(Bitbake rev: a54734b4ac2ddb3bce004e576cf74d6ad6caf62a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes the stream mode to do "batch" processing. This means that the
sending and reciving of messages is done simultaneously so that messages
can be sent as fast as possible without having to wait for each reply.
This allows multiple messages to be in flight at once, reducing the
effect of the round trip latency from the server.
(Bitbake rev: e768d0f17bdb97f6ff013ec3a41f182fecd47a55)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Task dependency data is becoming unwieldy with the number of indices it
contains. Convert it to use a named tuple instead, which allows members
to be indexed by a named property or an index (which allows it to retain
backward compatibility).
(Bitbake rev: 26446cca4d22734c3f1b328a205c169dadb7e494)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a --quiet option to suppress the message for each ping, and report
the median ping time.
(Bitbake rev: 3c85b5e2d9b9c39507ed362aaa115b7f6f155966)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improve the rehash code to query unihashes in parallel since this is more
efficient on slower links.
(Bitbake rev: c1949d5350342eaaf6ab988d7bfba99496d55523)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improve the runqueue init code to call unihash queries in parallel since
this is faster and more efficient, particularly on slower links with longer
round trip times.
The call to the function from cooker is unneeded since that function calls
prepare() and hence this functionality will already have run, so drop
that obsolete call.
(Bitbake rev: 721c97a115a7a4bf21955be79391bd6e0099f40e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The initial hash serve loop exits in the case where interrupts are present
but probably checks a bit too often. Tweak that and also allow the slow
rehash loop to break on interrupt, improving bitbake Ctrl+C response.
(Bitbake rev: 4534365591fd17bcc2b684900863b67bc69519ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With hashserve enabled, there are two slow paths/loops, one at initial runqueue
generation and also during the rehash process when new outhashes are found.
Add timing information at the hashserve log level for when these loops
take longer than 30s or 60s overall. This will leave evidence in the logs when
things are running particularly slowly.
(Bitbake rev: 6c357ede08e0b2a93bdaad2c1d631994faf2b784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improves the way statistics are reported for the stress test. This makes
it easier to compare them to the ping test
(Bitbake rev: ce166ae25793c11b0a190c531bef0c296fd74497)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a ping subcommand to bitbake-hashclient which can be useful to
measure connection latency
(Bitbake rev: 337487fdffae92091fc33b2346d46c39db5a130f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I don't know why there is hardcoded knowledge of S in the fetcher but there
shouldn't be and the OE unpack changes highlight this doing things it
shouldn't.
Drop the S reference and use rootdir which is the only place it should
be touching.
(Bitbake rev: 84f102954e10a3390fca9c26d5c3c639e952a2c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplify the importone() hook:
- to make it independent from the "history" mode which is
client specific.
- remove the "history" parameter
- we want all values to be imported for binary
reproducibility purposes.
- using the store_value() function (which warrants
you don't save the same value twice and doesn't write
when you're using a read-only server) is enough.
(Bitbake rev: 000704a53470ab1ead840403b5531f22ebf1fd49)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test_checksum_value() to test whether
a (version, pkgarch, checksum, value) entry already
exists in the database.
This is used to protect the store_value() function from
an error when trying to store a duplicate entry in the database.
Also check whether the current database is open in read-only mode.
(Bitbake rev: b7f6c085a7cf8ac83695242a0299e2d5f7abc69a)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove a possible race condition by allowing a read-only
server to create the PR table anyway. This avoids a failure
if both a read-only and read-write server try to access
an empty database at the same time.
(Bitbake rev: b171caec5ebbe579bf4b8b2005930240ae5c8ce2)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <jpewhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sqlite3 can allow multiple processes to access the database
simultaneously, but it must be opened correctly. The key change is that
the database is no longer opened in "exclusive" mode (defaulting to
shared mode). In addition, the journal is set to "WAL" mode, as this is
the most efficient for dealing with simultaneous access between
different processes. In order to keep the database performance,
synchronous mode is set to "off". The WAL journal will protect against
incomplete transactions in any given client, however the database will
not be protected against unexpected power loss from the OS (which is a
fine trade off for performance, and also the same as the previous
implementation).
The use of a database cursor enabled to remove the _execute() wrapper.
The cursor automatically makes sure that the query happens in an atomic
transaction and commits when finished.
This also removes the need for a "dirty" flag for the database and
for explicit database syncing, which simplifies the code.
(Bitbake rev: 385833243c495dc68ec26a963136c1ced3f272d0)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduce a PRSERVER_UPSTREAM variable that makes the
local PR server connect to an "upstream" one.
This makes it possible to implement local fixes to an
upstream package (revision "x", in a way that gives the local
update priority (revision "x.y").
Update the calculation of the new revisions to support the
case when prior revisions are not integers, but have
an "x.y..." format."
Set the comments in the handle_get_pr() function in serv.py
for details about the calculation of the local revision.
This is done by going on supporting the "history" mode that
wasn't used so far (revisions can return to a previous historical value),
in addition to the default "no history" mode (revisions can never decrease).
Rather than storing the history mode in the database table
itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode
is now passed through the client requests. As a consequence, the
table name is now "PRMAIN", which is incompatible with what
was generated before, but avoids confusion if we kept the "PRMAIN_nohist"
name for both "history" and "no history" modes.
Update the server version to "2.0.0".
(Bitbake rev: 48857ec3e075791bd73d92747c609a0a4fda0e0c)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tests quotes around `` and $() expansions, nested and multiple
expansions, and that escaped quotes are treated as characters by the
parser.
(Bitbake rev: d98130cb4d500c495bc692c56dde3e019f36320a)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current shell substitution mechanism only works without quotes. For
example:
var1=$(cmd1 ...)
Will work and add `cmd1` to the correspondind `run.do_*` file.
However, although quite common, this syntax is not supported:
var1="$(cmd1 ...)"
This commit adds this feature by adding a step to process_words() to
check whether we are dealing with quotes first, and by iterating on
what's between them to detect new shell substitution candidates. These
candidates are tested and parsed like before in the next step. The
original `part` being part of the candidates means the syntax
var1=$(cmd1 ...) is still valid.
(Bitbake rev: f56e1a37b2ba1773ed308043d7eb073cc2e6c06e)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an invalid provider is requested, error out early rather than trying
to build partial runqueue data structures as the taskdep UI will have
exited after seeing the bad provider.
(Bitbake rev: a478087998cb794cc4e31189b3ce07973d3949bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.
We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.
This hopefully fixes [YOCTO #15318].
(Bitbake rev: d9e5d313c79500e3c70ab9c3239b6b2180194f67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python 3.10 support is only available in websockets 10.0 and later:
08d8011132
Update the version for this case. This avoids failures on Ubuntu 22.04.
(Bitbake rev: 0e4767c4a880408750e1a6855270c5a4eef8383d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allow the use of new siggen API
(Bitbake rev: e53503546990adeab67b6d044fcce59dc5a3f455)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Checks that the minimum version of the websockets module is present, and
if not raises an ImportError. This allows the user to get earlier
feedback if using websockets is going to succeed
(Bitbake rev: 330ea6914aad65dc8b34c986c44779820c392f03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handles ImportError when creating a hash equivalence to ping the server.
This notifies user earlier with a more precise error if websockets can't
be used, and also prevents passing a known bad upstream value to the
local server
(Bitbake rev: aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Report which task dependencies in BB_TASKDEPDATA are included in the
taskhash. This allows tasks to identify which tasks dependencies may
change without the task re-running. Knowing this information is
important for tasks that want to transfer information from dependencies
(such as SPDX)
(Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add return false to supports_checksum for svn fetcher which fhis
fixes MIRROR usage for svn uris. Also add a testcase.
[YOCTO #15473]
(Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f)
Signed-off-by: Kari Sivonen <kari.sivonen@live.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cwd of the task is also a cleandirs, you would see warnings from bitbake
about being unable to obtain cwd during the task execution. Tweak the code
to detect this and avoid the warnings.
(Bitbake rev: 6c7fd60c10955b0f23f64b25b5b4e154eb22a8f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #15417]
When a layer adds a new dependency after it was added to a conf, it can not be
removed w/o this dependency in the setup. Even the dependent layer can not be
added, as the tinfoil setup will fail.
Adapt --force to not perform the tinfoil at all, the use will be at own risk,
i.e. the added layers might not parse properly afterwards.
This is not merged into the force option with -F as it even changes the loading of
plugins from other layers and is hence even more invasive as force. Instead
force can now be speciefied multiple times and is counted.
(Bitbake rev: 541fa7f582133949563e65f2d43c4b16e873e5c1)
Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cooker attempts to connect to the upstream hash equivalent server to
warn the user early if it is misconfigured. However, this was making the
assumption that it was a raw TCP connection and failed when attempting
to use a websocket upstream server. Fix this by creating an hash client
and using the ping API to check the server instead of using a raw
socket.
(Bitbake rev: 5e84c13a6c594ed34c341849806657ddda206714)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Careful reading of the code can contrive cases where poorly timed
ConnectionError's will result in the client mode being incorrectly reset
to MODE_NORMAL when it should actual be a stream mode for the current
command. Fix this by no longer attempting to restore the mode when the
connection is setup. Instead, attempt to set the stream mode inside the
send wrapper for the stream data, which means that it should always end
up in the correct mode before continuing.
Also, factor out the transition to normal mode into a invoke() override
so it doesn't need to be specified over and over again.
(Bitbake rev: 0cd276fd98eeca463518d4a42675fffb18d6b3de)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the bitbake hash server supports SSL connections, we need to
capture a few environment variables which can affect the ability to
connect via SSL. Note that the variables are only put in place to affect
the environment while actually invoking the server
[RP: Tweak to use BB_ORIGENV as well]
[RP: Tweak to handle os.environ restore correctly]
(Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A function accidentally defined as:
somefunction() {
:
}
which is unclosed due to the space at the end, would currently silently
cause breakage. Have the parser throw and error for this.
[YOCTO #15470]
(Bitbake rev: a7dce72da6be626734486808f1b731247697e638)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
