I don't know why there is hardcoded knowledge of S in the fetcher but there
shouldn't be and the OE unpack changes highlight this doing things it
shouldn't.
Drop the S reference and use rootdir which is the only place it should
be touching.
(Bitbake rev: 84f102954e10a3390fca9c26d5c3c639e952a2c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplify the importone() hook:
- to make it independent from the "history" mode which is
client specific.
- remove the "history" parameter
- we want all values to be imported for binary
reproducibility purposes.
- using the store_value() function (which warrants
you don't save the same value twice and doesn't write
when you're using a read-only server) is enough.
(Bitbake rev: 000704a53470ab1ead840403b5531f22ebf1fd49)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test_checksum_value() to test whether
a (version, pkgarch, checksum, value) entry already
exists in the database.
This is used to protect the store_value() function from
an error when trying to store a duplicate entry in the database.
Also check whether the current database is open in read-only mode.
(Bitbake rev: b7f6c085a7cf8ac83695242a0299e2d5f7abc69a)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove a possible race condition by allowing a read-only
server to create the PR table anyway. This avoids a failure
if both a read-only and read-write server try to access
an empty database at the same time.
(Bitbake rev: b171caec5ebbe579bf4b8b2005930240ae5c8ce2)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <jpewhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sqlite3 can allow multiple processes to access the database
simultaneously, but it must be opened correctly. The key change is that
the database is no longer opened in "exclusive" mode (defaulting to
shared mode). In addition, the journal is set to "WAL" mode, as this is
the most efficient for dealing with simultaneous access between
different processes. In order to keep the database performance,
synchronous mode is set to "off". The WAL journal will protect against
incomplete transactions in any given client, however the database will
not be protected against unexpected power loss from the OS (which is a
fine trade off for performance, and also the same as the previous
implementation).
The use of a database cursor enabled to remove the _execute() wrapper.
The cursor automatically makes sure that the query happens in an atomic
transaction and commits when finished.
This also removes the need for a "dirty" flag for the database and
for explicit database syncing, which simplifies the code.
(Bitbake rev: 385833243c495dc68ec26a963136c1ced3f272d0)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduce a PRSERVER_UPSTREAM variable that makes the
local PR server connect to an "upstream" one.
This makes it possible to implement local fixes to an
upstream package (revision "x", in a way that gives the local
update priority (revision "x.y").
Update the calculation of the new revisions to support the
case when prior revisions are not integers, but have
an "x.y..." format."
Set the comments in the handle_get_pr() function in serv.py
for details about the calculation of the local revision.
This is done by going on supporting the "history" mode that
wasn't used so far (revisions can return to a previous historical value),
in addition to the default "no history" mode (revisions can never decrease).
Rather than storing the history mode in the database table
itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode
is now passed through the client requests. As a consequence, the
table name is now "PRMAIN", which is incompatible with what
was generated before, but avoids confusion if we kept the "PRMAIN_nohist"
name for both "history" and "no history" modes.
Update the server version to "2.0.0".
(Bitbake rev: 48857ec3e075791bd73d92747c609a0a4fda0e0c)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tests quotes around `` and $() expansions, nested and multiple
expansions, and that escaped quotes are treated as characters by the
parser.
(Bitbake rev: d98130cb4d500c495bc692c56dde3e019f36320a)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current shell substitution mechanism only works without quotes. For
example:
var1=$(cmd1 ...)
Will work and add `cmd1` to the correspondind `run.do_*` file.
However, although quite common, this syntax is not supported:
var1="$(cmd1 ...)"
This commit adds this feature by adding a step to process_words() to
check whether we are dealing with quotes first, and by iterating on
what's between them to detect new shell substitution candidates. These
candidates are tested and parsed like before in the next step. The
original `part` being part of the candidates means the syntax
var1=$(cmd1 ...) is still valid.
(Bitbake rev: f56e1a37b2ba1773ed308043d7eb073cc2e6c06e)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an invalid provider is requested, error out early rather than trying
to build partial runqueue data structures as the taskdep UI will have
exited after seeing the bad provider.
(Bitbake rev: a478087998cb794cc4e31189b3ce07973d3949bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.
We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.
This hopefully fixes [YOCTO #15318].
(Bitbake rev: d9e5d313c79500e3c70ab9c3239b6b2180194f67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python 3.10 support is only available in websockets 10.0 and later:
08d8011132
Update the version for this case. This avoids failures on Ubuntu 22.04.
(Bitbake rev: 0e4767c4a880408750e1a6855270c5a4eef8383d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allow the use of new siggen API
(Bitbake rev: e53503546990adeab67b6d044fcce59dc5a3f455)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Checks that the minimum version of the websockets module is present, and
if not raises an ImportError. This allows the user to get earlier
feedback if using websockets is going to succeed
(Bitbake rev: 330ea6914aad65dc8b34c986c44779820c392f03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handles ImportError when creating a hash equivalence to ping the server.
This notifies user earlier with a more precise error if websockets can't
be used, and also prevents passing a known bad upstream value to the
local server
(Bitbake rev: aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Report which task dependencies in BB_TASKDEPDATA are included in the
taskhash. This allows tasks to identify which tasks dependencies may
change without the task re-running. Knowing this information is
important for tasks that want to transfer information from dependencies
(such as SPDX)
(Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add return false to supports_checksum for svn fetcher which fhis
fixes MIRROR usage for svn uris. Also add a testcase.
[YOCTO #15473]
(Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f)
Signed-off-by: Kari Sivonen <kari.sivonen@live.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cwd of the task is also a cleandirs, you would see warnings from bitbake
about being unable to obtain cwd during the task execution. Tweak the code
to detect this and avoid the warnings.
(Bitbake rev: 6c7fd60c10955b0f23f64b25b5b4e154eb22a8f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #15417]
When a layer adds a new dependency after it was added to a conf, it can not be
removed w/o this dependency in the setup. Even the dependent layer can not be
added, as the tinfoil setup will fail.
Adapt --force to not perform the tinfoil at all, the use will be at own risk,
i.e. the added layers might not parse properly afterwards.
This is not merged into the force option with -F as it even changes the loading of
plugins from other layers and is hence even more invasive as force. Instead
force can now be speciefied multiple times and is counted.
(Bitbake rev: 541fa7f582133949563e65f2d43c4b16e873e5c1)
Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cooker attempts to connect to the upstream hash equivalent server to
warn the user early if it is misconfigured. However, this was making the
assumption that it was a raw TCP connection and failed when attempting
to use a websocket upstream server. Fix this by creating an hash client
and using the ping API to check the server instead of using a raw
socket.
(Bitbake rev: 5e84c13a6c594ed34c341849806657ddda206714)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Careful reading of the code can contrive cases where poorly timed
ConnectionError's will result in the client mode being incorrectly reset
to MODE_NORMAL when it should actual be a stream mode for the current
command. Fix this by no longer attempting to restore the mode when the
connection is setup. Instead, attempt to set the stream mode inside the
send wrapper for the stream data, which means that it should always end
up in the correct mode before continuing.
Also, factor out the transition to normal mode into a invoke() override
so it doesn't need to be specified over and over again.
(Bitbake rev: 0cd276fd98eeca463518d4a42675fffb18d6b3de)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the bitbake hash server supports SSL connections, we need to
capture a few environment variables which can affect the ability to
connect via SSL. Note that the variables are only put in place to affect
the environment while actually invoking the server
[RP: Tweak to use BB_ORIGENV as well]
[RP: Tweak to handle os.environ restore correctly]
(Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A function accidentally defined as:
somefunction() {
:
}
which is unclosed due to the space at the end, would currently silently
cause breakage. Have the parser throw and error for this.
[YOCTO #15470]
(Bitbake rev: a7dce72da6be626734486808f1b731247697e638)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Useful for connecting a PR server to an upstream one
- "test-package" checks whether the specified package
version and arch is known in the database.
- "test-pr" checks a specified output hash is found in the database.
Otherwise it returns 'None' instead of a new value.
- "max-package-pr" returns the highest PR number for
(version, arch) entries in the database, and None if not found
Add new DB functions supporting the above, plus test_value()
which tells whether a given value is available for the specified
package and architecture.
(Bitbake rev: 0f1474a30f741b760ca81c19dd1d8f3bd5647251)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In both the PRServerClient and PRClient objects.
This aligns with what is done in hashserv/server.py and makes it
possible to benefit from possible specializations of the logger
in the corresponding super classes, instead of using
always the global logger.
(Bitbake rev: 5fc6d2b1a5db617e16c1eb9fbd25e821237611d8)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv.py: simplify the PRServerClient() interface by passing the
server object instead of multiple arguments, and then retrieving
the data through this object.
This replicates what is done for ServerClient() in hashserv/server.py
(Bitbake rev: d3be073218feb4d6e68a751832da4936da485dbc)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fedora 40 introduces wget2 as a drop-in replacement for wget. This
rewrite does not currently have support for FTP. This causes
the wget fetcher to fail complaining about an unrecognized option.
Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem. It also gives us
an opportunity to build the older wget as a host tool.
(Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for hashserver credentials to be specified in the
SignatureGenerator
(Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Git uses a lock file to prevent concurrent modifications to the global
config, so if unpack tasks for different recipes try to run "git lfs
install" simultaneously the operation can fail:
error: could not lock config file /home/build/.gitconfig: File exists exit status 255
Run `git lfs install --force` to reset Git configuration.
Adding "--local" sets the smudge and clean filters in the local
repository's config instead of modifying the user's global config.
(Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7)
Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running in a cgroup which is limited to a subset of cpus (via
cpuset.cpus), cpu_count() should return the number of cpus that can be
used instead of the number of cpus the system has.
This also aligns the semantics with nproc.
(Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a command is run with a non-null log, errors are only output to the
log and are not returned in the exception. In that case direct users to
that logfile instead of telling the command had no output.
(Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a)
Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.
For example: http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:
File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
self.query = self._param_str_split(urlp.query, "&")
File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)
In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.
The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.
Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().
(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Red text on a black background can make it difficult for people with
visual impairments to read the text of error messages. Respect the
presence of a non-empty NO_COLOR environment variable as an indication
the user doesn't want colorization to be automatically enabled.
See: https://no-color.org/
(Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85)
Signed-off-by: Peter A. Bigot <pab@pabigot.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FIXES [YOCTO #15404]
When using git fetcher on a repo with parentheses in its URL, the
invocation of the git clone command will fail. The clone directory
is not quoted thus the shell will return an error and won't execute
the command.
(Bitbake rev: b5624ee564)
Cc: Philippe Rivest <privest@genetec.com>
(Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6)
Signed-off-by: Philippe Rivest <technophil98@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Limit the split to key & value (2 items) instead of the n items one
can get if there are '=' characters in the value.
Fixes [YOCTO #15447]
(Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb)
Signed-off-by: david d zuhn <david.zuhn@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
