We've talked about having this for long enough. Add a command which queries a single
variable value with history. This saves "bitbake -e | grep" and avoids the
various pitfalls that has.
It also provides a neat example of using tinfoil to make such a query.
Parameters to limit the output to just the value, to limit to a variable flag
and to not expand the output are provided.
[YOCTO #10748]
(Bitbake rev: 4c1881b620e885f55d7772f8626b8a76c2828333)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the previous patch this code is now pointless as we'd have hit a TypeError
before now.
(Bitbake rev: 6301a99055c79d89b715f72182cd0ef1b781b89a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer. There's some type confusion here:
bitbake-server is called with a string representation of the timeout,
which may be None. If the timeout is not set, pass 0 instead of None.
Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.
[ YOCTO #14350 ]
(Bitbake rev: c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows metadata to rely upon BB_DEFAULT_UMASK.
(Bitbake rev: 969ac64adab236ce2d5196bcc294005a497913ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently each task has to have a umask specified individually. This
is leading to determinism issues since it is easy to miss specifying
this for an extra task.
Add support for specifing the default task umask globally which
simplifies the problem.
(Bitbake rev: 3e664599fd54a8a37ce587022fcbce5ca26f2ed3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Short form arguments are added for convenience.
(Bitbake rev: 921199a4923ce383b27e23c9b7e34eb785c8bae3)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hashserv server already implements support for pulling hash data
from another "upstream" server. Add the -u/--upstream argument to the
bitbake-hashserv app to expose this functionality to users.
(Bitbake rev: 8de510f1de35e581bcd5858edf23619c6a4cf923)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The -r/--readonly argument is added to the bitbake-hashserv app. If this
argument is given then clients may only perform read operations against
the server. The read-only mode is implemented by simply not installing
handlers for write operations, this keeps the permission model simple
and reduces the risk of accidentally allowing write operations.
As a sqlite database can be safely opened by multiple processes in
parallel, it's possible to start two hashserv instances against a single
database if you wish to export both a read-only port and a read-write
port.
(Bitbake rev: 492bb02eb0e071c792407ac3113f92492da1a9cc)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing occasional issues where builds fail as events were written from the
worker children in the form <event>partial data<event>full event</event>.
This causes failures as bitbake server can't parse that and exits. This could
be due to short writes to the worker event pipe which we weren't checking. Check
this and loop accordingly. Also add some asserts to detect other potential causes.
Thanks to Joshua Watt for help in spotting the issue.
[YOCTO #14181]
(Bitbake rev: a9451746a4bd7ccedf4c72cd03ad4ff0ab0143aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect function was previously removed from the hashserv Client
class but the bitbake-hashclient app was not updated. The client is
connected during hashserv.create_client() anyway so not separate connect
call is needed.
(Bitbake rev: 99bdb236bceeffa0083a0fa529280b217c1d310d)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 4fca9a07f2d6b0544977112672b786982d7bb8f2.
The default python handler is to ignore SIGPIPE errors yet by adding
this line, we see:
2020-09-23 22:10:20,325 - oe-selftest - INFO - recipetool.RecipetoolTests.test_recipetool_appendfile_subdir (subunit.RemotedTestCase)
2020-09-23 22:10:20,325 - oe-selftest - INFO - ... FAIL
2020-09-23 22:10:20,325 - oe-selftest - INFO - 11: 30/44 227/415 (14.85s) (recipetool.RecipetoolTests.test_recipetool_appendfile_subdir)
2020-09-23 22:10:20,325 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 226, in test_recipetool_appendfile_subdir
_, output = self._try_recipetool_appendfile('selftest-recipetool-appendfile', '/usr/share/selftest-replaceme-subdir', self.testfile, '', expectedlines, ['testfile'])
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 85, in _try_recipetool_appendfile
return self._try_recipetool_appendcmd(cmd, testrecipe, expectedfiles, expectedlines)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 49, in _try_recipetool_appendcmd
bbappendfile = self._check_bbappend(testrecipe, recipefile, self.templayerdir)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/devtool.py", line 186, in _check_bbappend
result = runCmd('bitbake-layers show-appends', cwd=self.builddir)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake-layers show-appends' returned non-zero exit status 141:
NOTE: Reconnecting to bitbake server...
where status 141 is SIGPIPE.
We expect SIGPIPE not to be raised like this. I tried the commands in the original
commit to pipe through less with the change reverted and I don't see the error
shown. I therefore believe this is safe to revert.
(Bitbake rev: 94a18f1f2e51dd5d3928e6e0c17f2c19c8b5db49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While discussing with Richard we thought these might help document
and safeguard the basic requirements of clean_basepath.
A 'bonus' performance testcase is added but commented out since its
runtime is long and test machine specific. It is intended for developers
to test before and after their changes to the target function as a due
diligence verification.
(Bitbake rev: ee41549f26952d5f7af19a9b3d8a8b969866e2ef)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to have a new python process forked off an original doesn't work
out well and ends up having race issues. To avoid this, exec() a new
bitbake server process. This starts with a fresh python interpreter
and resolves various atexit and other multiprocessing issues once
and for all.
(Bitbake rev: 9501dd6fdd7a7c25cbfa4464cf881fcf8c049ce2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
(Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script took a documentation.conf and generated an HTML index of
each of the variables. The data in the yocto-docs manuals is now far
superior to this. The script has bitrotted and doesn't run, hasn't for
some time and wasn't reported until recently.
Easiest is just to remove it as anyone would reimplement it quite
differently now.
The referenced bug does have a patch which cleans up the code but I
don't see value in keeping this around.
[YOCTO #13881]
(Bitbake rev: 0829af7090065c429ba558d06423c2e7e7e05e90)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Directly support the various 'manage' commands from the Toaster
executable, so that users do not have to manually set up the required
environment and paths.
Examples:
$ . toaster manage createsuperuser
$ . toaster manage lsupdates
[YOCTO #13170]
(Bitbake rev: fd844e55bb885a51fe5ef8da1f625b34e646cf5f)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding the default log filter here is unnecessary because there are no
defined logging domains when it is called, which means it does no actual
filtering.
(Bitbake rev: dcdb8f2c14f09ce34d0a1facc33a441570912c05)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passes around the actual logging level as the default log level variable
instead of the debug count. This makes it easier to deal with logging
levels since the conversion from debug count and verbose flag only has
to occur once when logging is initialized and after that actual log
levels can be used
(Bitbake rev: 41bd155faf7f65cb0727fcce972715769b26ca89)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake-worker child on the SIGTERM signal handling send the SIGTERM to all
processes in it's process group. In cases when the bitbake-worker child got
SIGTERM after registering own SIGTERM handler and before the os.setsid() call
it can send SIGTERM to unwanted processes.
In the worst case during SIGTERM processing the bitbake-worker child can be in
the group of the process that started BitBake itself. As a result it can kill
processes that not related to BitBake at all.
(Bitbake rev: b97b1ef0b1b00848a4a44b34eca123ccf33188f4)
Signed-off-by: Ivan Efimov <i.efimov@inango-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows metadata to depend on SignatureGeneratorUniHashMixIn which was recently added.
(Bitbake rev: f0f814407fdd2fffa7071c36c011b489bfcd53da)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
(Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful, particularly in the local developer model of usage, for
bitbake to start and stop a hash equivalence server on local port,
rather than relying on one being started by the user before the build.
The new BB_HASHSERVE variable supports this.
The database handling is moved internally into the hashserv code so that
different threads/processes can be used for the server without errors.
(Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a compelling usecase for tasks being able to notify runqueue
that their "unihash" has changed. When this is recieved, the hashes of
all subsequent tasks should be recomputed and their new hashes checked
against existing setscene validity. Any newly available setscene tasks
should then be executed.
Making this work effectively needs several pieces. An event is added
which the cooker listen for. If a new hash becomes available it can
send an event to notify of this.
When such an event is seen, hash recomputations are made. A setscene
task can't be run until all the tasks it "covers" are stopped. The
notion of "holdoff" tasks is therefore added, these are removed from
the buildable list with the assumption that some setscene task will
run and cover them.
The workers need to be notified when taskhashes change to update their
own internal siggen data stores. A new worker command is added to do this
which will affect all newly spawned worker processes from that worker.
An example workflow which tests this code is:
Configuration:
BB_SIGNATURE_HANDLER = "OEEquivHash"
SSTATE_HASHEQUIV_SERVER = "http://localhost:8686"
$ bitbake-hashserv &
$ bitbake automake-native
$ bitbake autoconf-native automake-native -c clean
$ bitbake m4-native -c install -f
$ bitbake automake-native
with the test being whether automake-native is installed from sstate.
(Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need some tests for runqueue, its been something which has been hard to test
for a long time. Add some dummy metadata to allow this, mirroring the OE
structure in spirit.
(Bitbake rev: 37564d7440c5d7aa05ec537f3b79026b1c83bb68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It used 2 spaces as indent which wasn't clear enough, and might cause
confusions, people might think it was in wrong format.
Fixed:
$ bitbake bc-native -ccleansstate -Snone
$ bitbake bc-native -ccleansstate -Snone
$ bitbake-diffsigs tmp/stamps/x86_64-linux/bc-native/1.07.1-r0.do_cleansstate.sigdata.*
* Before:
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
* Now
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
(Bitbake rev: 5127a8d8e6d53f5f43a6ada7fd09b6b0c24ae989)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements a reference implementation of the hash equivalence server.
This server has minimal dependencies (and no dependencies outside of the
standard Python library), and implements the minimum required to be a
conforming hash equivalence server.
[YOCTO #13030]
(Bitbake rev: 1bb2ad0b44b94ee04870bf3f7dac4e663bed6e4d)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The unique hash is now passed to the task in the BB_UNIHASH variable
[YOCTO #13030]
(Bitbake rev: aab80b099f6f259e4b57cba2c26dd385d07c5947)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Follow dependent hash changes recursively also when specifying two
signature files explicitly. Previously this was only done when using the
--task option.
(Bitbake rev: 353f0f3c77c3cdd75c1be2a565234a5e53dba3ef)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The functionalities of bitbake-diffsigs and bitbake-dumpsig are so
similar that they can be merged into one. Add an option --dump to make
bitbake-diffsigs dump the last signature data instead of comparing it.
Keep bitbake-dumpsig as a symbolic link to bitbake-diffsigs. When it is
called as bitbake-dumpsig, it behaves as if --dump was specified.
Also make -D the short option for --debug again (the way it used to be,
and still was for bitbake-dumpsig), so that -d can be used as the short
option for --dump.
(Bitbake rev: 3635b829e4eb940ada2b52bfb5b5e5be93a3b0aa)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a test suite for testing the persistent data cache
[YOCTO #13030]
(Bitbake rev: 96a4155049e834af17069d981cc2215e50d18c1a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.
[YOCTO #13030]
(Bitbake rev: 1e86d8c1bec7ea5d016a5ad2097f999362e29033)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updates for YOCTO #12891 allowed a user to have a directory
structure different to that of yocto (bitbake isn't inside
oe-core) whereas the definition of OE_ROOT in the main toaster
binary still assumes the same while checking for .templateconf
and hence we see an error on the cmdline in such cases:
bash: <repo-path>/bitbake/bin/../../.templateconf: No such file or directory
The change here now allows the user to provide OE_ROOT through
the environment in such cases and otherwise defaults to the older
mechanism to fix this issue.
(Bitbake rev: 9cc6fe4be797e106899d2448797a4b3af8eace4e)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
