The previous syntax of the example was clearly incorrect
The example and description were also ambiguous, one could think that
it was "bitbake recipe" instead of "bitbake <recipename>"
(Bitbake rev: 9d6664bbb68fac9bb4fbcbe0b35beb0a022a5a1f)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The examples and tests use non-standard override names, convert these to
the new syntax by hand.
(Bitbake rev: a6c40eca1146c0160da7e4e0bd7ac52fef2029e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This runs the overrides conversion script in OE-Core over the bitbake code
base including the docs. A handful of things were excluded in toaster
and for the Changelog file.
(Bitbake rev: 47f8d3b24fd52381bf3b41e2f55a53e57841344c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change updates the datastore to use the new override syntax using
colons instead of underscores exclusively. It is expected layers would
have to be converted to work with bitbake after this change.
Supporting mixed syntax isn't possible, it is only feasible to have
one internal representation of overrides.
Whilst we can't warn for every possible override that may be set in the
old format, show errors for _append/_prepend/_remove since those should
never be present.
(Bitbake rev: 7dcf317cc141dc980634f8c18bfa84f83e57206a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that ":" is a valid character in variable key names, it needs to be
allowed by the variable expansion code too, to match.
(Bitbake rev: 019251649a38754d5877759d13b664e28dea77de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing an anonymous python function, bitbake generates a name for
the function based on the full path to the file in which it was found.
As not all characters which are valid in file paths are valid in Python
function names we have a translation table. However, this translation
table was missing an entry for '~'.
(Bitbake rev: b201c0b284e25c20685d9d206797c4cc650eeca0)
Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the SIGTERM signal is sent to an asyncrpc server before it has
installed the SIGTERM handler in the main loop, it may miss the signal
which will can cause the calling process to wait forever on the join().
To resolve this, the calling process should mask of SIGTERM before
forking the server process and the server should unmask the signal only
after the handler is installed. To simplify the usage of the server, an
new helper function called serve_as_process() is added to do this
automatically and correctly.
Thanks: Scott Murray <scott.murray@konsulko.com> for helping debug
(Bitbake rev: ef2865efa98ad20823267364f2159d8d8c931400)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is an issue with the format of the reply given by the server then we
should print this reply line in the error message. Printing the message which
the client sent doesn't illuminate anything here.
(Bitbake rev: bd8f8d7b055da15cd7bdd0b383061852a0f54cb7)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a variable has a glossary entry and some rST files write about those
variables, it's better to point to the glossary entry instead of just
highlighting it by surrounding it with two tick quotes.
The script that is used to do the replacement of ``FOO`` by :term:`FOO`
is the following Python code:
import re
from pathlib import Path
from runpy import run_module
import contextlib
import io
import sys
re_term = re.compile(r'variables.html#term-([a-zA-Z_0-9]*)')
terms = []
new_terms = set()
with contextlib.redirect_stdout(io.StringIO()) as f:
run_module('sphinx.ext.intersphinx', run_name='__main__')
objects = f.getvalue()
match = re_term.search(objects)
while match:
if match.group(1):
terms.append(match.group(1))
match = re_term.search(objects, match.end())
for rst in Path('.').rglob('*.rst'):
with open(rst, 'r') as f:
content = "".join(f.readlines())
for term in terms:
content = re.sub(r'``({})``(?!.*\s+[~=-]{{{:d},}})'.format(term, len(term)), r':term:`\1`', content)
with open(rst, 'w') as f:
f.write(content)
This script takes one argument as input: an objects.inv which can be
gotten from doc/_build/html/objetcs.inv after running `make html`.
Note that this excludes from replacement terms that appear in section
titles as it requires refs to be changed too. This can be automated too
if need be but right now it looks a bit confusing to have an anchor link
(for sections) also have a term/reference link in it. I am not sure this
is desired today.
(Bitbake rev: aba88f40c47133ed9bc999e0298aca3bc8490912)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Even though, care should be taken to have the terms in the glossary
sources alphabetically ordered, it is possible some terms might be in
the wrong place.
This makes sure that whatever the order of terms in the glossary
sources, the generated medium is correctly sorted.
(Bitbake rev: 98809ebc6ad51f0a94cedccfaff9c11d3744dc0d)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-W is for turning warnings into errors, --keep-going to make sure the
whole documentation is built and all warnings will be printed before
failing.
Since there's currently no warning, it's time to introduce it!
(Bitbake rev: 6b0b3d707f662ca7b1d0de99fe032e4f35867d10)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the client process never sends cooker data, the server timeout will
be 0.0, not None. This will prevent the server from exiting, as it is
waiting for a new client. In particular, the client will disconnect with
a bad "INHERIT" line, such as:
INHERIT += "this-class-does-not-exist"
Instead of checking explicitly for None, check for a false value, which
means either 0.0 or None.
(Bitbake rev: 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a library that implements file-like objects (similar to
gzip.GzipFile) that can stream to arbitrary compression programs. This
is utilized to implement a LZ4 and zstd compression API.
(Bitbake rev: 61c3acd058ea018696bd284b3922d0b458838d05)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is becomming increasingly clear we need to find a way to show what
is/is not an override in our syntax. We need to do this in a way which
is clear to users, readable and in a way we can transition to.
The most effective way I've found to this is to use the ":" charater
to directly replace "_" where an override is being specified. This
includes "append", "prepend" and "remove" which are effectively special
override directives.
This patch simply adds the character to the parser so bitbake accepts
the value but maps it back to "_" internally so there is no behaviour
change.
This change is simple enough it could potentially be backported to older
version of bitbake meaning layers using the new syntax/markup could
work with older releases. Even if other no other changes are accepted
at this time and we don't backport, it does set us on a path where at
some point in future we could
require a more explict syntax.
I've tested this patch by converting oe-core/meta-yocto to the new
syntax for overrides (9000+ changes) and then seeing that builds
continue to work with this patch.
(Bitbake rev: 0dbbb4547cb2570d2ce607e9a53459df3c0ac284)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously access to AWS S3 was expected to be preconfigured and
credentials to be stored in ~/.aws/credentials. With this change
one can use Bitbake s3 fetcher without AWS credentials stored
permanently as above, just with them exported as the following
environment variables:
- AWS_ACCESS_KEY_ID,
- AWS_SECRET_ACCESS_KEY.
- AWS_DEFAULT_REGION.
(Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48)
Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It looks like git gc can interrupts the package creation when
BB_GENERATE_MIRRORS_TARBALL is in use.
Log excerpts:
tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output:
tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it
tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
(Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877)
Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the BitBake glossary.
Remove an erroneous statement probably coming from the introduction
to the YP/OE glossary.
(Bitbake rev: 398a1686176c695d103591089a36e25173f9fd6e)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently setting those flags to even the empty string "" causes it to be set,
which is contrary to the documentation. In a future version of BitBake, we'd
like to change the behavior so that setting the flag to "" does not set it.
This will allow conditionally setting noexec, using variable expansion or
inline Python.
I found no places in poky or meta-openembedded where this warning would trigger.
[YOCTO #13808]
(Bitbake rev: 1e7655c4f765ba7b4791c4cca048a69bf8d9c93d)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable was removed in July 2019 with commit
5deaa5df730a "runqueue: Drop unused BB_SETSCENE_VERIFY_FUNCTION2".
There's no replacement, so let's update the documentation to not mention
this variable anymore.
This was found by running:
git grep -hoP '^ :term:`\K\w+(?=`)' doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst | \
xargs -I{} sh -c \
"grep -Rq --include='*.py' --include='*.conf' --include='*.bbclass' --include='*.bb' --include='*.bbappend' --include='*.inc' \
-w -E {}'_*(_[a-z]+[0-9a-z]*[a-z]+)*' || \
echo {}"
(Bitbake rev: c2c0f5126c7c784bfd7a08f127e161a58c6b5d12)
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This method is needed to support startup of the prservice. As it is so
generic we can add it to the common asyncrpc module.
(Bitbake rev: 25ccd697ea76f66b813be2296866b2d3405b079c)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No error was being reported when the hash string was set to empty.
For example: SRC_URI[md5sum] = ""
On a related note (not a bug):
Because whitespace in the string will result in a checksum mismatch, the error
message was updated to make it a little clearer why the error was thrown.
For example: SRC_URI[md5sum] = " " or
SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1"
Now creates a message like this:
File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5
checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected
[YOCTO #14232]
(Bitbake rev: a13510d0028e234ea2f4744b0d0c38558395c70f)
Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen cases where an OOM error causes bitbake server to hang:
9171 02:21:09.127810 Command Completed
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in <module>
bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer
server.run()
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run
ret = self.main()
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main
ready = self.idle_commands(.1, fds)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands
bb.event.fire(heartbeat, self.cooker.data)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
fire_class_handlers(event, d)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
ret = handler(event)
File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats
write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval")
File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data
output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8')
File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
We need to wrap the calls in the same high level wrapper as idle function calls
and trigger an exit upon an unhandled exception.
(Bitbake rev: 74042b5b89d5a170013fc1a327ce3a6530fbf7d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've talked about having this for long enough. Add a command which queries a single
variable value with history. This saves "bitbake -e | grep" and avoids the
various pitfalls that has.
It also provides a neat example of using tinfoil to make such a query.
Parameters to limit the output to just the value, to limit to a variable flag
and to not expand the output are provided.
[YOCTO #10748]
(Bitbake rev: 4c1881b620e885f55d7772f8626b8a76c2828333)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
svn was printing a message when encountering HTTP redirects.
This confused the revision parser.
(Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146)
Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The prserver process is cleanly separated from the main bitbake process
so requests can be handled in the main thread. This removes the need for
a request queue and a separate request handling thread.
(Bitbake rev: 6b09415bed6b5e7c12aaf39b677d9ef72844e233)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code to start the prservice process as a daemon is extracted out of
the PRServer class and simplified. This makes the PRServer class easier
to modernise as it no longer needs to worry about process management.
(Bitbake rev: 39c7c158c52157b18f5ccbbd673e3298e6402f52)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can use the modern multiprocessing support in Python instead of
manually using fork to start the prserver process. To do this we need
to set up the signal handlers for the prserver process in the
work_forever function (which is now used as the main function for this
process).
The old code to start the prserver process using fork is not removed in
this commit as it is tightly intertwined with the daemonization code
which will be refactored in a following commit.
(Bitbake rev: b3da56240c0f92efab1c0b293738c35c0f1ee6ab)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename can fail for example an incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than trying to fix every call site, add a wrapper in bb.utils
for renames. We can then handle cross device failures and
fall back to shutil.move. The reason os.rename is still used is
because shutil.move is too slow for speed sensitive sections of code.
[YOCTO #14301]
(Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b)
Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig
build), we need to ensure that the hash change propagates through to all the tasks
else the build will run multiple copies of the task, sometimes with oddly differing
results as the outhashes of native tasks built in differing locations can confuse
things.
(Bitbake rev: 2db571324f755edc4981deecbcfdf0aaa5a97627)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were testing the validity of deferred tasks setscene status "up front" which
is very unlikely to succeed and leads to cache invalidation issues. With the
change to rebuild the deferred task list, this status becomes out of sync. The
result was tasks being executed when they should not have been leading to extra
work for the build unnecessarily.
Instead, don't process validity status for deferred tasks and assume their
data will become available. If it doesn't, this will now result in a build
error as the setscene task will fail and the main task will run instead.
In theory we could try and track the state changes in the deferred list and
re-test validity then but I'm not sure it is worth the effort when the other
code path and errors in setscene tasks will give a pretty good idea of what
is happening anyway.
(Bitbake rev: edcafac13b3b241b6687419e59018d21811507a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "-l" command line options to enable specific logging domains wasn't
working with the switch to structured logging because they were only
being used to set the legacy logging domains. Fix this by implementing
the logic to parse the user options into the logging configuration.
(Bitbake rev: 005fc7a8c588d0b0bca382469645cbf481ad8e30)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds progress support for fetchers from S3.
(Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
S3 commands need to handle different CR only line endings, update the handler
to cope with this.
(Bitbake rev: 3f7b9c1b429a4c68240e80832a8ef93ee210e5ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function abstracts the setup of a PR service client connection so
that openembedded-core doesn't need to be updated any time the details
are changed.
(Bitbake rev: d892287b31f81b075983ba500be265f75b53df64)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake no longer supports Python 2 so this version check is obsolete.
(Bitbake rev: 45eb6c6e124e507012df9c288f1fbde0e7899e5d)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The asyncrpc module can now be used to provide the json & asyncio based
RPC system used by hashserv.
(Bitbake rev: 5afb9586b0a4a23a05efb0e8ff4a97262631ae4a)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hashserv module implements a flexible RPC mechanism based on sending
json formatted messages over unix or tcp sockets and uses Python's
asyncio features to build an efficient message loop on both the client
and server side. Much of this implementation is not specific to the
hash equivalency service and can be extracted into a new module for
easy re-use elsewhere in bitbake.
(Bitbake rev: 4105ffd967fa86154ad67366aaf0f898abf78d14)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Python built-in ConnectionError type can be used instead of a custom
HashConnectionError type. This will make code refactoring simpler.
(Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
