This patch fixes a handful of remaining warnings reported by Sphinx.
(Bitbake rev: 35fdc18549607e195d424f554e8d4b74e87fd245)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some notes with links, the links were lost during the pandoc
conversion.
(Bitbake rev: e11d2dd1adbcb35b45d6d0f3a47f8d06caab0b49)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we converted the list of variables into a Sphinx glossary, the
automatic conversion from Pandoc does not produce proper links. We fix
them up using a Python regexp.
Similarly some http links were not converted correctly, and can also
be fixed up with a regexp.
This patch was generated by running the following regexp:
line = re.sub("` <(https?://.*)>`__",
"\\1",
line)
line = re.sub("`+(\w+)`* <#var-bb-\\1>`__",
":term:`\\1`",
line)
(Bitbake rev: 57300955beb200c81762880f44f1a499451b7432)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is possible to override CSS settings from the theme, by providing
custom snippets of CSS stylesheet. Support for that is added in
conf.py file.
Most of the CSS customization is inherited from the DocBook CSS style.
(Bitbake rev: fa304c0129538d7a0b7a3922cd1fde2e242d6cab)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To install this additional theme:
pip3 install sphinx_rtd_theme
(Bitbake rev: 29af1cd2a4f379096f5dd6f4cacc0a9c9f211e7c)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sphinx has a glossary directive. From the documentation:
This directive must contain a reST definition list with terms and
definitions. The definitions will then be referencable with the 'term'
role.
So anywhere in *any* manual, we can do :term:`VAR` to refer to an item
from the glossary, and create a link.
An HTML anchor is created for each term in the glossary, and can be
accessed as:
<link>/ref-variables.html#term-<NAME>
To convert to a glossary, we needed proper indentation (e.g. added 3
spaces to each line)
(Bitbake rev: e8359fd85ce0358019e2a32b4c47ba76613f48f0)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is autogenerated pandoc to generate an inital set
of reST files based on DocBook XML files.
A .rst file is generated for each .xml files in all manuals with this
command:
cd <manual>
for i in *.xml; do \
pandoc -f docbook -t rst --shift-heading-level-by=-1 \
$i -o $(basename $i .xml).rst \
done
The conversion was done with: pandoc 2.9.2.1-91 (Arch Linux).
Also created an initial top level index file for each document, and
added all 'books' to the top leve index.rst file.
The YP manuals layout is organized as:
Book
Chapter
Section
Section
Section
Sphinx uses section headers to create the document structure.
ReStructuredText defines sections headers like that:
To break longer text up into sections, you use section headers. These
are a single line of text (one or more words) with adornment: an
underline alone, or an underline and an overline together, in dashes
"-----", equals "======", tildes "~~~~~~" or any of the
non-alphanumeric characters = - ` : ' " ~ ^ _ * + # < > that you feel
comfortable with. An underline-only adornment is distinct from an
overline-and-underline adornment using the same character. The
underline/overline must be at least as long as the title text. Be
consistent, since all sections marked with the same adornment style
are deemed to be at the same level:
Let's define the following convention when converting from Docbook:
Book => overline === (Title)
Chapter => overline *** (1.)
Section => ==== (1.1)
Section => ---- (1.1.1)
Section => ~~~~ (1.1.1.1)
Section => ^^^^ (1.1.1.1.1)
During the conversion with pandoc, we used --shift-heading-level=-1 to
convert most of DocBook headings automatically. However with this
setting, the Chapter header was removed, so I added it back
manually. Without this setting all headings were off by one, which was
more difficult to manually fix.
At least with this change, we now have the same TOC with Sphinx and
DocBook.
(Bitbake rev: 6bf6c8d63787aed7624793c24af3fa603b5ac961)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used sphinx-quickstart to generate top level config and
Makefile.sphinx, to allow side by side DocBook and Sphinx
co-existence.
(Bitbake rev: 84ccba0f4aff91528f764523fe1205a354c889ed)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Blocks SIGINT in the worker processes to prevent them from running the
parent process signal handler, which causes them to deadlock under
certain circumstances.
[YOCTO #14034]
(Bitbake rev: 9f4207f4b598f549cbd4159841c720276736f23b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're either hitting rate limiting with freedesktop.org or the
servers have intermittent network connections. Use our own mirror
of these repositories instead.
(Bitbake rev: a1b7ab5c9d5e64969f5ca0e41c0ac13c723e3761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing leaking open socket connections when errors occur and
tinfoil is in use. Improve the exception handling so the sockets
are closed even if exceptions occur, allowing more robust behaviour
when things go wrong.
(Bitbake rev: cefbec9ff47ca973a74ec7300cd736f3e0f0bce0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the recent init changes, if a client disconnects before issuing a
command, the cooker can break in the reset handlers. Add some guards
in the code to prevent this.
(Bitbake rev: 12605e30e4c4e1ae6a67c97363b892ebf0b9566c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following code costs a lot of time when there are lot of layers and recipes:
for collection in collections:
collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''
My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K
(3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a
keyword argument to get_file_layer() can fix the problem, it can save about 90%
time in my build (6min -> 40s).
(Bitbake rev: f08a6601c9bb09622855d62e1cedb92fafd2f71d)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
6min) in my build (more than 3000 recipes)
The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
each recipe, and get_file_layer() compare the file with each item in BBFILES
which makes it very time consuming when there are a lot of recipes and items in
BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't
make sense to go on the loop when file is matched.
And use fnmatchcase to replace of fnmatch since the comparison should be
case-sensitive.
(Bitbake rev: 8d64181d29dc262e066a6114dd51e5f2d04f47de)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The original code saved BBFILES back to BBFILES without any changes which isn't
usefule, so remove that line. Now save prioritized BBFILES to
BBFILES_PRIORITIZED which can accelerate the query a lot for the one which
relies on it such as bb.utils.get_file_layer().
(Bitbake rev: 49bdb5dfa57b41b3ed399961e947c404f9195998)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a contrib/ script that repeatedly runs bitbake and interrupts
parsing to try and reproduce parsing deadlocks.
(Bitbake rev: 0457482e252f216618a6fccad0030fcd6c5a304f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otherwise this can interfer with multiprocessing exit handling.
(Bitbake rev: b88816c4c84fa4f5ad39c263f5e75b96476e9768)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you make parsing fail (e.g. add something like:
X := "${@d.getVar('MCMACHINES').split()[1]}"
to meson.bbclass, then run "while true; do bitbake -g bash; done"
it will eventually hang. It appears the cancel_join_thread() call the
parsing failure triggers, breaks the results_queue badly enough that it
sits in read() indefintely (called from self.result_queue.get(timeout=0.25)).
The timeout only applies to lock aquisition, not the read call.
I've tried various other approaches such as using cancel_join_thread()
in other places but the only way things don't lock up is to avoid
cancel_join_thread() entirely for results_queue.
I do have a concern that this may adversely affect Ctrl+C handling
but equally, its broken now already and this appears to improve
things.
[YOCTO #14034]
(Bitbake rev: 9c61a1cc7be46c23da1f4ef3bee070fb83c4be57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During cooker shutdown, its possible the parser isn't cleaned up. Fix
this (which may partially explain why threads were left hanging around
at exit).
(Bitbake rev: 928609f30f3a20aaa2f88afc18044a4e10199488)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its hard to tell from the server logs whether commands complete or not
(or how long they take). Add extra info to allow more debugging of
server timeouts.
(Bitbake rev: 56285ada585ec1481449522282b335bcb5a2671e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Knowing which process printed which messages and the timestamp of the
message is useful for debugging, so add this. Ensure the log parsing
isn't affected by using search() instead of match().
(Bitbake rev: 1d043666710df1fa9d9586fd974c0371dd1514b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Showing "leftover process" messages when a new server has started and is being
used by some UI is horrible. Compare the PID data from the lockfile to
avoid this (and the ton of confusing log data it generates).
Also, move the time.sleep() call to be after the first lock attempt, which
reduces noise in the logs significantly.
(Bitbake rev: ce1897a31afb5a14997bc3d2f459b90d43eecb7d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This could potentialy account for some of the missing thread cleanup
we're seeing.
(Bitbake rev: 8f2d690428de8934868b406b79c4699a8ebe902c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lsof/fuser error if the file doesn't exist. It can be deleted by something
else so ignore this if it happens and loop.
(Bitbake rev: b100d22ce37b7548b50e59a71802bcc903acd6ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cookerlog output goes to a file and its misleading to look at it and
not have it up to date with what the cooker is actually doing. Ensure
written data is flushed.
Ultimately this should be using python's logging but that is for another
day, we need simple fixes right now.
(Bitbake rev: d95c3dd2b8ac50423976a7baf0a51e9580871761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The binary string printed into the output is ugly, parse this so the
linebreaks come out in the logs and make them much more readable (I
was misssing the information initially despite looking for it).
(Bitbake rev: c2dd8bb434d5738fedf9019651074b90affff3b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Lets use valid python even if it the original happens to work.
(Bitbake rev: 343187c57e1459b0e57f90463843782f3a3a8443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling sys.exit() in the middle of the code is rather antisocial. We catch
this in various places but we shouldn't have to. In all these cases we have
already sent events explaining to the user what happened. This means the
correct exception is BBHandledException.
The recent startup changes have moved the point a lot of this code gets
called to inside the UI, with memres it would have always been possible
from there anyway. This change makes things much more consistent.
(Bitbake rev: 91699f366d24480ff3b19faec78fb9f3181b3e14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With an invalid layer, the desired error output should be:
ERROR: The following layer directories do not exist:
ERROR: /this_path_does_not_exist
ERROR: Please check BBLAYERS in .../build-invalid-layer/conf/bblayers.conf
Instead we were met with a backtrace:
Traceback (most recent call last):
File "/scratch1/fray/xilinx/poky/bitbake/bin/bitbake", line 36, in <module>
cookerdata.CookerConfiguration()))
...
File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 267, in parseBaseConfiguration
self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 358, in parseConfigurationFiles
sys.exit(1)
SystemExit: 1
(Bitbake rev: 3a2503c785a5cd9dca0dc68c3aec31b4bec7684b)
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new server startup code means exceptions can happen when we aren't
setup to show them to the user correctly, leading to ugly tracebacks.
Add in some special case handling of BBHandledException to at least
ensure that common case doesn't traceback and the user sees meaningful
output.
In the future, the logging setup can likely be improved, as can the way
runCommand handles exceptions, they all should likely become real
exceptions again on the UI side.
[YOCTO #14022]
[YOCTO #14033]
(Bitbake rev: 6059d0e77f60ddb679049bd34478f41b1ab7995d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Convert the test suite that was in COW.py into something that
will actually run as part of bitbake-selftest. This is in
preparation for some cleanups I plan in COW.py.
(Bitbake rev: a73d45cb6010e14bf93fec857303bc7ff321066f)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I strongly suspect this function doesn't work with modern python so
and its unused now, drop it.
(Bitbake rev: a3033cea089c66c8b4614e7ee57c166f4262c590)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I have a suspicion based on process traces that the flock() call is no
longer interrupted by SIGALRM and hence the timeout doesn't work. We
were relying on EINTR triggering around syscalls but python is likely
protecting us from that in modern versions.
Re-implement this code with a different mechanism which doesn't have
that potential issue.
(Bitbake rev: 8eb52afdfd4c3e6478d4f8cc56e99def3f1c924c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that compat.py just imports Python standard library stuff, get rid
of the layer of indirection.
(Bitbake rev: e2be6defbb9fcf25f9df04c3b452d0dba48dfd03)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using sys.executable ensures we're using the same python binary for the server
as the client, which is probably advisable. "bitbake-server" is left as the process
name as its more distinctive in process listings.
(Bitbake rev: 387a339b330e8122a62a148249beb3f064dd4e3d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Globbing in file:// urls is terminally broken. Currently when its used, the
file checksum code is basically bypassed. This means changes to the source
files don't change the task checksum, the task doesn't rebuild when the
inputs change and things generally break.
To make globbing work generically, we'd have to scan the file system for
all possible matches to the glob and log whether they exist or not. We can't
simply log the files which exist, we have to also know which files could
later exist and influence the choice of file so we know when to reparse.
For a simple file://xxx/*, this could be done but for bigger patterns,
it becomes much more problemtic. We already support file://xxx/ in urls.
So, lets decide we'll not support globs in file://urls. Worse case users
can put files in a directory and reference that, moving files into place
if needed.
Remove all the glob special cases (see the comments if anyone doesn't
believe this is terminally broken) and error to the user if they have
such urls.
(Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.
(Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5)
Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only the env variables which were added to the datastore were being
updated. We need to update the whole copy, we just only trigger a
reparse if any of the variables making it into the datastore change.
This avoids a bug where variables such as DISPLAY in a new UI context
would break under memory resident bitbake.
(Bitbake rev: 4bb71b627767297269e762b414443e15e28bfac4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should not be using sys.argv inside the server to decide which targets
the user added on the commandline. There might not even be a commandline.
Thankfully the targets variable is easily accessible in this context
and contains this exact data we want.
(Bitbake rev: 5b12bf30bccdd00262e74964223220c649040be4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to have a new python process forked off an original doesn't work
out well and ends up having race issues. To avoid this, exec() a new
bitbake server process. This starts with a fresh python interpreter
and resolves various atexit and other multiprocessing issues once
and for all.
(Bitbake rev: 9501dd6fdd7a7c25cbfa4464cf881fcf8c049ce2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dump info into the logs if there are extra threads left at process exit
time.
(Bitbake rev: 1c9496797b753e67351bd5cb98ef2b8e9435d51e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes to configuration.env were not updating BB_ORIGENV, fix this.
(Bitbake rev: c5fbd8452f87e0a2d234eaf27d0450aacdeb8891)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.
This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.
The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.
(Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sock object isn't used client side so we can just created it in
the server process and save passing around the fd/object.
(Bitbake rev: ee5d2c92dcce89ccb701e028ffc6419eb315f5ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
