By including the full path to bblayers.conf the remove-layer
command can be executed from any location, not only from the
build directory.
(Bitbake rev: 25cb4e17e8a4c0310df018b9df184b0155b267eb)
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By including the full path to bblayers.conf the add-layer
command can be executed from any location, not only from the
build directory.
(Bitbake rev: 77aa230bf2a053fc0941723a6abbc798ebe53a19)
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Assigning a return value which is potentially None to a tuple and
catching TypeError is pretty ugly. Rewrite the code to explicitly check
the value for clarity.
(Bitbake rev: f4ebb27616ac2df27c29a6052b1526a4c48db607)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the export/unexport/network flags are acted on only by presence
or lack of in the data store. This is deliberate due to performance reasons
since a given recipe shell task might export ~80-90 variables and this means
expanding flags for every one of them.
This does catch users unaware since values which expand to nothing don't work
e.g. ${@""} and setting values to "0" wouldn't work either.
The downside to this patch is slow down in parsing speed of around 4%.
The network flag is easier to change and doesn't affect performance, it was
made to operate the same as export/unexport for consitency reasons.
(Bitbake rev: 3d96c07f9fd4ba1a84826811d14bb4e98ad58846)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling SystemExit doesn't work well with server/client usage since the string
isn't printed to the right place. Use bb.fatal() instead which prints the right
log output and raises and handled exception which then shows correctly on the
UI.
(Bitbake rev: aaefb43b41a0d9b16a59643136268eb6e5d48cd2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you are behind a corporate proxy, the npm fetcher uses
the proxy IP address already passed in the list of exports.
However, it will error if the proxy injects its own root
CA certificate. Thus, the NODE_EXTRA_CA_CERTS environment
variable must be passed so the user can include their
company's root CA as a trusted CA inside node's
certificate store.
(Bitbake rev: 5daecc25a59ce5bf6c31f5085e6caa7cb543517e)
Signed-off-by: George Kelly <george.kelly1097@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython
has an optimization for statements in the form "a = a + b" or "a += b".
It seems that this line does not get optimized, because it has a form a = a + b + c:
data = data + "./" + f.split("/./")[1]
For that reason, it does a copy of data for each iteration, potentially copying megabytes
of data for each iteration.
Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds
instead of 45 seconds on my test setup where SRC_URI points to a big directory.
Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython:
"do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b"
However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup:
data = ''.join((data, "./", f.split("/./")[1]))
I have changed the other lines to also use += for consistency only, however those were in the form a = a + b
and were optimized already.
Co-authored-by: JJ Robertson <jrobertson@snap.com>
(Bitbake rev: 195750f2ca355e29d51219c58ecb2c1d83692717)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Increase the `ljust` value for each column in show-layers
output. This improves readability when long layer paths are
included
(Bitbake rev: a56cd53e0e67a368acace9dab184bd94b9b67b1b)
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace the layer directory name with the name from BBFILE_COLLECTIONS
which is useful when assigning a LAYERDEPENDS value
(Bitbake rev: 517cc3f5aee2a7d6362ce0202d4ee709a5378096)
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).
Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.
In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.
Also remove the return value as nothing ever checked it.
[ YOCTO #15000 ]
(Bitbake rev: c19035e8e71c419c5688a86bfc9c946c96f638e8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the codeparser cache is set from CACHE, which is typically in
bitbake.conf which means we can't read/write any cache until it is found/read.
We may well have python expressions to parse before that happens.
The net result is suboptimal functioning of the codeparser cache since it will
often be invalidated by data that is never written.
This patch changes the codeparser and filechecksum caches to use BB_CACHE as
their setting and defaults it to ${TOPDIR}/cache.
The patch doesn't change where the "persistent" data such as prserver and
hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing
cache resists (still currently CACHE). I've left those for a later patch.
The patch does ensure data parsed by the core datastore parsing calls is
written back since this is now much more useful after this change.
(Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Update to latest langdale 4.1.2 and kirkstone 4.0.6
* Re-instate dunfell and update to 3.1.22
- drop comments about bitbake crash
(Bitbake rev: f90c45df5cf4640e0714fde13e311db3327f9fd4)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-getvar does not have a way to silence bitbake
server's logger and that makes the tool hard to use for
text processing. This is especially true when one wants to
get a bitbake value to be piped to some other utility and
instead we get uncontrolled logging messages or warnings
together with bitbake's variable value.
Example without quiet:
bitbake-getvar --value MACHINE
NOTE: Starting bitbake server...
qemux86-64
With quiet:
bitbake-getvar --value MACHINE --quiet
qemux86-64
(Bitbake rev: af354e975d0b4c26d0e91e3c82946b093bc11b45)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some users of _findVar don't need the override data and even
getVarFlag doesn't need it in some common cases (parsing=True).
Rearrange the code as the current overridedata approach doesn't need
to be in the _findVar code anyway. This removes some search overhead
from a critical path.
(Bitbake rev: fcb64e1138a20eb19560af3fc5d1fa748cc9cf34)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the codeparser cache ends up being extended for every parse run
since there are values in the functions such as the result of os.getpid()
from LOGFIFO in OE-Core.
Digging into that issue, there are also lots of similar but different
functions being parsed where the change might just be a path to WORKDIR,
a change in PN or PV or something like DATE/TIME.
There is no reason we have to use these changing values when computing the
dependenies of the functions. Even with a small tweak like:
BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn"
the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run
to around 1.3MB and remaining static for oe-core and meta-oe. In my local
build, admittedly heavily experimented with, the cache had grown to 120MB.
The benefits of doing this are:
* faster load time for bitbake since the cache is smaller to read from disk
and load into memory
* being able to skip saving the cache upon shutdown
* lower memory footprint for bitbake
* faster codeparser data lookups (since there is less data to search)
We only use these special values when passing code fragments to the codeparser
to parse so the real variable values should otherwise be used in the hash data.
The overall effect of this change, combined with others to avoid saving unchanged
cache files can be ~2s on a ~16s parse on my local system and results in a more
responsive feeling bitbake. It also allows parsing performance to be investigated
more consistently.
(Bitbake rev: f24bbaaddb36f479a59a958e7fc90ef454c19473)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When draining the result queue from the parsing processes, cache objects
can be created even if they are then immediately destroyed. The reset
in the sync code needs to happen after any objects have been created.
Change the ordering to fix this. This ordering has caused various
cache errors, particularly when interrupting parsing with Ctrl+C.
(Bitbake rev: f45a94e6720dacf7f51ac147c115a6f608769093)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By writing the cache files only if there is data to write, we can save a
bit of time.
(Bitbake rev: abeff1f80bb1c690b92d535d472dff9df7a56067)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to reset the cache before the start() call which starts the
parsing processs, tweak the code to ensure this is the case.
(Bitbake rev: 4712d6237df5d2188238294fbdb9f107b068a3a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to check against the first line of the file, fix the typo.
(Bitbake rev: 4abc598fb01d426394f4222dfc752e620a8e1b7b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While the parameter is not required it allows testing of possible
regression in fetcher code when parameter specified and mirrors are
used.
(Bitbake rev: 1a21918049091a6d77426dbf8868ffdc14ba1003)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Basic test for mercurial functionality including PREMIRRORONLY with
disabled network
(Bitbake rev: c0e349a5fc92e9c90afc53800c7e83495393ff79)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If memory resident bitbake is active and the build directory is renamed upon
build completion, several bad things can happen:
* the old build directory could be re-created to contain a lockfile
leaving an empty directory behind
* a lockfile for a new build could be found and attempt to be locked
This patch avoids creating an empty directory (not perfectly, but should
work in the majority of cases - an empty directory is cosmetic).
It also now compares the lock file contents to it's own pid and
just exits if it doesn't match, it is clearly then belonging to some
new process.
This will be combined with bitbake shutdown calls on the autobuilder to
ensure "saved" build directories, or build directories being deleted by
clobberdir don't do strange things.
(Bitbake rev: b986eac18b6a8bf633f5ef15f32f68de4c86173b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sqlite can't open the cachefile when its path is too long (>= 505 char by
my tests).
We do have a path length check in sanity.bbclass but this code is called
before sanity checks.
Treat the error in the exception instead of checking before hand in case
sqlite eventually fixes this.
Fixes [YOCTO #12374]
(Bitbake rev: bf681d173263cd42ffc143655f61abe0573fd83c)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than risk the heartbeat event code locking up the server control
socket, handle it in the 'idle' thread with the other work. The aim
is to remove it as a possible issue with some ongoing hangs.
(Bitbake rev: 0f9a0c7853b181817bf01863a26da21412376294)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Often, '--buildfile/-b' is used to mean "Do not execute anything but
this recipe", so it make sense to have it in the "Execution control
options" group.
Fixes [YOCTO #12018].
(Original suggestion by Ross Burton)
(Bitbake rev: 412a6d8701a92efe4dd6db8ec713c0013f7db64d)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shutdown() call should definitely not be waiting for idle, since we
expect execution and events to continue even after setting either shutdown
state. This was causing the UI to appear to hang at Ctrl+C interrupts.
Also ensure that if we reset cooker, we stop any active parser processes
since these currently appear to be being left behind.
(Bitbake rev: 4f73c2eb12ee074f3b7d4717380649f6ca8f3def)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are three levels of shutdown, the initial "wait for current tasks",
then "stop current tasks" and "exit now". Change the code so that we don't
instantly exit after "stop current tasks" but allow the events to come
through from the server first. The new shutdown level allows that to then
be broken out of too.
(Bitbake rev: af38345b91cfc8a6bb5c38f5753255e7635bfe2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for some reason the bitbake server dies, we shouldn't loop indefinitely
waiting for events within tinfoil. Add a ping test and exit if things
have somehow failed.
(Bitbake rev: 32da768ae67cfe0b2d154ce4667a7cfdfa5bf4b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
currentAsyncCommand currently doesn't have any locking and we have
a conflict in "idle" conditions since the idle functions count needs
to be zero *and* there needs to be no active command.
Move the changes/checks of currentAsyncCommand to within the lock
and then we can add it to the condition for idle, simplifying some
of the code.
(Bitbake rev: b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Indirecting the async command handler via cooker is confusing and no
longer needed. Drop it to make things slightly clearer.
(Bitbake rev: 4a41a7d0594e6a84a67b9de70a505858aebcd84a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #12018].
Also, I have included some small fixes:
* added a '.' at the end of sentence where it was missing
* split some long lines
(Bitbake rev: 9257c48c3e36daaecb5e15da22d0bed24865f02c)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We no longer need to abstract the inotify callback handler, remove the
abstraction and simplify/clean up the code.
(Bitbake rev: af4ccab8acc49e91bf7647f209d69f4858618466)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The loop submitting the parser results was not efficient on many core systems
as may of the parsers could block for a long time waiting to send the results.
While a result was pending, the code wouldn't parse further jobs.
Change the code to parse further jobs even if the results can't be submitted
and lower the result submission timeout to keep the parsers busy.
(Bitbake rev: 9ece4a2e406509bd89dbce8dac80a02e600b0ecf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.
To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).
Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.
(Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are a couple of failure cases that runCommands needs to better
handle, ensuring finishAsyncCommand is called. This doesn't matter if the
server is about to shut down but for memory resident bitbake and with
threading enabled, correctness is more important this could could in
theory lead to problems with the recent code changes.
By using the idleFinish class, it ensures the current async command
is terminated correctly and the various state pieces around commands
finishing are called. This also makes the code more uniform matching
the other exception handling code.
(Bitbake rev: 367a83ed46c7fbcdd06579b6cc3a6e48a89ca7fb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous commit was injecting a generator object instead of
a string, which happened to fix the issue but we should fix it
properly! Thanks to Christopher Larson for spotting.
(Bitbake rev: 36589f13c5babb3b2af82666c8a96338e6ea4eb9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES variables,
we need to be deterministic. The random ordering from the sets was causing
unexpected reparses.
(Bitbake rev: 8c405c97430ac830837e25438e8795f6f7abbdaa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing failures where the UI hangs if the server disappears. Ping
the cooker/server if we've not had any events in the last minute so we can
check if it is still alive.
(Bitbake rev: 6567ad6181f9e39812097f0154647e4b38238fdd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When bitbake is off running heavier "idle" commands, it doesn't service it's
command socket which means stopping/interrupting it is hard. It also means we
can't "ping" from the UI to know if it is still alive.
For those reasons, split idle command execution into it's own thread.
The commands are generally already self containted so this is easier than
expected. We do have to be careful to only handle inotify poll() from a single
thread at a time. It also means we always have to use a thread lock when sending
events since both the idle thread and the command thread may generate log messages
(and hence events). The patch depends on previous fixes to the builtins locking
in event.py and the heartbeat enable/disable changes as well as other locking
additions.
We use a condition to signal from the idle thread when other sections of code
can continue, thanks to Joshua Watt for the review and tweaks squashed into this
patch. We do have some sync points where we need to ensure any currently executing
commands have finished before we can start a new async command for example.
(Bitbake rev: 67dd9a5e84811df8869a82da6a37a41ee8fe94e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for adding splitting bitbakes work into two threads,
add locking around the idle functions list accesses.
(Bitbake rev: a9c63ce8932898b595fb7776cf5467d3c0afe4f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the move to a server idle thread, we always need threading. The
existing accessor functions could end up turning this off!
I was going to hold the lock whilst changing it, check if the value
was already set, cache the result and also fix the event code to always
release the lock with a try/finally.
Instead, disable the existing functions and use a with: block
to handle the lock, keeping things much simpler.
(Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When idle handlers want to exit, returning "False" isn't very clear
and also causes challenges with the ordering of the removing the idle
handler and marking that no async command is running.
Use a specific class to signal the exit condition allowing clearer code
and allowing the async command to be cleared after the handler has been
removed, reducing any opportunity for races.
(Bitbake rev: 102e8d0d4c5c0dd8c7ba09ad26589deec77e4308)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When finishing a command, we need to ensure any parsing processes that may have
been started are cleaned up before we reset the cooker state.
(Bitbake rev: 6569ab64bea35de21acc89053ba76e2828163f3f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the idle functions loop suffers a traceback, it is
silently dropped and there is no log message to say what happened.
This change at least means the traceback is in the cooker log, making
some debugging possible.
Add some logging to show when handlers are added/removed to allow
a better idea of what the server code is doing from the server log
file.
(Bitbake rev: 9cf3102dc36513124fe5ead2f1e448b51833b6ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As far as I could tell, the current code could result in some strange
situations where some data was set back to the original data store copy
but the multiconfig data was not restored. There are also some changes made
to the datastore which did not persist.
The data store was also being reset at every client reset, which seems
a little excessive if we can reset it to the original condition properly.
Move the __depends -> __base_depends rename into databuilder along with
the __bbclasstype change so these are saved in the original data.
Tweak the databuilder code to be clearer and easier to follow on which
copies are where, then save copies of all the mc datastores.
Finally, drop the cache invalidation upon reset for the base config
as we shouldn't need that now (oe-selftest -r tinfoil works with memory
resident bitbake which was the original reproducer requiring that change).
(Bitbake rev: 5f8b9b9c35b4ec0c8c539bf54ca85f068f4a5094)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently heartbeat events are always generated by the server whilst it is
active. Change this so they only appear when builds are running, which is
when most code would expect to be executed. This removes a number of races
around changes in the datastore which can happen outside of builds.
(Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
