To avoid reparsing the bbclass code all the time, move the function
to the python function library code which is more efficient.
(From OE-Core rev: 7c8c4cfd9355a9cee1144e0444e1b54402f1951c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.
To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).
Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.
(Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OE-Core has changes which do need careful evaluation for compatibilty,
such as the addpylib directive. Move the core later names to mickledore
so layers can mark their compatibility as such.
Also increase the version number for core. If we do make further changes
that need layer changes, we'll update the version again so layers can mark
compatibility within the series.
(From OE-Core rev: 57239d66b933c4313cf331d35d13ec2d0661c38f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Brings following changes
* f47a8cdd ldso: fix invalid early references to extern-linkage libc.page_size
* 377218cb pthread_atfork: fix return value on malloc failure
* 29e43191 fix double-processing of DT_RELR relocations in ldso relocating itself
* b50eb8c3 fix strverscmp comparison of digit sequence with non-digits
* ad5dcd39 fix async thread cancellation stack alignment
* 8f925945 fix return value of gethostby{name[2],addr} with no result but no error
* 63402be2 clean up dns_parse_callback
* 0a7b4323 dns response handling: don't treat too many addresses as an error
* 41603c77 dns response handling: ignore presence of wrong-type RRs
* cf76df0e fix missing synchronization of pthread TSD keys with MT-fork
* 5ff3eea9 fgets: avoid arithmetic overflow when n==INT_MIN is passed
* d8f35e29 fix AS-safety of close when aio is in use and fd map is expanded
* 26c76a90 fix use of uninitialized dummy_fut in aio_suspend
* aebd6a36 fix potential deadlock between multithreaded fork and aio
* d64148a8 fix potential unsynchronized access to killlock state at thread exit
* 36b72cd6 fix potential deadlock in dlerror buffer handling at thread exit
* 833a4691 configure: disable TBAA optimization because most compilers are buggy
* e6e82132 disable MADV_FREE usage in mallocng
* 25e6fee2 remove LFS64 programming interfaces (macro-only) from _GNU_SOURCE
* 246f1c81 remove LFS64 symbol aliases; replace with dynamic linker remapping
* dec8f0a4 dns query core: detect udp truncation at recv time
* 8c408937 getaddrinfo dns lookup: use larger answer buffer to handle long CNAMEs
* 759bf785 arpa/nameser.h: update RR types list
* 51d4669f dns: implement tcp fallback in __res_msend query core
* e2e95176 res_send: use a temp buffer if caller's buffer is under 512 bytes
* c87d75f2 adapt res_msend DNS query core for working with multiple sockets
* 85050ac5 getaddrinfo: add EAI_NODATA error code to distinguish NODATA vs NxDomain
(From OE-Core rev: 3b27f6356a3bf0138da0c3dcd87aa8e4a93ae56b)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It needs to be fixed to honor _FILE_OFFSET_BITS before we can enable
64bit time_t
(From OE-Core rev: 206ab9522963aee471004d987181ed2f8363f1ad)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
soup 2.x support has been removed upstream.
libportal is no longer optional.
epiphany now also needs gtk-update-icon-cache
and update-desktop-database executables.
(From OE-Core rev: ace8d5cb5ebb027ec2f62bf4d1104d319ec86e95)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This used to be required by epiphany, then it became
optional and was moved to meta-oe, now it is again required.
The recipe was originally provided by... actually me :-)
(From OE-Core rev: e3c1fb52a0720d88dbc6eb91935c82733cdaea44)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are a couple of failure cases that runCommands needs to better
handle, ensuring finishAsyncCommand is called. This doesn't matter if the
server is about to shut down but for memory resident bitbake and with
threading enabled, correctness is more important this could could in
theory lead to problems with the recent code changes.
By using the idleFinish class, it ensures the current async command
is terminated correctly and the various state pieces around commands
finishing are called. This also makes the code more uniform matching
the other exception handling code.
(Bitbake rev: 367a83ed46c7fbcdd06579b6cc3a6e48a89ca7fb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous commit was injecting a generator object instead of
a string, which happened to fix the issue but we should fix it
properly! Thanks to Christopher Larson for spotting.
(Bitbake rev: 36589f13c5babb3b2af82666c8a96338e6ea4eb9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES variables,
we need to be deterministic. The random ordering from the sets was causing
unexpected reparses.
(Bitbake rev: 8c405c97430ac830837e25438e8795f6f7abbdaa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
devtool modify/upgrade are not currently equipped to handle conditional local files
in SRC_URI, and provide only the main no-override set in a workspace under
source/component/oe-local-files/ (this is done via meta/classes/devtool-source.bbclass).
On the other hand, updating the changes from workspace into a recipe
is run iteratively against all overrides; this works for patches (as they
all are directed into their own override branches in the workspace
git source tree), but breaks down when trying to match local files
in a workspace against local files in overridden SRC_URI lists, resulting in
bad recipe breakage.
(there's an additional twist here: existing code has a guard against this
but the guard relies on metadata in workspace .bbappend that is only there
in modify operations, but not upgrades. This commit replaces the guard
with a general check that will work everywhere).
Implementing multiple sets of local files is significant work; let's for now
simply not touch local files in recipes except when on the no-override variant.
Also, adjust the selftest cases to include conditional local files in sample
recipes, so the situation is covered by the tests.
(From OE-Core rev: 3a8654b860fa98f94e80c3c3fff359ffed14bbe7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes build with musl where the LFS64 functions are deprecated
(From OE-Core rev: f543f51b2060a718685c82ee6f7d59bb0723e7bf)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Qemu 7.2 finally allows us to move beyond building for original Core 2/Core i7 era hardware,
and this patch adds support for the newer generations. But first, a bit of
background:
Recently toolchains gained support for specifying x86-64 'levels' of
instruction set support; v3 corresponds to 2013-era Haswell CPUs
(and later), with AVX, AVX2 and a few other instructions that
were introduced in that generation. I believe this is preferrable
to picking a specific CPU model as the baseline.
Here's Phoronix's feature article that explains the feature and the available levels:
"Both LLVM Clang 12 and GCC 11 are ready to go in offering the new x86-64-v2, x86-64-v3, and x86-64-v4 targets.
These x86_64 micro-architecture feature levels have been about coming up with a few "classes" of Intel/AMD CPU processor support rather than continuing to rely on just the x86_64 baseline or targeting a
specific CPU family for optimizations. These new levels make it easier to raise the base requirements around Linux x86-64 whether it be for a Linux distribution or a particular software application where
the developer/ISV may be wanting to compile with greater instruction set extensions enabled in catering to more recent Intel/AMD CPUs."
https://www.phoronix.com/news/GCC-11-x86-64-Feature-Levels
Here's gcc docs for it:
https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html
And here's the formal specification (click on the pdf link):
https://gitlab.com/x86-psABIs/x86-64-ABI
The actual tune file was created by copying corei7 tunes and doing
search/replace on them. Qemu options were dropped as unnecessary.
32 bit tune was dropped as well, as there is no 32 bit only CPU
that also supports these new instructions; all of the v3 capable
chips are 64 bit.
(From OE-Core rev: ac041f90e71dba83b7144c91f929de88aaeae519)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The same rust getrandom patch is needed for cargo on target as well
as it uses the same crate
(From OE-Core rev: 44441c727a5301ab99ab8b4d8b8b1f61f0a810af)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* environment.d/rpm.sh sets the right environment in SDK
and we don't need to use them
(From OE-Core rev: 34f1121ba9684b1ab978438db2b16c1e42ea5973)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* the paths in wrapper don't work for rpmdeps which is installed in
${libdir}/rpm unlike other wrapped bins from ${bindir} these relative
paths don't work there
* replace environment.d-rpm.sh with here-doc so that we can use
OE variables
* in the end it might be better to just get rid of the wrappers at
this point and depend on environment.d to always set right values
* the wrappers were added in:
commit 760103cdaed3e820888d8984ec0b76cfc831d534
Author: Ovidiu Panait <ovidiu.panait@windriver.com>
Date: Fri May 25 10:48:29 2018 +0800
nativesdk-rpm: Add wrappers for nativesdk support
When installing the SDK to a non-default path, running "rpm --showrc" from the
sdk will produce the following error:
error: Unable to open /opt/windriver/wrlinux-small/10.17.41/sysroots/x86_64-wrlinuxsdk-linux/usr/lib/rpm/rpmrc for reading: No such file or directory.
Fix this by adding wrappers that dynamically export the RPM_CONFIGDIR,
RPM_ETCCONFIGDIR and MAGIC environment variables, pointing to the proper
sdk locations.
* the rpm.sh in environment.d a bit later:
commit 5f16fd0bf774314c79572daf4ba7e4a8ae209ba1
Author: hongxu <hongxu.jia@windriver.com>
Date: Wed Jul 29 01:22:06 2020 -0700
nativesdk-rpm: adjust RPM_CONFIGDIR paths dynamically
While installing/extracting SDK to a non-default dir(not /opt),
run rpm failed:
$ python3 -c "import rpm"
|error: Unable to open /opt/windriver/wrlinux-graphics/20.31/sysroots/
x86_64-wrlinuxsdk-linux/usr/lib/rpm/rpmrc for reading: No such file or
directory.
This patch adds a flexible way to configure RPM_CONFIGDIR in SDK.
(From OE-Core rev: f40a2658f5be6739c5dddab7f9f11e1f85a17102)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
newer musl do not provide lfs64 functions anymore since off_t is always
64bit on musl using normal functions would suffice
(From OE-Core rev: f01b2ab83068e4d7f263c31dca2a3fa9ef77a98e)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing failures where the UI hangs if the server disappears. Ping
the cooker/server if we've not had any events in the last minute so we can
check if it is still alive.
(Bitbake rev: 6567ad6181f9e39812097f0154647e4b38238fdd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When bitbake is off running heavier "idle" commands, it doesn't service it's
command socket which means stopping/interrupting it is hard. It also means we
can't "ping" from the UI to know if it is still alive.
For those reasons, split idle command execution into it's own thread.
The commands are generally already self containted so this is easier than
expected. We do have to be careful to only handle inotify poll() from a single
thread at a time. It also means we always have to use a thread lock when sending
events since both the idle thread and the command thread may generate log messages
(and hence events). The patch depends on previous fixes to the builtins locking
in event.py and the heartbeat enable/disable changes as well as other locking
additions.
We use a condition to signal from the idle thread when other sections of code
can continue, thanks to Joshua Watt for the review and tweaks squashed into this
patch. We do have some sync points where we need to ensure any currently executing
commands have finished before we can start a new async command for example.
(Bitbake rev: 67dd9a5e84811df8869a82da6a37a41ee8fe94e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for adding splitting bitbakes work into two threads,
add locking around the idle functions list accesses.
(Bitbake rev: a9c63ce8932898b595fb7776cf5467d3c0afe4f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the move to a server idle thread, we always need threading. The
existing accessor functions could end up turning this off!
I was going to hold the lock whilst changing it, check if the value
was already set, cache the result and also fix the event code to always
release the lock with a try/finally.
Instead, disable the existing functions and use a with: block
to handle the lock, keeping things much simpler.
(Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When idle handlers want to exit, returning "False" isn't very clear
and also causes challenges with the ordering of the removing the idle
handler and marking that no async command is running.
Use a specific class to signal the exit condition allowing clearer code
and allowing the async command to be cleared after the handler has been
removed, reducing any opportunity for races.
(Bitbake rev: 102e8d0d4c5c0dd8c7ba09ad26589deec77e4308)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When finishing a command, we need to ensure any parsing processes that may have
been started are cleaned up before we reset the cooker state.
(Bitbake rev: 6569ab64bea35de21acc89053ba76e2828163f3f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the idle functions loop suffers a traceback, it is
silently dropped and there is no log message to say what happened.
This change at least means the traceback is in the cooker log, making
some debugging possible.
Add some logging to show when handlers are added/removed to allow
a better idea of what the server code is doing from the server log
file.
(Bitbake rev: 9cf3102dc36513124fe5ead2f1e448b51833b6ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As far as I could tell, the current code could result in some strange
situations where some data was set back to the original data store copy
but the multiconfig data was not restored. There are also some changes made
to the datastore which did not persist.
The data store was also being reset at every client reset, which seems
a little excessive if we can reset it to the original condition properly.
Move the __depends -> __base_depends rename into databuilder along with
the __bbclasstype change so these are saved in the original data.
Tweak the databuilder code to be clearer and easier to follow on which
copies are where, then save copies of all the mc datastores.
Finally, drop the cache invalidation upon reset for the base config
as we shouldn't need that now (oe-selftest -r tinfoil works with memory
resident bitbake which was the original reproducer requiring that change).
(Bitbake rev: 5f8b9b9c35b4ec0c8c539bf54ca85f068f4a5094)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently heartbeat events are always generated by the server whilst it is
active. Change this so they only appear when builds are running, which is
when most code would expect to be executed. This removes a number of races
around changes in the datastore which can happen outside of builds.
(Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are events queued and there is an exception in the main loop
of the UI code, it will print tracebacks on the console indefinitely.
Avoid that by improving the loop exit conditions.
(Bitbake rev: 2d0940b920a22b244f3ba6849c7cd019578386b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Joshua Watt pointed out maintaining the counting on both sides of the
connection isn't needed. Remove the receiver side counting and simplify
the code, also allowing errors if the counts do go out of sync.
(Bitbake rev: aeacfd391903fe68ae600470fc2bbad0502d401e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By inspection, tinfoil handles two of the three command exit cases but
one is missing. Add the CommandExit case in case this is the cause of
one of our recipetool/devtool hangs. Regardless, the fix is necessary.
(Bitbake rev: eadddd94835b6b6a8517dfed3d29e6dbb2d35988)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're still seeing occasional SiggenRecipeInfo coherency issues, add
some further reset points into the parsing code to ensure the cache is
cleared before reparsing.
(Bitbake rev: 26ed783caf11dc9ebf53d3790681eb44c0c360f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've been seeing event handlers where 'd' seems to disappear half way through
event handler execution. This is problematic when multiple threads are active
since this code assumes single threading.
The easiest fix is to change the handler function calls to contain d as a
parameter as we do elsewhere for other functions. This will break any non-text
handlers but I was only able to spot one of those in runqueue. It will also
break handlers than call functions that assume 'd' is in the global namespace
but those failures should be obvious and we can fix those to pass d around.
This solution avoids manipulating builtins which was always a horrible thing
to do anyway and solves the issue without needing locking, thankfully.
(Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a bug in the current cache code where the restored data structures
were "inverted" leading to some very weird issues, not sure how anything worked
like that, this patch fixes it.
Also fix some issues with multiconfig cache ordering problems by resetting
the stream counters when appropriate.
(Bitbake rev: cd06beb948eff5eaf2d474f5b127d51a61b0b2ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
