Pass additional arguments in the fileslocked() context manager to the
underlying lockfile() function. This allows the context manager to be
used for any types of locks (non-blocking, shared, etc.) that the
lockfile() function supports.
(Bitbake rev: ce9fe70156e8f909a3a81da017b89ea61bc6fe38)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user does need to be told about this but it isn't really a warning,
just something they may need to be aware of. Drop the level accordingly.
(Bitbake rev: bc85c044ec250001855f2f9f0717ac031feab7c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent new tasks from being scheduled if the memory pressure is above
a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY"
variable in the conf/local.conf file. This is an extension to the
following commit and hence regulates pressure in the same way:
48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation
Memory pressure is experienced when time is spent swapping, refaulting
pages from the page cache or performing direct reclaim. This is why
memory pressure is rarely seen but might be useful as a last resort to
prevent OOM errors.
(Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c)
(Bitbake rev: 4ada86cb6b05e6e3aabc8015a6e73aacb14a3388)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent the scheduler from starting new tasks if the current cpu or io
pressure is above a certain threshold and there is at least one active
task. This threshold can be specified through the
"BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf.
The threshold represents the difference in "total" pressure from the
previous second. The pressure data is discussed in this oe-core commit:
061931520b buildstats.py: enable collection of /proc/pressure data
where one can see that the average and "total" values are available.
From tests, it was seen that while using the averaged data was somewhat
useful, the latency in regulating builds was too high. By taking the
difference between the current pressure and the pressure seen in the
previous second, better regulation occurs. Using a shorter time period
is appealing but due to fluctations in pressure, comparing the current
pressure to 1 second ago achieves a reasonable compromise. One can look
at the buildstats logs, that usually sample once per second, to decide a
sensible threshold.
If the thresholds aren't specified, pressure is not monitored and hence
there is no impact on build times. Arbitary lower limit of 1.0 results
in a fatal error to avoid extremely long builds. If the limits are higher
than 1,000,000, then warnings are issued to inform users that the specified
limit is very high and unlikely to result in any regulation.
The current bitbake scheduling algorithm requires that at least one
task be active. This means that if high pressure is seen, then new tasks
will not be started and pressure will be checked only for as long as at
least one task is active. When there are no active tasks, an additional task
will be started and pressure checking resumed. This behaviour means that
if an external source is causing the pressure to exceed the threshold,
bitbake will continue to make some progress towards the requested target.
This violates the intent of limiting pressure but, given the current
scheduling algorithm as described above, there seems to be no other option.
In the case where only one bitbake build is running, the implications of
the scheduler requirement will likely result in pressure being higher
than the threshold. More work would be required to ensure that
the pressure threshold is never exceeded, for example by adding pressure
monitoring to make and ninja.
(Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f)
(Bitbake rev: f4954b878d404a72156fe98609387aa9c5747af0)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <randy.macleod@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I realised only the first logging message was being displayed in a given
parsing process. The reason turned out to be the UI handler failing
with a "pop from empty list". The default handler was then lost and
no further messages were processed.
Fix this by catching the exception correctly in the connection writer code.
(Bitbake rev: b8fd6f5d9959d27176ea016c249cf6d35ac8ba03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d3e64f64525187f1409531a0bd99df576e627f7f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently have no API to be able to remove all the potential stamps of a
task. It is unusual to need to do this, particularly as you could race against
other things happening in the system but we do have a use case for this in
cleaning up sysroots in OE-Core. The alternative is to mess with CLEANMASK in
OE-Core but that is just going to add potential for errors.
We need the first part of the make_stamp() function so separate that out so
it can be called seperately.
(Bitbake rev: 494fcfcb7c4469915c67f544997104d81c417266)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4d671504a25863018ac51c21c005cef0a4d8f05c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SRC_URI contains python function that extends vardepvalueexclude its
value is being tracked by sstate-cache, which can lead to rebuilds if
value is set dynamically (for example gerrit replicas).
Return empty string if vardepvalueexclude is checked to fix this
behaviour.
(Bitbake rev: 943701ee8cb55307996545f0237721413edfb168)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f5f9a7b89a7d8321f03184e61ad6d5ed8d0f840e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current fetcher seemed to have some issues that made it difficult when
trying to use the same. This patch fixes the following
* Make consistent use of the path that needs to be used as oscdir
* The path mentioned in os.access in download function was not same as
ud.moddir which would result into invoking of fetch command instead of
update command even if directory already existed
* Before creating oscrc, make sure oscdir exists and create it if it does
not exist
* Updated the configuration to use apiurl and added a new parameter to
control whether http or https needs to be used to connect to apiurl
(Bitbake rev: 8ac6e09447d884e658c556388d6014279c50f202)
Signed-off-by: Gunjan Gupta <viraniac@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3ec78686f3c0ea2304097b86a965f9be4b0cb879)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't have less in HOSTTOOLS in OE and this can confuse git. Force the
pager to cat to be consistent and minimal everywhere.
(Bitbake rev: 59c16ae6c55c607c56efd2287537a1b97ba2bf52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d3d406e8552fdd865dc58b419a84411736475ad2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File names containing colons cause split() to return a list with more
than two elements, which will lead to a stack trace ending in:
ValueError: too many values to unpack (expected 2)
Split only once at the last colon, thereby making sure that only two
elements are returned.
(Bitbake rev: e14ed7dd866334ab40cd335d9c006e5d7b447abd)
Signed-off-by: Roland Hieber <rhi@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a70a7376a8708bde07959deb5d5842d7f84ee5f8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're using the wrong data store when trying to locate siginfo files,
fix this. Thanks to Gregory Lumen <gregorylumen@microsoft.com> for
spotting.
[YOCTO #14774]
(Bitbake rev: 7eb0ef75fd08b6e4ca1e9dca9c96a7b590e5147b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0ed800e19a3197f8e622c8d3b630aae384e60aba)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The path has been encoded by urllib.parse.quote(), so decode it back for ssh.
Fixed when fetch from PREMIRRORS via ssh:
$ bitbake bonnie++ libsigc++-2.0 -cfetch
scp: /path/to/downloads/libsigc%2B%2B-2.10.7.tar.xz: No such file or directory
(Bitbake rev: 3969786a787eea34e096b932d52cd02978aacb8e)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c1c8fc678eb4783cea3974328a5fa8d1b79f1266)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This probably means the osc fetcher isn't being used but fix the missing
parameter.
(Bitbake rev: 73fbb743a2def2037d4053605e77e09d2d8a9fd0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a23c201cb6efc5c0abf763c26f905442f0eebb68)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is set at the start of the loop anyway so it does nothing. Drop
the pointless code.
(Bitbake rev: dcf78788daa177bf5c438f33b3c9f7ced7aea8ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e6a3173c9cdf349ccbd4cf612868f92cce8717c8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The function has a loop where the variable is never used which I was going
to fix but the entire function never seems to be called so remove it entirely.
(Bitbake rev: d739799a1e68dc2ad0414d4ae7d9e079cedcee3c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3bcb20f025907f4e88bbe3d14f5638d5f01010cb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix an issue where two tests have the same name with one overwriting the
other.
(Bitbake rev: 38842a8150f44f5d31d9bd2b0b6ec0502acc971d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit da812d938fd79e2cc7bdf355ccf5b0f9ead684c4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During parsing, Python raises
RuntimeError: dictionary changed size during iteration
in getRuntimeProviders, if you happen to have a recipe
with an explicit RDEPENDS on a dynamic package containing a '+'
character, such as 'gtk+3-locale-en'.
This is because we're using the modified pattern as the
key into the packages_dynamic dict to append to rproviders,
and since that key doesn't exist, the dict is getting modified
to add a new, empty, entry for it. So even without the runtime
error, we'd be generating an incorrect result.
Fix this by using a local variable for modifying the pattern
and using the original key to retrieve the value on a match.
(Bitbake rev: 69d3b86449be23b07f794e302f6e18f3a2c46424)
Signed-off-by: Matt Madison <matt@madison.systems>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 07de375c3e57f17ab7b47569186f24ecd9896825)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
logger.debug was giving an integer value (2) as event message, causing
knotty to crash when running with debug enabled.
bitbake/lib/bb/ui/knotty.py", line 685, in main
event.msg = taskinfo['title'] + ': ' + event.msg
TypeError: can only concatenate str (not "int") to str
Same issue also happens in the original code that was taken from
oe-core (openembedded-core/meta/lib/crate.py honister) / meta-rust.
(Bitbake rev: c212b0f3b542efa19f15782421196b7f4b64b0b9)
Signed-off-by: Ricardo Salveti <ricardo@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use bitwise operators to manipulate the received event mask in
_ProcessEvent.
Also minor clarification & clean up of the related comments.
(Bitbake rev: 2ab60c7be124d928d304ab1fb73f0dbff29964ae)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In `runtaskhashes`, the keys contain the absolute paths to the recipe. When
working with shared sstate caches (where these absolute paths can be different)
we see that compare_sigfiles does not identifiy a changed hash of a dependent
task as "changed", but instead as "removed"&"added", preventing the function
from recursing and continuing the comparison.
By calling `clean_basepaths` before comparing the `runtaskhashes` dicts, we
avoid this.
(Bitbake rev: 7358378b90b68111779e6ae72948e5e7a3de00a9)
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you use one of the functions from EXPORT_FUNCTIONS,
the meaning of cleandirs and fakeroot are lost. This leads to the function
changing in behaviour depending upon it's caller context. This isn't intended
so add mapping for the cleandirs and fakeroot flags too.
This does break devtool in OE-Core and there is a separate fix for that.
[YOCTO #8621]
(Bitbake rev: b074f4aff00923acc5bf6649d204d541a79fd2b6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GIT_SSH_COMMAND is more convinient to use if arguments have to be passed
and the user doesn't want to create a wrapper script around ssh.
(Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5)
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the owner information in the mirror tarballs generated using
BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit
0178ab83, which used the original pokybuild:user information, but failed
to clean up the numerical user and group ids. Now set the more canonical
values of oe:oe and 0:0.
(Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
CC: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are spaces in the URI filenames it can break the code.
We already solved this issue once somewhere else in the code so
use the same regex trick here as well.
We should ultimately refactor this code but at least fix the issue
for now.
(Bitbake rev: 57e2fc4d7f60afea4d4b2c84761324dd99e74a87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While the progress bar is good at conveying information about how much
of a task has executed, the elapsed time of the task is still very
much relevant to show.
(Bitbake rev: 41eeb4f34fb2306303f7688ec5e0ae965a573aa4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was only meant to be added for the handle_contains function in
a previous commit, fix it.
(Bitbake rev: 7399be398df39bc29e1b5eaac23b29cfae017abd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The kernel inotify code can set more than one of the bits in the mask,
fsnotify_change() in linux/fsnotify.h is quite clear that IN_ATTRIB,
IN_MODIFY and IN_ACCESS can arrive together. We don't care about two
of these from a bitbake perspective but it probably explains why in
real world builds, we've seen:
pyinotify.ProcessEventError: Unknown mask 0x00000006
This module code assumes only one mask bit can be present. Since we
don't care about two of these events, just mask them out for now.
The "upstream" code is unmainained since 2015.
(Bitbake rev: 7fb93c2ce6dacd9b53fc3a227133a3493e6a6a1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes can cause entire build trees to be monitored
which is suboptimal for performance.
Rather than trying increasingly convoluted tricks to try and handle add/removed
directories, rebuild the inotify watch when we reparse the configuration or
metadata.
(Bitbake rev: 3df322a200c28b45af1f2c92478c85eb7d20c38b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The python gc can trigger whilst we're holding the event stream lock
and when cleaning up objects, they can trigger warnings. This translates
into a new event which would then need the lock and we can deadlock.
Disable gc whilst we hold that lock to avoid this unfortunate and
problematic situation.
(Bitbake rev: 96a6303949cefd469bcf5ed250ff512271354357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding bb.utils.filter('WARN_QA', 'patch-fuzz', d) when WARN_QA is in
BB_BASEHASH_IGNORE_VARS or in vardepsexclude should not add a dependency
on WARN_QA.
Fix it and add some tests.
(Bitbake rev: 6aecc2fe51a52020f6f13be08449e18d42e7a6b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're still seeing issues with unclosed asyncio event loops. At the
init site, make sure any existing one is closed first to try and avoid
this.
(Bitbake rev: 78dee3c03c75a27531fcff26f9298fce2519bdde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we'd never have setup a siggen so don't error in that case.
(Bitbake rev: bbaaf2cf7b5a9339d3790610e622020c19d52f5a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure any exiting hash server connection is terminated before we start
a new bitbake session. This avoids errors seen with memory resident bitbake
when the asyncio event loop isn't closed correctly.
(Bitbake rev: 42ff9de77f24e2a0bec48a14b64c4b538e00b4af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When memory resident bitbake is active and we re-parse, the old module
configuration is present which can lead to strange errors. Reset this
when reparsing so the state is consistent. This fixes memory resident
bitbake errors.
(Bitbake rev: 951942c3c284ec2c62e730e145688033190af9b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix for inotify wasn't quite correct as we need to modify
bbseen before calling add_filewatch(). We also need to ensure the parse
mtime cache is cleared when directories are added/removed. There was also
a typo in the original fix and the wrong watcher was being changed. Fix
the various issues which improves memory resident bitbake testing results.
(Bitbake rev: 66cadd6be58bce5f7a56556cf92efd8159fb0b0e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Replace hyphens by em dashes when necessary
See https://www.grammarly.com/blog/hyphens-and-dashes/
- No uppercase after em dashes
- No uppercase after colons if what follows is not a
complete sentence.
- Fix spacing before colons ":"
- Replace em-dashes with colons for consistency in a section
(Bitbake rev: 72230d6a9976b3bfca1f1e6fb09736fec195e2fe)
(Bitbake rev: f1c4ac816e927f490fb9852c12aa408e8c9403b1)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "force" option to parser shutdown was often the cause of lockups and
there is no good reason we should have two different behaviours.
Change and unify the codepaths to always:
* Wait for longer for a controlled shutdown of a process (0.5s). Usually
it will be much faster if it has finished so the delay doesn't really matter.
* Send processes a SIGINT
* Failing that, send a SIGTERM
* Call .close() if available to release zombies
This means we no longer need the "force" parameter to the function so it is removed.
(Bitbake rev: de88c62ef873e9fce78ba162f5311d846de96f2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a parser process is terminated while holding a write lock, then it
will lead to a deadlock (see
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).
With SIGTERM, we don't want to terminate holding the lock. We also don't
want a SIGINT to cause a partial write to the event stream.
I tried using signal masks to avoid this but it doesn't work, see
https://bugs.python.org/issue47139
Instead, add a signal handler and catch the calls around the critical section.
We also need a thread lock to ensure other threads in the same process don't
handle the signal until all the threads are not in the lock.
(Bitbake rev: a40efaa5556a188dfe46c8d060adde37dc400dcd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not sure why this is so convoluted but we should simplify it!
(Bitbake rev: 6195343c46ba9d2685fc2d42366922f88ff3f369)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an exception occurs in the parsing process, ensure the cleanup
is called via all codepaths using a try/finally.
(Bitbake rev: e1ba2a69f1fc02f01a851bce20b1badf0b991f03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
