CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add
a parameter to the commands in the tests to allow this to continue to work.
(Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test for special characters in user and password to qualify
decodeurl() inspired by a bug report describing that '=' signs in a
password was problematic.
Add a second test to qualify decodeurl() as related to the change in
commit 628c4bf6c89b [fetch2/__init__: handle @ in package names].
Relates to [YOCTO #14476]
(Bitbake rev: ee04cf09c7022168c035affa654773652a49793e)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case where hashlib is not available, the try would fail and fall
through resulting in a backtrace on the usage of the 'sig'. The backtrace
itself was confusing and made it difficult to determine what went wrong.
Update the import to be in it's own try block with an appropriate
message to indicate what went wrong.
Note, the current version of ply all of this code has been restructured
so this is not applicable upstream.
Additionally, some versions of hashlib don't appear to implement the
second FIPS related argument. Detect this and support both versions.
(Bitbake rev: 484ab42f440070c0369b81f5c69da860fa47a798)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be used to enable the loopback interface, typically after calling
disable_network().
Also correct a typo in a debug message.
(Bitbake rev: 0d317209d4234c5f05a9fcdc13c52f502f104018)
Signed-off-by: Mattias Jernberg <mattias.jernberg@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of runfetchcmd is always empty in this case, as
the test doesn't produce any output.
SSH either returns 0 or 1, which is handled via exceptions.
This means the current check is not only unnecessary but prevents the
function from working.
We can just assume that if we reach the end of the function that the
file exists and return True.
(Bitbake rev: d599af48635fab587e5b913591b95daf87b40080)
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
c212b0f3 change the debug log level unintentional when tryng to fix a knotty issue.
This will maintain the same debug log level 2 as before.
(Bitbake rev: 19f8265023281f3b1d5d0a02e47f8d7d08cfcc16)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are many messed up calls for the debug log,
so is better to warm about this as they will not work
as expected.
The level need to be an integer and the msg a string.
(Bitbake rev: c1d9c1d25ce36848040dc0ce182835e497ccbb82)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
f68682a7 ("logging: Make bitbake logger compatible with python logger")
replaced several .debug() calls to make them comply with the standard
python logging API, but a few were missed.
(Bitbake rev: eb25cd4d64b9a4e8e2b2ce7fbccc123d00b2fc2b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The condition will always evaluate to true and
thus is redundant.
(Bitbake rev: be1ee681e8a566564549068dcf90c95c36544815)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
local file fetches now validate checksums. The checksums for mirror
tarballs of repositories will not match so ignore these checksums.
(Bitbake rev: 6424f4b7e9c1ba8db81346e8b3a806dd035d4551)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect_unix() call had a bug where if a relative path to a socket
was passed (which the non-async client always does), and the current
working directory was changed after the initial call, it would fail to
reconnect if it became disconnected, since the socket couldn't be found
relative to the new current working directory.
To work around this, change the socket connection for UNIX domain
sockets to be synchronous and change current working before connecting.
This isn't ideal since the connection could block the entire event loop,
but in practice this shouldn't happen since the socket are local files
anyway.
Help debugging and resolving from Joshua Watt.
(Bitbake rev: 5964bb67bb20df7f411ee0650cf189504a05cf25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If two recipes have conflicting checksums for a file, the code will currently
remove the existing file when a mismatch is downloaded, even if another task
successfully fetched it.
This changes the code to verify the checksum (if possible) before replacing
the file. This removes a potential race window and stops builds failing
everywhere from one incorrect checksum.
To make this work, we need to be able to override localpath and avoid
NoChecksum errors being logged.
(Bitbake rev: 4b8de2e7d12667d69d86ffe6e9f85a7932c4c9a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PN is correct here, bitbake has no knowledge of BPN.
This reverts commit d613e48c07d4b12219270c1359cbf2f390b848dd.
(Bitbake rev: cffcfacb747d41304c857b17bfea646e220b2389)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When checking for the non-existing file, BPN is actually the acutal recipe
name. And we should use BPN for the error message and it also fix the below
test when multilib is enabled.
$ oe-selftest -r bbtests.BitbakeTests.test_invalid_recipe_src_uri
(Bitbake rev: d613e48c07d4b12219270c1359cbf2f390b848dd)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The class and exception aim to test rare cases there deadlocks are
possible.
Can be used in context managers:
with Timeout(<value>):
do_deadlock()
(Bitbake rev: c5fcdd804d422f959a189b270d72123a50e74da6)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If submodule refers to specific revision of the parent repository it
causes deadlock in bitbake locking mechanism (lock is acquired to fetch
the parent and cannot be released before all submodules are fetched).
raise FetchError in such situation to prevent deadlocking.
[Yocto 14045]
(Bitbake rev: 0361ecf7eb82c386a9842cf1f3cb706c0a112e77)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hit this error while building nlf-native recently:
{
"error": {
"summary": "URI malformed",
"detail": ""
}
}
Some poking about led me to discover that:
1) The npm.py tool replaces npm:// with http://, not https://
2) Some versions of the npm tool don't handle 301 redirects properly,
choosing to display the above error instead when using the default
nodejs registry
It would be good to go fix npm to handle the redirect properly, but it
seems like it would also be good to assume secure http when contacting a
registry, hence, this patch
(Bitbake rev: 2cd76e8aabe4e803c760e60f06cfe1f470714ec7)
Signed-off-by: Neil Horman <nhorman@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Plugins may want to use it (e.g. the layers-setup plugin that would
want to discover writer sub-plugins with it), and so it makes sense
to make tinfoil available a bit eariler.
(Bitbake rev: 2f6c7523a622f59ddf84a1a196927492bc5fa7a2)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass additional arguments in the fileslocked() context manager to the
underlying lockfile() function. This allows the context manager to be
used for any types of locks (non-blocking, shared, etc.) that the
lockfile() function supports.
(Bitbake rev: 7a8eb8da8e8495051e174721062da08e06168024)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user does need to be told about this but it isn't really a warning,
just something they may need to be aware of. Drop the level accordingly.
(Bitbake rev: 9bdedc8074990e613c9567e2cd8072f8d885f07f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have some confusion for users since some classes are meant to work
in the configuration space (or "globally") and some are meant to be
selected by recipes individually.
The cleanest way I could find to clarify this is to create "classes-global"
and "classes-recipe" directories which contain the approproate classes and
have bitbake switch scope between them at the appropriate point during
parsing. The existing "classes" directory is always searched as a fallback.
Once a class is moved to a specific directory, it will no longer be found
in the incorrect context. A good example from OE is that
INHERIT += "testimage"
will no longer work but
IMAGE_CLASSES += "testimage"
will, which makes the global scope cleaner by only including it where it
is useful and intended to be used (images).
(Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where copyright headers were not present, add them to make things
clear.
(Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than recursing into the conf handler code, simply call into
the parse code directly when inheriting files as we've already resolved
the paths and don't need anything the other codepath brings. This
makes the codepath clearer at the expense of some slight duplication.
(Bitbake rev: 0f4f3af6d93a0018df58b8a1d8d423c78ba6526d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than relying on later code to error if the class isn't found,
exit earlier and more clearly from a code perspective.
(Bitbake rev: f7c55c8147329670fd5bc55b1ae3f47f25b89bab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful when debugging as it helps understand possible
race conditions between tasks of diferent recipes.
(Bitbake rev: 950a2ea4c91d6e13d7587104367fa85cc7efe01c)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent new tasks from being scheduled if the memory pressure is above
a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY"
variable in the conf/local.conf file. This is an extension to the
following commit and hence regulates pressure in the same way:
48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation
Memory pressure is experienced when time is spent swapping, refaulting
pages from the page cache or performing direct reclaim. This is why
memory pressure is rarely seen but might be useful as a last resort to
prevent OOM errors.
(Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signature generation uses mkstemp() to get a file descriptor to a unique
file and then write the signature into it. However, the unique file name
generation in glibc is based on the system timestamp, which means that
with highly parallel builds it is more likely than one might expect
expected that a conflict will occur between two different builder nodes.
When operating over NFS (such as a shared sstate cache), this can cause
race conditions and rare failures (particularly with NFS servers that
may not correctly implement O_EXCL).
The signature generation code is particularly susceptible to races since
a single "sigtask." prefix used for all signatures from all tasks, which
makes collision even more likely.
To work around this, add an internal implementation of mkstemp() that
adds additional truly random entropy to the file name to eliminate
conflicts.
(Bitbake rev: 97955f3c1c738aa4b4478a6ec10a08094ffc689d)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm 99% certain this failing of a scenequeue task corrupts runqueue and
causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted
and did weird things so drop this code.
We've seen builds where the deadlock triggers and it then tries to run tasks
where the SQ task already ran with very confusing failures. It is likely it
is this code causing it.
(Bitbake rev: 8efced47fcb47851a370fd6786df6fb377f99963)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the deadlock breaking messages to be explict about which task is
blocked on which other task. The messages currently imply it is "freeing"
the blocking task which is confusing.
(Bitbake rev: cf7f60b83adaded180f6717cb4681edc1d65b66d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have to prefer one multiconfig over another when deferring tasks, else
we'll have cross-linked build trees and nothing will be able to build.
In the original population code, we sort like this but we don't after
rehashing. Ensure we have the same sorting after rehashing toa void
deadlocks.
(Bitbake rev: 27228c7f026acb8ae9e1211d0486ffb7338123a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent the scheduler from starting new tasks if the current cpu or io
pressure is above a certain threshold and there is at least one active
task. This threshold can be specified through the
"BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf.
The threshold represents the difference in "total" pressure from the
previous second. The pressure data is discussed in this oe-core commit:
061931520b buildstats.py: enable collection of /proc/pressure data
where one can see that the average and "total" values are available.
From tests, it was seen that while using the averaged data was somewhat
useful, the latency in regulating builds was too high. By taking the
difference between the current pressure and the pressure seen in the
previous second, better regulation occurs. Using a shorter time period
is appealing but due to fluctations in pressure, comparing the current
pressure to 1 second ago achieves a reasonable compromise. One can look
at the buildstats logs, that usually sample once per second, to decide a
sensible threshold.
If the thresholds aren't specified, pressure is not monitored and hence
there is no impact on build times. Arbitary lower limit of 1.0 results
in a fatal error to avoid extremely long builds. If the limits are higher
than 1,000,000, then warnings are issued to inform users that the specified
limit is very high and unlikely to result in any regulation.
The current bitbake scheduling algorithm requires that at least one
task be active. This means that if high pressure is seen, then new tasks
will not be started and pressure will be checked only for as long as at
least one task is active. When there are no active tasks, an additional task
will be started and pressure checking resumed. This behaviour means that
if an external source is causing the pressure to exceed the threshold,
bitbake will continue to make some progress towards the requested target.
This violates the intent of limiting pressure but, given the current
scheduling algorithm as described above, there seems to be no other option.
In the case where only one bitbake build is running, the implications of
the scheduler requirement will likely result in pressure being higher
than the threshold. More work would be required to ensure that
the pressure threshold is never exceeded, for example by adding pressure
monitoring to make and ninja.
(Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <randy.macleod@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we call the remove with recurse=True we first check if the
remove operation is safe in _check_unsafe_delete_path.
But the check is been done on the path instaed of the expanded
python glog.
(Bitbake rev: 7236488b898309ec5f1880936ddae22a28ccf5d3)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds TCP Keep Alive support to the async RPC server. This should help
prevent file descriptor exhaustion on the server when client connections
are interrupted and the socket never closes (e.g. no FIN is sent from
the client).
A keep alive is sent after 30 seconds of inactivity, then every 15
seconds after that up to a maximum of 2 minutes.
(Bitbake rev: 68f4ce662cad28fed739900addbdee949ad3c1e8)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A long time ago, we made DL_DIR a final fallback for the local fetcher.
Since then we added checksum support and task hashes and the world
has changed. There were warnings added some time ago if this fallback
triggers and it is now time to drop it entirely.
The original use case was for sstate however the sstate code now sets
FILESPATH correctly so DL_DIR is no longer needed.
There have been a few small bugs exposed by this change, missing mkdir
calls and some minor test issues that needed tweaks. In general this
simplifies and improves the fetcher code flow though.
This completes a cleanup that ensures local files are correctly covered
at parse time which ensures rebuilds and reparses happen at the right
times.
(Bitbake rev: 3e1444e536c71d3885ef6b9d428807163c309640)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the mirrors code is trying to create a symlink and the
parent directory doesn't exist, as might be the case for sstate
mirrors where the fetch is into a subdir, it can silently fail.
Ensure the directory exists in this case to avoid issues.
(Bitbake rev: eff16e474ee7dc49ae433420a4c8d15d3314a618)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the local fetcher was not able to find the file anywhere but it
was included in the SRC_URI for checksumming just make it a fatal
error. Ensure a list of searched locations is included too to match
the runtime error that would have resulted.
(Bitbake rev: 5e3b2ad90d9cd0f248b1cb740637caa24442d101)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to
be one character shorter and because fetcher uses:
if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
to decide which SRCREV values are fixed SRCREVs this one was
considered as tag or branch name, because it was only 39 chars long
The original error message wasn't very helpful as it doesn't show
which repo or which SRCREV was considered missing:
do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)
with SRCPV included in PV as error recomments it's a bit better:
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16
The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T
with this change the first error will read:
do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)
(Bitbake rev: 04dc17bef9b762cef9eecdf91c9f37738d8ae44d)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you trigger one of the comment errors, the newline characters
are stripped and the line numbers are incorrect. In one case it prints
the empty line which is also unhelpful.
Rework the code around these errors so the line numbers are correct
and the lines in question are more clearly displayed complete with newlines
so the user can more clearly see the error.
I also added a couple of simplistic test cases to ensure that errors
are raised by the two known comment format errors.
[YOCTO #11904]
(Bitbake rev: 712da71b24445c814d79a206ce26188def8fce0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
