The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).
Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.
This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.
(cherry picked from commit cfae1556bf671acec119a6c8bbc4b667a856b9ae)
(Bitbake rev: 40fd5f4eef7460ca67f32cfce8e229e67e1ff607)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.
The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.
lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.
lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.
lfs=0 actually disables git-lfs functionality even if supported.
(cherry picked from commit d2be7f7f652360f13cd66d0850f3e19ffe2afb0a)
(Bitbake rev: 2a6fd774405a58f3ef2953a0dc3eca063e0cf890)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Not restoring the mocked _find_git_lfs leads to other tests
failing.
(cherry picked from commit 70f848631450bd723c223227c21c60e815ee033d)
(Bitbake rev: 1d5d4a46c772f44ab652f95b37f508b890828e67)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Python 3 interprets string literals as Unicode strings, and therefore
\s is treated as an escaped Unicode character which is not correct.
Declaring the RegEx pattern as a raw string instead of unicode is
required for Python 3.
(Bitbake rev: 7efed7bc09c56e41d3074a26388581a62f145625)
Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Specifically:
/srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
if isinstance(node.args[0], ast.Str):
/srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
value = node.args[0].s
(Bitbake rev: 6c19b6cf105ac321ec89da1a876a317020c45ab7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
(Bitbake rev: ee090484cc25d760b8c20f18add17b5eff485b40)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
A recipe variable handles its dependencies even on the "contains"
variables within the "inline Python expressions" like bb.utils.filter().
And it also handles those in the append operator correctly, but the
problem is that it does not so in the remove operator.
Fix it by adding the missing dependencies every time the remove
operator has been handled.
Also add a test case to check if the override operators handle
dependencies correctly.
(Bitbake rev: 23639edfbbb3fced7606dce211db8a31c5766585)
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cherry-picked from master: b90520eedb1dbc7f6a3928d089fe74fafb864eb5
- Conflicts in data.py are resolved as the master branch moved
handle_contains() and handle_remove() out of the try block.
- The test code in codeparser.py are modified as the master branch
added three more arguments to the build_dependencies().
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Verify that an existing layer path was given when adding a new
layer.
Manually using the shell for globbing is unnecessary, use the glob
function instead for cleaner code.
(Bitbake rev: de1cf0e31f3836a449cbd17490a2f6b6a86b17f9)
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
* it might be a bit confusing as it shows number of threads before making
the decision to start more tasks and also it can show only a few tasks
running, but not because of pressure when there just aren't many tasks
left or wait for their dependencies to be finished first
* example output:
NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads
NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads
(Bitbake rev: 42a1c9fe698a03feb34c5bba223c6e6e0350925b)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
* with latest bitbake I'm seeing very low number of bitbake tasks
executed in parallel, probably due to pressure regulation
show the values this is based on in the note
* also simplify a bit by counting the pressure and exceeds signs
only once
(Bitbake rev: 1050145c3004861ebede4777fd4fbd89d0470716)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The current calculation is not correct because if tdiff is less than
1.0, it's not taken into consideration when calculating the current
pressure.
Also, make it clear that the 1.0s is the psi accumulation cycle,
which might be changed in the future. We have this cycle because it
could largely avoid the 0 result issue, that is, if the interval
between checks are too small, the result might be 0. With this
accumulation logic, which has been there but let's make it clear,
this 0 result problem could be mitigated.
(Bitbake rev: 76889ff0a8938a3d77603d2af176aa9e264df839)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
It is currently hard to tell when bitbake is throttling task execution
due to system pressure changes. Add notes to the console output to make
this clearer, only generating output when the values change.
(Bitbake rev: b5f77e8159ad321f31999af8304f082a2c56b537)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The current PSI check calculation does not take into consideration
the possibility of the time interval between last check and current
check being much larger than 1s. In fact, the current behavior does
not match what the manual says about BB_PRESSURE_MAX_XXX, even if
the value is set to upper limit, 1000000, we still get many blocks
on new task launch. The difference between 'total' should be divided
by the time interval if it's larger than 1s.
(Bitbake rev: 2b2997ab6bdda730e4b638f416311a73e0c42156)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Error occured while running bitbake on cephfs:
WARNING: The free inode of path is running low (-0.001K left)
ERROR: Immediately halt since the disk space monitor action is "HALT"!
(Bitbake rev: a7f6c3e67bd9170e93b2b94676e84018faf0df91)
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Using multiconfig to target baremetal pieces of the system and building
corresponding toolchains for them results in hundreds and hundreds of
"Deferring %s after %s" and "Deferred task %s now buildable".
To clean up the output and to reduce risk of missing important warnings,
convert these notice messages to debug messages.
(Bitbake rev: 3505d8d8c02b041946670ab6bc5751e54fe292ff)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Denys Dmytriyenko <denys@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
If Tinfoil is initialized with setup_logging = False and
Tinfoil.prepare() is called with config_only = False, then it fails
because self.localhandlers is only initialized when
setup_logging = True.
This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`:
Traceback (most recent call last):
File ".../bitbake/bin/bitbake-getvar", line 41, in <module>
tinfoil.prepare(quiet=2)
File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare
for handler in self.localhandlers:
AttributeError: 'Tinfoil' object has no attribute 'localhandlers'.
Did you mean: 'oldhandlers'?
(Bitbake rev: e452c6d7ba5bb4f78a1d2bfb742794efdf171dbc)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 616101ddb630e2c9975022068b52a87c4cf647f6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
declaring queue=b"" creates an object of types bytes().
bytes() is an immutable object, and therefore doing "self.queue = self.queue + r"
creates a new object containing "self.queue" concatenated with "r".
On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker,
so those copies significantly slow down the initialization of the bitbake-worker.
Rather use bytearray() which a mutable type, and use extend() to avoid copies.
In my test setup, byterray.extend() is 10.000 times faster than copying the queue,
for a queue size of 180MB.
(Bitbake rev: 9993a89e5b97dda5f3657e5a7cc3a4fa94ff7111)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
If there are several multiconfigs in play for example a non-multiconfig with
a task with one hash and then three multiconfigs for the same task, different
architectures but the same hash (different to the non-mc), the three mcs
will be deferred until after the non-mc task but then will all run together
and race against each other.
Change the code to re-enable deferred tasks one at a time. This way, if they do
race, they won't run in parallel against each other.
(Bitbake rev: 907416ee1062f87f5844ab0638b54616abfc1a22)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
(Bitbake rev: 0c6f86b60cfba67c20733516957c0a654eb2b44c)
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a4ce040a6fd540a1cac52f808f909f9fcf8c961c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:
6f054f9fb3
But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:
* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink
(Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code appears to be dangerous, it swallows exceptions, turning them into
"handled" versions which then show no errors to the user. This is a pretty
poor user experience and I can't see why this code should be swallowing
such things. Drop the worst bits of code.
(Bitbake rev: a29ffcc12df24737e702e198aeb8f6892884932b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Martin Jansa reported that if you put a syntax error into an imported
module such as qa.py in OE, no error is shown.
Part of the issue appears to be that the catch_parse_error() decorator only
catches certain exceptions and SyntaxError isn't one of them. As far as I can
tell we should remove all the special cases and use the more advanced code
in all cases, not just expansion errors.
I confirmed this now prints a proper error message for a qa.py syntax error.
(Bitbake rev: e66012bfda60ffe1658473e25879aa67909ae65f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling SystemExit doesn't work well with server/client usage since the string
isn't printed to the right place. Use bb.fatal() instead which prints the right
log output and raises and handled exception which then shows correctly on the
UI.
(Bitbake rev: b9ae7164d9e744e8eb9aaff79218f57233a449b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some variables may be set as:
X = 1
as well the more usual
X = "1"
so add support to to_boolean to handle this case.
(Bitbake rev: e7df13a61911b7431802af2b4d7472b2aaf346fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to
be one character shorter and because fetcher uses:
if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
to decide which SRCREV values are fixed SRCREVs this one was
considered as tag or branch name, because it was only 39 chars long
The original error message wasn't very helpful as it doesn't show
which repo or which SRCREV was considered missing:
do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)
with SRCPV included in PV as error recomments it's a bit better:
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16
The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T
with this change the first error will read:
do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)
(Bitbake rev: 9bbdedc0ba7ca819b898e2a29a151d6a2014ca11)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
compare_sigfiles() recursively calculates differences on all dependent
tasks with changed hashes. This is done in arbitrary/alphabetical order, and
only the last of those results is returned, while everything else is discarded.
This changes the behavior to instead return the first difference and not calculate
any more, which significantly speeds up diffs of tasks with many dependencies.
(Bitbake rev: 89f13cd4a927a73de98998c27082c63b07671525)
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ea6a676c9aa2864c2eff40eea41ba09ce903a651)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case where hashlib is not available, the try would fail and fall
through resulting in a backtrace on the usage of the 'sig'. The backtrace
itself was confusing and made it difficult to determine what went wrong.
Update the import to be in it's own try block with an appropriate
message to indicate what went wrong.
Note, the current version of ply all of this code has been restructured
so this is not applicable upstream.
Additionally, some versions of hashlib don't appear to implement the
second FIPS related argument. Detect this and support both versions.
(Bitbake rev: d26ed38c233583b35ad1451e521f07d9c2329f4e)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 484ab42f440070c0369b81f5c69da860fa47a798)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython
has an optimization for statements in the form "a = a + b" or "a += b".
It seems that this line does not get optimized, because it has a form a = a + b + c:
data = data + "./" + f.split("/./")[1]
For that reason, it does a copy of data for each iteration, potentially copying megabytes
of data for each iteration.
Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds
instead of 45 seconds on my test setup where SRC_URI points to a big directory.
Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython:
"do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b"
However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup:
data = ''.join((data, "./", f.split("/./")[1]))
I have changed the other lines to also use += for consistency only, however those were in the form a = a + b
and were optimized already.
Co-authored-by: JJ Robertson <jrobertson@snap.com>
(Bitbake rev: 590ae6fde9da75db3a368e5c0d47920696c33ebf)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Namespace in this context means a branch, a tag, etc., clarify
it in the description. Also, fix a typo "a any", replace with
plain "any".
This patch is based of feedback on new applied patch
d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata")
(Bitbake rev: 86f2fa5261da959cda706c794a0047e5e89d4d6b)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every
single object in the remote repository. This works poorly with gitlab
and github, which use the remote git repository to track its metadata
like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/*
and refs/keep-around/* and they all contain massive amount of data that
are useless for the bitbake build purposes. The amount of useless data
can in fact be so massive (e.g. with FDO mesa.git repository) that some
proxies may outright terminate the 'git fetch' connection, and make it
appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such
that it only fetches refs/heads/* and refs/tags/* . Avoid using negative
refspecs as those are only available in new git versions.
Per feedback on the ML, Gerrit may push commits outsides of branches or
tags during CI runs, which currently works with the 'nobranch=1' fetcher
parameter. To retain this functionality, keep fetching everything in case
the 'nobranch=1' is present. This still avoids fetching massive amount of
data in the common case, since 'nobranch=1' is rare. Update 'nobranch'
documentation.
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
(Bitbake rev: c17fc1468ab84663b919e2809606b1b8ea2bebd9)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).
Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.
In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.
Also remove the return value as nothing ever checked it.
[ YOCTO #15000 ]
(Bitbake rev: ed0dcc40f80c48839bac20298013d70043858a4e)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression
in submodules path parsing. As the result gitsm fetcher fails on each
submodule which name begins from the name of the parent repo which is
totally valid usecase [Yocto #14045] [1]
Fix the code to error out only if submodule's name is equal to parent
name but not if it's part of it.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4
(Bitbake rev: 7e268c107bb0240d583d2c34e24a71e373382509)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We identified a use case where a native recipe (autoconf-native) was
rebuilt with no change in output yet the sstate for do_package tasks
wasn't being used.
The issue is that do_package tasks have a hard dependency on
pseudo-native:do_populate_sysroot. That task was one of the many
tasks being rehashed when autoconf-native's hash was changed.
If update_tasks processed a recipe before it had processed pseudo-native,
that recipe would be marked as not possible from sstate and would
run the full tasks.
The fix is to split the processing into two passes, first to handle
the existing covered/notcovered updates, then in the second pass,
check whether there are "harddep" issues.
This defers the do_package tasks until after pseudo-native is installed
from sstate as expected and everything works well again.
(Bitbake rev: 72a3afd99e8b785cb2a2f687e71a58e08cdd9c74)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e479d1e418a7d34f0a4663b4a0e22bb11503c8ab)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add
a parameter to the commands in the tests to allow this to continue to work.
(Bitbake rev: c90d57497b9bcd237c3ae810ee8edb5b0d2d575a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect_unix() call had a bug where if a relative path to a socket
was passed (which the non-async client always does), and the current
working directory was changed after the initial call, it would fail to
reconnect if it became disconnected, since the socket couldn't be found
relative to the new current working directory.
To work around this, change the socket connection for UNIX domain
sockets to be synchronous and change current working before connecting.
This isn't ideal since the connection could block the entire event loop,
but in practice this shouldn't happen since the socket are local files
anyway.
Help debugging and resolving from Joshua Watt.
(Bitbake rev: 6603c3e39f1cf746669ec6c9f0be8c6e6ece426e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5964bb67bb20df7f411ee0650cf189504a05cf25)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If submodule refers to specific revision of the parent repository it
causes deadlock in bitbake locking mechanism (lock is acquired to fetch
the parent and cannot be released before all submodules are fetched).
raise FetchError in such situation to prevent deadlocking.
[Yocto 14045]
(Bitbake rev: 987712c4c8fefd86a1f5116c11ee86e296e852ee)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hit this error while building nlf-native recently:
{
"error": {
"summary": "URI malformed",
"detail": ""
}
}
Some poking about led me to discover that:
1) The npm.py tool replaces npm:// with http://, not https://
2) Some versions of the npm tool don't handle 301 redirects properly,
choosing to display the above error instead when using the default
nodejs registry
It would be good to go fix npm to handle the redirect properly, but it
seems like it would also be good to assume secure http when contacting a
registry, hence, this patch
(Bitbake rev: bb5c43220f5f1c3d82334c65aff1ce13008db8d9)
Signed-off-by: Neil Horman <nhorman@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where copyright headers were not present, add them to make things
clear.
(Bitbake rev: e591325b2bd901c381003deb96a7b32a7148e93e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signature generation uses mkstemp() to get a file descriptor to a unique
file and then write the signature into it. However, the unique file name
generation in glibc is based on the system timestamp, which means that
with highly parallel builds it is more likely than one might expect
expected that a conflict will occur between two different builder nodes.
When operating over NFS (such as a shared sstate cache), this can cause
race conditions and rare failures (particularly with NFS servers that
may not correctly implement O_EXCL).
The signature generation code is particularly susceptible to races since
a single "sigtask." prefix used for all signatures from all tasks, which
makes collision even more likely.
To work around this, add an internal implementation of mkstemp() that
adds additional truly random entropy to the file name to eliminate
conflicts.
(Bitbake rev: 63bb5591e833de0e7b552963ad9bc4b39e56fda9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 97955f3c1c738aa4b4478a6ec10a08094ffc689d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm 99% certain this failing of a scenequeue task corrupts runqueue and
causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted
and did weird things so drop this code.
We've seen builds where the deadlock triggers and it then tries to run tasks
where the SQ task already ran with very confusing failures. It is likely it
is this code causing it.
(Bitbake rev: f386298fc056ef130c2eb6dabf25eafbd55f55ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8efced47fcb47851a370fd6786df6fb377f99963)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the deadlock breaking messages to be explict about which task is
blocked on which other task. The messages currently imply it is "freeing"
the blocking task which is confusing.
(Bitbake rev: d1b84e3cfe9fb8d282d4b700a9fe31891e00d837)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cf7f60b83adaded180f6717cb4681edc1d65b66d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have to prefer one multiconfig over another when deferring tasks, else
we'll have cross-linked build trees and nothing will be able to build.
In the original population code, we sort like this but we don't after
rehashing. Ensure we have the same sorting after rehashing toa void
deadlocks.
(Bitbake rev: 513bfd771d9095fcb6a8bf93806673dbf988a4de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we call the remove with recurse=True we first check if the
remove operation is safe in _check_unsafe_delete_path.
But the check is been done on the path instaed of the expanded
python glog.
(Bitbake rev: 280ea5a776436eab7e664fccea2df2e7ce47e586)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 7236488b898309ec5f1880936ddae22a28ccf5d3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the mirrors code is trying to create a symlink and the
parent directory doesn't exist, as might be the case for sstate
mirrors where the fetch is into a subdir, it can silently fail.
Ensure the directory exists in this case to avoid issues.
(Bitbake rev: ff3afb1c1bb236c4a52c62a74f2917071e0af55b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eff16e474ee7dc49ae433420a4c8d15d3314a618)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you trigger one of the comment errors, the newline characters
are stripped and the line numbers are incorrect. In one case it prints
the empty line which is also unhelpful.
Rework the code around these errors so the line numbers are correct
and the lines in question are more clearly displayed complete with newlines
so the user can more clearly see the error.
I also added a couple of simplistic test cases to ensure that errors
are raised by the two known comment format errors.
[YOCTO #11904]
(Bitbake rev: 01d27562c11d4b05eb30c7f9fefd58b6599fdd15)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 712da71b24445c814d79a206ce26188def8fce0a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As specified by git submodule manual relative urls can start either
with '..' or './', second case was incorrectly managed leading to an
interpretation of urls starting with './' as absoulte urls.
(Bitbake rev: d828cd2a16ddf4f084e61ffe44471483e132653a)
Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
