On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.
Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.
(Bitbake rev: 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tests quotes around `` and $() expansions, nested and multiple
expansions, and that escaped quotes are treated as characters by the
parser.
(Bitbake rev: d98130cb4d500c495bc692c56dde3e019f36320a)
Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add return false to supports_checksum for svn fetcher which fhis
fixes MIRROR usage for svn uris. Also add a testcase.
[YOCTO #15473]
(Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f)
Signed-off-by: Kari Sivonen <kari.sivonen@live.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.
For example: http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:
File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
self.query = self._param_str_split(urlp.query, "&")
File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)
In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.
The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.
Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().
(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).
Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.
This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.
(Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before, everything up to the last slash was removed when extracting the
names of the tags. This would lead to that a tag such as "agent/11.0.0"
would be incorrectly identified as "11.0.0", which would then be treated
as a correct version matching "^(?P<pver>\d+(\.\d+)+)".
(Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases, the version found by latest_versionstring() may be higher
than the real version. Make it possible to specify a maximum version so
that this case can be detected.
(Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch'
failed even with all contents available in PREMIRRORS.
Add a test case to ensure no regression in the future.
(Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fetch from premirror few times to emulate multiple machines sharing same
clonedir or few rebuilds of the package from (pre)mirror
Regression test for [Yocto #15369]
(Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When git is configured with safe.bareRepository=explicit [1], the
bitbake selftests fail miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: a45e14a7343e36101e45639931322e5649587f57)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* github no longer provides support for subversion clients:
https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients
it was shut down on 2024-01-08:
https://github.blog/2023-01-20-sunsetting-subversion-support/
and this test was now failing with:
======================================================================
ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn
fetcher.download()
File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download
raise FetchError("Unable to fetch URL from any source.", u)
bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source.
Stdout:
Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2
Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output:
A trunk/README.md
U trunk
Checked out revision 2.
svn: warning: W205011: Error handling externals definition for 'trunk/bitbake':
svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git'
svn: E205011: Failure occurred processing one or more externals definitions
in the rare cases where subversion was still installed on the host
running bitbake-selftest :).
to avoid this use still alive repository from https://svn.apache.org/
and pick something rather small and only the trunk subdirectory which
is fast to fetch:
svn co https://svn.apache.org/repos/asf/serf/trunk
takes just 2 sec here
adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of
"trunk/README"
(Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617)
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have two classes, both of which set EXPORT_FUNCTIONS for the same funciton
and a standard funciton definition for the function that is exported, the export
function can sometimes overwrite the standard one.
The issue is that the internal flag the code uses isn't ovweritten if the variable
is giving a new value. Fix the issue by using a comment in the code that is injected
so that we know if it is ours or not.
Also add some testing for EXPORT_FUNCTIONS, not perfect but a start.
(Bitbake rev: 66306d5151acb0a26a171c338d8f60eb9eb16c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add test_lineno_in_eventhandler to test lineno in eventhandler.
(Bitbake rev: 4e5de537bebb68180c5755858c81b095eb9ae2f6)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Found a duplicate test, added _2 suffix to one, 74 tests now pass up from 73.
(Bitbake rev: ae2a19dadb4f3065b8731a61f45f29e6a70af402)
Signed-off-by: Richard Haar <rh@richhaar.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When downloading a Git repository containing an LFS, the Git hooks
are not always initialized correctly to perform the download. This
change updates the Git downloader to run the "git lfs install" command
in order to "smudge" the LFS files. The tests have also been updated
to do checks for situations in which git lfs is not installed, as
the application was required to be installed for any of the tests
to run previously.
The Git LFS functionality was working to some degree previously,
but this change also updates the fetcher to allow LFS downloads for
nobranch SRC_URIs.
(Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217)
Signed-off-by: Desone Burns II <dburns@seegrid.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create repository with few branches and test if fetcher can work
with such repository as PREMIRROR
(Bitbake rev: a1737610e5d5b61e126ec3632d7f27b337a87818)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The infrastructure changes highlighed an issue where the top level directory
mtime stamp isn't included in the tarballs. This isn't an issue the test is
meant to check for so ignore it.
(Bitbake rev: 5a1e5b8dc55d6bb4d93bac3492f8c43ff957e712)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A recipe variable handles its dependencies even on the "contains"
variables within the "inline Python expressions" like bb.utils.filter().
And it also handles those in the append operator correctly, but the
problem is that it does not so in the remove operator.
Fix it by adding the missing dependencies every time the remove
operator has been handled.
Also add a test case to check if the override operators handle
dependencies correctly.
(Bitbake rev: b90520eedb1dbc7f6a3928d089fe74fafb864eb5)
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runtaskdep data in siginfo files was written out with full paths to
the bb files, matching bitbake's internal "unique key" ID for recipes/tasks.
When originally implemented this made sense.
Over time, the main use for the data in siginfo files has become to match
against other siginfo files to debug changes of hash calcuations. The
recipename data is not useful for this as the siginfo filenames use PN
instead which can often be derived from the recipe filename but not always.
It is time to throw away the 'tid' data format and switch over the use a
hybrid PN form which includes the multiconfig. That can be easily stripped
off in the find_siginfo code in oe-core.
The other purpose of having a sortable dependency ID is retained and the
multiconfig needs to be included to allow the taskhashes to be processed
and calculated correctly. PN is meant to be unique between recipes, only
one would ever be built so using PN in this location is fine.
The one risk of this change is there isn't any compatibility to the old
format. I'm not convinced we should spend time complicating the code
with it. This change will change the taskhashes everywhere so the only
mixing of old and new siginfo files will be either through hash equivalence
or through users using the tool against old and new info files manually
which will give some weird output but it should be clear they're in
different formats as there would be large paths from the old files not
present in the new ones.
We have options to add backwards compatibility if some issue is found
to need that.
(Bitbake rev: 637933e2e5a59228a8d17aae4160551cab5f2f61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test_python_snippet_function_reference to not require the 'func'
flag, now that we know the real function will be returned for context
functions without the flag.
(Bitbake rev: 83f41281ec3d9b4327ffc8e2312e1fb8f53cbf02)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test makes sure that '${@eval()}' calls the eval builtin, even if an 'eval' variable is defined in the metadata.
(Bitbake rev: e9150447738a48f772240874b3512b08e982b19b)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that any change to the presence of builtins in inline
python execs will be noticed.
(Bitbake rev: ee22d3d51c60db2da97422b2be1e42239b7a2324)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a test for an issue seen long ago, to avoid regressions, where a
reference to a def'd function in the metadata would return the string
value from the metadata rather than the function in inline python.
(Bitbake rev: 9f7cb22febd557817c164e25a93f5660e9c06358)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.
(Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19)
Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, pytest captures all stdout and exposes it using its built-in
fixtures (capsys, caplog etc), so stdout does not support getvalue().
To support running tests using both unittest and pytest, use assertLogs
to capture logging and assert on the log output instead.
(Bitbake rev: 2d28caa01bab9540d2bbaf713ae3e5c563d003f5)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git config returns an error code if user.email or user.name are not set,
so bb.process.run raises an ExecutionError, failing the test.
Improve the logic by catching the ExectionError and using this to set a
default value.
Change a direct invocation of 'git init' to use self.git_init so that
it follows the code path above.
Remove the related comment from the README now that git sets up the user
details automatically.
(Bitbake rev: 0c7079d0609a13713b890a9675abd9fba032f199)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TestCase.assertRaisesRegexp was renamed to assertRaisesRegex in Python
3.2, so rename to fix a warning during test execution.
(Bitbake rev: 9bc2ca3197bff7a5ef6e0ece6bfef0cda68b808e)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
pytest test collection attempts to collect all classes containing
"Test". TestTimeout is not a test class so add the __test__ = False
attribute to indicate this.
(Bitbake rev: b5455b98093a3eaf122ebe3d29c3bef949fd2440)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a host where the git config for initial branch
name is something else than 'master' the unittest will fail
as they assume the default branch name is 'master'. Fix this
by explicitly set the intial branch name at 'git init'.
(Bitbake rev: 5bcd213c23da30a84baba88b775b1740e6bb77d0)
Signed-off-by: Peter Bergin <peter.bergin@windriver.com>
Signed-off-by: Peter Bergin <peter@berginkonsult.se>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The crate fetcher handles a crate with a name that matches the recipe's
name specially by placing the unpacked code in the current directory
(which typically is ${S}) rather than together with the sources for the
other crates. This broke when the URI names for all crates were changed
recently to include the version in the name.
Correct the crate fetcher to test against ${BP} instead of ${BPN}.
Also add a test case to the selftests to avoid this breaking again.
[Yocto #15012]
(Bitbake rev: 9af52b748357d0c843ce2507ce4d119fd9c37008)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recommendation from server maintainers is that the https protocol
is both faster and more reliable than the dedicated git protocol at this point.
Switch to it where possible.
(Bitbake rev: 879b830fcc60a022682ef22f58c04e7e07d03c63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update regex pattern to allow variable flag name with a single character.
Regression tests have also been updated in `bb.parse` and
`bin/bitbake-selftest -k ParseTest` has been successfully executed.
Eliminate a trailing space as well.
(Bitbake rev: bb9e523291a3cad6e1596ee6a1e715b5e5feba8f)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is common for rust packages to depend on different versions of the
same crate. E.g.
| crate://crates.io/windows_x86_64_msvc/0.42.2 \
| crate://crates.io/windows_x86_64_msvc/0.48.0 \
Identification only by the plain crate name makes the sha256sum
ambiguous
| SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
| SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"
and requires lot of manual work to fix the SRC_URI list.
Making the 'name' property unique by appending the version allows
direct copy & paste of reported sha256sum errors to complete the
crates list.
(Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change brings checksum verification of each crate
in a recipe, e.g
| SRC_URI += " \
| crate://crates.io/aho-corasick/0.7.20 \
| crate://crates.io/atomic-waker/1.1.0 \
| crate://crates.io/cc/1.0.79 \
| "
|
| SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
| SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599"
| SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f"
That will require to move the checksum initialization
after the possible call to urldata_init method in order
for the crate fetcher to parse the url.
Another way of doing could have been implementing a decodeurl
method that would have been specific for each fetcher class.
(Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch enables the usage of the '@' character in variable flag
names.
One use case of variable flags is to assign the network namespaces of
some systemd services/targets to configure other parts of the build
process of some system. The filenames of systemd services/targets might
contain the '@' character if they are template unit files that can take
in a single parameter/argument and be instanced multiple times, as
indicated by systemd's official manual page.
The '@' character is disallowed as the first character in a variable
flag name. Imposing more restrictions on the first character is a
compromise to make parsing easier and to allow for more options in the
future to extend the syntax.
This patch is successfully verified by creating a custom BitBake recipe
that sets and unsets the value of a variable flag with the '@' character
in its name and ensuring that no ParseError is being thrown. Regression
tests have also been added to `bb.parse`.
`bin/bitbake-selftest` has also been successfully executed and all tests
passed.
(Bitbake rev: 00f9ab2cacfbd2a63b6b4959cf5401babae7e32a)
Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For readability of following patches, rename this internal variable to
allow for others in a similar format.
(Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allow to have classic fetch parameters
(like destsuffix, sha256, name ...) not being
considered by crate fetcher itself (and so mess
up its download)
Moreover, it allow to overload the name of the downloaded
crate (maybe usefull if there is a naming clash between
two crates coming from different repositories)
(Bitbake rev: 278bd2f1758b8af97552af8d23d16ffb5127a131)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From the npm-install documentation [1] the CLI provides a set of
short forms when the install fetches from git. These include
"github:"
example: npm install github:mygithubuser/myproject
"gist:"
example: npm install gist:101a11beef
"gitlab:"
example: npm install gitlab:mygitlabuser/myproject
"bitbucket:"
example: npm install bitbucket:mybitbucketuser/myproject
Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in
npm shrinkwrap version] by Stefan Herbrechtsmeier added support for
the "github:" but the others would marked as 'Unsupported dependency'.
The other prefixes are added in this commit, along with extending the
tests to cover some of these.
However, there is one more short form for github which npm-install
allows which forgoes the prefix altogether.
example: npm install mygithubuser/myproject
Unfortunately this format is a bit problematic as it lacks any easily
identifiable 'marker' to match against, and it could be either the
github short form or install from folder format. Experimentation shows
that the folder format requires a leading './' or '/', so we use this
to rule out the ambiguity.
If this approach to folder and github formats disambiguation is
incorrect it won't matter anyways as the folder format is unrecognized
by the code as-is and thus with this change or without, things would
fail.
Since we have to be less strict in the check for git operations we
move it to be the last install format which we check, such that the
less ambiguous formats can be sorted out first.
[1] https://docs.npmjs.com/cli/v9/commands/npm-install
[Yocto #14236]
(Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.
The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.
lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.
lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.
lfs=0 actually disables git-lfs functionality even if supported.
(Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not restoring the mocked _find_git_lfs leads to other tests
failing.
(Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the codeparser cache ends up being extended for every parse run
since there are values in the functions such as the result of os.getpid()
from LOGFIFO in OE-Core.
Digging into that issue, there are also lots of similar but different
functions being parsed where the change might just be a path to WORKDIR,
a change in PN or PV or something like DATE/TIME.
There is no reason we have to use these changing values when computing the
dependenies of the functions. Even with a small tweak like:
BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn"
the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run
to around 1.3MB and remaining static for oe-core and meta-oe. In my local
build, admittedly heavily experimented with, the cache had grown to 120MB.
The benefits of doing this are:
* faster load time for bitbake since the cache is smaller to read from disk
and load into memory
* being able to skip saving the cache upon shutdown
* lower memory footprint for bitbake
* faster codeparser data lookups (since there is less data to search)
We only use these special values when passing code fragments to the codeparser
to parse so the real variable values should otherwise be used in the hash data.
The overall effect of this change, combined with others to avoid saving unchanged
cache files can be ~2s on a ~16s parse on my local system and results in a more
responsive feeling bitbake. It also allows parsing performance to be investigated
more consistently.
(Bitbake rev: f24bbaaddb36f479a59a958e7fc90ef454c19473)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While the parameter is not required it allows testing of possible
regression in fetcher code when parameter specified and mirrors are
used.
(Bitbake rev: 1a21918049091a6d77426dbf8868ffdc14ba1003)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Basic test for mercurial functionality including PREMIRRORONLY with
disabled network
(Bitbake rev: c0e349a5fc92e9c90afc53800c7e83495393ff79)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the move to a server idle thread, we always need threading. The
existing accessor functions could end up turning this off!
I was going to hold the lock whilst changing it, check if the value
was already set, cache the result and also fix the event code to always
release the lock with a try/finally.
Instead, disable the existing functions and use a with: block
to handle the lock, keeping things much simpler.
(Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've been seeing event handlers where 'd' seems to disappear half way through
event handler execution. This is problematic when multiple threads are active
since this code assumes single threading.
The easiest fix is to change the handler function calls to contain d as a
parameter as we do elsewhere for other functions. This will break any non-text
handlers but I was only able to spot one of those in runqueue. It will also
break handlers than call functions that assume 'd' is in the global namespace
but those failures should be obvious and we can fix those to pass d around.
This solution avoids manipulating builtins which was always a horrible thing
to do anyway and solves the issue without needing locking, thankfully.
(Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moving code into python modules is a very effective way to reduce parsing
time and overhead in recipes. The downside has always been that any
dependency information on which variables those functions access is lost
and the hashes can therefore become less reliable.
This patch adds parsing of the imported module functions and that dependency
information is them injected back into the hash dependency information.
Intermodule function references are resolved to the full function
call names in our module namespace to ensure interfunction dependencies
are correctly handled too.
(Bitbake rev: 605c478ce14cdc3c02d6ef6d57146a76d436a83c)
(Bitbake rev: 91441e157e495b02db44e19e836afad366ee8924)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All siggens in common use should now support multiconfig, drop the
compatibility code.
(Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When obtaining latest upstream versions, the code needs
to check if the existing tarball is in a versioned directory
(e.g. component-name/x.y/component-name-x.y.z.tar.gz) and
if it is, it needs to first obtain the list of all
such versioned directories and then check all of them by going
one step up in the directory hierarchy.
Existing code was returning a correct match when the component
name did not have numbers, e.g. a check on 'source/epiphany/43/'
would return 43, but was stopping too soon when the component
name itself had numbers ('source/libxml2/2.10/' would return libxml2).
This change ensures the last match is taken instead of the first.
Also, adjust the fetcher tests to check that versioned directories
are correctly traversed in this case (e.g. the step to go one level
up is taken and a new tarball is discovered in a different versioned
directory).
(Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
