By switching from print() to bb.plain() to print failing task logs, we
allow them to be saved in BB_CONSOLELOG.
Fixes [YOCTO #15798]: This allows AB reproducibility test to save the
full log of the failing tasks and helps debugging.
(Bitbake rev: d3b3ad32da7c7ebf61814fc807f8667a37aa149b)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test case to ensure git shallow fetch succeeds for SRC_URI
with tag containing slash.
For example, we want to succeed for SRC_URI like below:
SRC_URI = "git://salsa.debian.org/debian/debianutils.git;protocol=https;branch=master;tag=debian/${PV}"
See the following link for more information:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15862
(Bitbake rev: 919d4cf6e688e67229c46d30c84d523b21936377)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a tag contains slash, e.g., debian/5.22, then shallow clone
fails because it's using a wrong ref.
To reproduce the issue, add the following lines in local.conf:
BB_GIT_SHALLOW = "1"
BB_GENERATE_SHALLOW_TARBALLS = "1"
And then run 'bitbake debianutils -c fetch'.
What the original os.path.basename(ref) wanted to do is to remove
the strings such as refs/heads/. So we do it explitly to fix this
issue.
Fixes: [YOCTO #15862]
(Bitbake rev: c6d6999f1ed01e7445b8f177a888038edacf555c)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test case to verify that the gitsm fetcher properly handles
repositories storing objects with LFS.
The test case verifies that LFS objects are fetched on the initial clone
but also ensures that consecutive updates extend the original clone with
any newly referenced LFS objects.
(Bitbake rev: 2a8722ddd155596862029f6ea34e1e92c77e0b7f)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Skipping smudging prevents the LFS objects from replacing their
placeholder files when `git submodule update` actually checks out the
target revision in the submodule. Smudging cannot happen earlier as the
clone stored in `.git/modules` is bare.
This should be fine as long as all LFS objects are available in the
download cache (which they are after the other fixes are applied).
(Bitbake rev: d270e33a07c50bb9c08861cf9a6dc51e1fd2d874)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code would pass `True` or `False` to the git fetcher. As
the fetcher expects `lfs` to be set to `1` this always lead to LFS
fetching being disabled.
(Bitbake rev: 5e487a5a096400271ed1e29b0df72903f2304e49)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its not clear which exact git-lfs versions failed to work with bare
repositories, however git-lfs 2.13.2 which is shipped by Debian 10 (i.e.
the oldest supported distribution by scarthgap) shows no issue when
fetching into a bare repository. Switch to git-lfs fetch which in turn
eliminates issues seen when using the gitsm fetcher with submodules
utilizing lfs.
In these scenarios, fetching of LFS objects did not actually happen as
the gitsm fetcher parametrizes the to be fetched repositories with
`bareclone=1` which in turn means that the target revision was never
checked out (and therefore no LFS objects were fetched).
(Bitbake rev: 501d3317df5780a5b0e3c2562b26ed276c1dbe90)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its unclear why this function does not operate on the desired source
revision to begin with (which really should be the decider on whether a
particular source revision uses LFS or not). Simplify the decision logic
by always checking the `.gitattributes` file of the target revision.
(Bitbake rev: b3faa0ce5c0a6945f26b5b303a7f38c00d132397)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So far, existence of `git-lfs` was only checked during unpacking. As the
binary is also used in earlier steps also check for its existence there.
Additionally, factor out the LFS existence check into a dedicated
function and call it wherever git-lfs is used for the first time.
(Bitbake rev: 5818367db9b261b7e07c347d38044e6cba8f9727)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid multiple import statements for anything that is used more than
once. Additionally, drop no longer used imports.
(Bitbake rev: 7c74310440f4d6ec47cf5bacf597e18308b3bb20)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no need to pass `name` when it is no longer used.
(Bitbake rev: b132d35dee643e270e3e6dd536dcc90334a0111c)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Format the docstrings of the utils modules to be automatically
documented with the autodoc Sphinx extensions. Sphinx syntax can be used
in those for proper formatting. Cross-referencing with :term: is not
possible in these.
(Bitbake rev: 2fa1c7ad43639c6d25c94b7794bcce5f5ff74e10)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lock_timeout_nocheck() can be interrupted immediately after enterring
the try-block and prior to initializing 'l', for example with a
ctrl-C, the code in finally will still be run and the 'if l' will
fail. Initialize 'l' as False to avoid this possiblity.
(Bitbake rev: 4885cd9d275ba2ab60e5c76aed856c34533cd3ae)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A lack of whitespace around variable assignment operators makes the
files harder to read.
There is a deeper issue in that a "+" character can sometimes be confused
between the variable name and the assignment operator.
Start showing warnings for such usage so we encourage people to use
consistent whitespace which helps with file readability in general.
(Bitbake rev: 24772dd2ae6c0cd11540a260f15065f906fb0997)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds bb.parse.vardeps bb.parse.excludevardeps function decorators that
can be used to explicitly add or exclude variables from a python
function parsed by bitbake
(Bitbake rev: 030fb3dee067640a3a50f24a53d200bdb5048376)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Correct typo Fash -> Fast.
With it also delete some trailing spaces which are unnecessary and
editors are trying to remove when changing anything in the file.
(Bitbake rev: b96040a0a066344527f0bc6ca4d09cda6b9d2f11)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before fast shallow tarballs the shallow tarball with lfs worked only
if the original .../downloads/git2/... directory existed. Once it was
not there, do_unpack on shallow tarball without lfs files failed due to
disabled network.
It was fixed for fast shallow tarballs and this commit fixes the other
scenarios, too.
This can happen when such shallow tarball is put to mirror or if
someone does cleanup to reclaim disk space.
(Bitbake rev: 4e6fb31a88448cdacf4e9f84cf0a8a8035f84d63)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using a shallow clone to simulate an outdated git mirror tarball does
not work in the intended way. A shallow clone already contains the
latest commit which can hide certain fetcher behavior. Simulate an
outdated mirror tarball, as the test titles indicate, by removing the
newer commits from the mirror.
(Bitbake rev: a51ee01f0a586fefd5a4061f4a1ca6cbf81b7046)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When invoking the original git fetcher after downloading a mirror
tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch
commands targeting the upstream source being executed silently. Ensure
setting BB_NO_NETWORK before invoking the original fetcher. While this
was only observed for git, setting this in general for all fetcher
types makes sense at this location.
(Bitbake rev: 1b1321f2b60c0a66159e3f20c6befcb0b3ccc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SRCREV points at an annotated tag, the comparision code can fail
as the resolved tag might not be the same sha.
Handle this by also resolving the SRCREV. We only need to do this if
they don't match in the first place for a minor performance win.
Also add a test for this.
(Bitbake rev: 136c06e251de68ed64355ec6b47a522ff3a372e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example the miniupnpd recipe has a SRC_URI like this:
http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz
In this case the path is /files/download.php, which isn't useful when
the latest_upstream logic bails early if there is no version in the path.
The logic now also checks in the downloadfilename, so add a test that
this works as expected.
(Bitbake rev: fffbf5d5e1c8556cddf0794e0b303bb0106747a0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some code paths in latest_versionstring() need PV to be set correctly.
(Bitbake rev: 0a9f90ff658e09feda63b398ec35715a65ff6193)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a named tuple so the test can access named members instead of just
accessing the data via index, which is harder to understand.
(Bitbake rev: 4b15652c84b06f0506c757e2647875a9b1cc7bfe)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Marking the test iterations as subtests means that when one fails, it
can identify clearly which iteration has failed.
(Bitbake rev: 52c55e681332d7cdbe06f3c9d9c8d77cb0cb93f6)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's best practise to use long-form arguments in scripts where the
conciseness of short arguments is less useful than in an interactive
terminal.
(Bitbake rev: 54039bc9d169871af6d36578df4c21bff296f6de)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the target file exists, we pass --continue so that wget will try to
resume the download instead of starting from the beginning. However if
the file doesn't exist --continue has no effect, so there's no need to
conditionalise the use of the argument.
(Bitbake rev: a5ee50d4d2b2e6c00abe1551382afd9799345dbf)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
latest_versionstring() currently looks at just the end of the URI when
guessing what the filename to look for is, but this doesn't work if the
URL filename is not simple.
For example, miniupnpd has a SRC_URI of:
http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz;downloadfilename=${BP}.tar.gz
The filename component of this is "download.php", which causes the
heuristics in latest_versionstring() to exit early.
Instead, if the downloadfilename is set then use that, as it's often the
actual filename that we're after.
(Bitbake rev: 2d5f135e997d13fabda0ad266fd5c928ee33f487)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Build the 'downloadfilename' parameter by replacing path separators in
the module path like the git fetcher builds the mirror tar ball name.
Copy the downloaded file in the fetcher's unpack method like the crate
fetcher instead of calling the base fetcher's unpack method.
(Bitbake rev: 7762cea087597019460d66b04268757bd46befdf)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the fetcher revisions changes, we need a new version marker to
match this with in OE-Core.
(Bitbake rev: 8cc976e2792fdde3900729f3b09dd18ab640b5e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).
We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.
This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.
Some test cases are removed since they are no longer needed.
(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Ubuntu 20.04 is heading to EoL and that we're having other issues
on that platform needing buildtools anyway, we're about to lose the
last python 3.8 platform we were supporting.
Bump the minimum version to 3.9 since there are many developers wanting
access to newer python features.
(Bitbake rev: 0d2e682d00dfc2ec776b3a89f25202db98024895)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add tests for git tag verification in both standard and shallow clones.
(Bitbake rev: f47127066d67e2ad80974fa1e7c0fcc7409161af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we want to validate a tag when using shallow clones, we need to ensure
the tag is imported to the clone. Adding to extra_refs allows this to happen
ensuring tag validation in shallow clones works.
(Bitbake rev: 03945475886b8d31c7eccc80fe594ab5306bcf5d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.
Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.
This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.
The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.
(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
finalizeData changes the value of variables, so, it needs to have data
tracking enabled to get a full and accurate variable value history.
Fixes [YOCTO #15652]
(Bitbake rev: 228979b506f7d0e24c643440bcc7becf684da283)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new context manager Tinfoil._data_tracked_if_enabled() is introduced to
replace the following structure:
if self.tracking:
self.run_command('enableDataTracking')
# Code that need data tracking
if self.tracking:
self.run_command('disableDataTracking')
(Bitbake rev: 0fea4555d2143c6b23a79d3d5cf791103a68141b)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes a bug where escapes in the url path would not be properly
restored for the git commands in the git fetcher. For example, a
space which is encoded as '%20' was not properly encoded before the
clone command.
e.g.
SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https"
resulted in
git clone 'https://git.openembedded.org/bitbake example/bitbake'
instead of
git clone 'https://git.openembedded.org/bitbake%20example/bitbake'
(Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680)
Signed-off-by: Patrik Nordvall <patrik.nordvall95@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing module files was changing the tash hashes but it was not invalidating
the parse cache, leading to tashhash mismatch errors during builds.
Add information from modulecode_deps to the configuration hash used for
cache invalidation to avoid this and trigger reparses when function library
code changes.
[YOCTO #15795]
(Bitbake rev: a121db3d8d28420c36369237b8bb11c2d0aaf5f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The command complete event triggers the UI to start new commands.
We should send this once we've cleaned up existing commands since
on very slow systems, this could otherwise cause failures. There
is a 30s timeout on incomming commands but slow systems have been
known to hit this on poorly configured I setups.
(Bitbake rev: d202298b3d989c39bc532b69c410e2e78b082a11)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a "parsing started" event in the cooker deamon log but we don't
currently log the corresponding "parsing complete" event which is
confusing. Add this so that the logs are more logical.
(Bitbake rev: 1aa491c1f1211bf9faab712c321b66629fb7be66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a custom python module is added thru BBIMPORTS and it
uses typing(Any,Tuple,Union...), codeparser will fail because
inspect.py raises TypeError exception if the object is a
built-in module, class, or function.
(Bitbake rev: 0ecfd0b8540220633e71d24cd73cc5306863ae3c)
Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A regular expression "URL" in PREMIRRORS and MIRRORS may contain a ? as
part of the regular expression. Make sure this does not cause
problems.
(Bitbake rev: 5af7fe4473cd7e75d4eb7f8b93c499bd157ff156)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f
which changes decodeurl() to use the URI class to parse the URL instead
of doing it itself. While reusing code is generally a good idea, using
urllib.parse.urlparse() (which the URI class does) to parse the regular
expression "URLs" that are used in PREMIRRORS and MIRRORS does not work.
A regular expression URL containing https?://... would be silently
ignored, while a URL using a negative lookahead such as
git://(?!internal\.git\.server).*/.* would result in a cryptic error:
Exception: re.error: missing ), unterminated subpattern at position 0
The problem is that urllib.parse.urlparse() treats the ? as the start of
URL parameters and thus stops parsing whatever part of the URL it was
parsing. Restore the old function and use it in the PREMIRRORS and
MIRRORS code.
(Bitbake rev: f8a7712754e6d0199a0d227fca288307b935368d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At the end of this function, self.overrides is cleared, which means we'll
construct a new self.overrides after this call. And a new self.overrides
will give out different expand_cache, so the old expand_cache should also
be cleared to avoid any wrong value.
Currently, there's a problem revealed by recent recipe specific virtual
provider patch. If we enable multilib and set "OVERRIDES:prepend" in local.conf,
things don't work.
Here's the error message:
ERROR: Nothing PROVIDES 'lib32-gcc-cross-x86_64'
Below are reproduce steps:
1. Add in local.conf the following lines:
MACHINE ?= "qemux86-64"
require conf/multilib.conf
MULTILIBS ?= "multilib:lib32"
DEFAULTTUNE:virtclass-multilib-lib32 ?= "core2-32"
OVERRIDES:prepend = "some-override:"
(Note that using :append and :remove also reproduces the issue.)
2. bitbake -n lib32-sysstat
(bitbake -n core-image-minimal also reproduces the issue)
The expandWithRefs calls getVar, which fills expand_cache. So when setting
OVERRIDES:prepend, this will fill the expand_cache. When overridevars are updated,
if we don't clear expand_cache, we'll retrieve wrong values.
Previously, things happened to work because there's a call to expand
PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc, which in turn expands
TARGET_VENDOR. Now what we expand is PREFERRED_PROVIDER_ virtual/cross-cc,
so the problem is revealed.
(Bitbake rev: 842e94234c9b9faf4e902721f516071f643e49c9)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When var matches __setvar_regexp__, which has the value as below,
__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$')
the keyword will not be __doc__ or __module__ and var will always
contain ":".
(Bitbake rev: 0af396917fe176df7b1ed0bd7c7267507ed12e03)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
