Fix an issue where two tests have the same name with one overwriting the
other.
(Bitbake rev: 38842a8150f44f5d31d9bd2b0b6ec0502acc971d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit da812d938fd79e2cc7bdf355ccf5b0f9ead684c4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the owner information in the mirror tarballs generated using
BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit
0178ab83, which used the original pokybuild:user information, but failed
to clean up the numerical user and group ids. Now set the more canonical
values of oe:oe and 0:0.
(Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
CC: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding bb.utils.filter('WARN_QA', 'patch-fuzz', d) when WARN_QA is in
BB_BASEHASH_IGNORE_VARS or in vardepsexclude should not add a dependency
on WARN_QA.
Fix it and add some tests.
(Bitbake rev: 6aecc2fe51a52020f6f13be08449e18d42e7a6b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs
of git repositories, they leaked local information: username, group and
time of the last fetch. Remove all these by setting fixed information:
* uname = pokybuild
* gname = users
* mtime = committer time of newest commit in repo
The username and group value were taken from the archives available on
the downloads.yoctoproject.org mirror. The modification time is chosen
so it still retains some relationship to the contents of the archive.
(Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS
BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS
the derived code variables basewhitelist and taskwhitelist have been
renamed to basehash_ignore_vars and taskhash_ignore_tasks,
respectively.
[RP: Added compatibility mapping code and tweaked naming]
(Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:
* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).
Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.
Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.
(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We started to require the branch parameter for git urls. This updates
the tests to match and quieten some noise.
(Bitbake rev: e5144887f2b6ae266f7b52151005619480e80ec6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously npm files that specify downloadfilename= in the SRC_URI
would be downloaded to the root of ${DL_DIR} rather than in the
${DL_DIR}/npm2 directory where all other npm files are downloaded.
This should make it simpler when setting up and configuring a
premirror with the downloaded npm packages.
(Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The solution implementated in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror) missed two corner cases. The
first is if the basename of the original URI also appears somewhere
else in the replacement URI, in which case it would also be replaced.
The second is if the basename of the original URI partially matches
the basename of the replacement URI, in which case the inital part of
the basename from the replacement URI would be left behind.
The second case caused test_npm_premirrors_with_specified_filename to
fail.
The solution is to prefix the basename with a slash when matching to
avoid partial matches, and only replace the basename at the end of the
URI.
This also adds two test cases that test for these problems. Before
this they would give the following errors:
- ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++ ++++++++++
- ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++
(Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two totally opposite use cases for how a premirror is
expected to behave in combination with specifying a downloadfilename=
parameter in the SRC_URI. On the one hand there is the expectation
that it works like any other mirror, which means the premirror is
expected to contain a file with the original name specified in the
SRC_URI. On the other hand there is the expectation that one can use
the artefacts downloaded by bitbake in ${DL_DIR} as a premirror, in
which case it is expected to contain a file with the name from the
downloadfilename= parameter.
The latter case has been how downloaded files have been handled until
commit 8a3ff9f3 (fetch2: fix premirror URI when downloadfilename
defined), where the fetcher was changed to store files as per the
first case. This is also when the test_npm_premirrors test case was
modified in commit 5ba191a0 (tests/fetch: add and fix npm tests) to
expect the first case.
The above change was later reverted in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror). However the
test_npm_premirrors test case was not updated to match, and has been
failing ever since. This has probably gone unnoticed because the npm
related test cases require that npm is installed on the host.
This commit updates test_npm_premirrors to expect that premirrors use
the filenames specified by downloadfilename= as this matches the
current fetcher implementation and also is the most likely use case
for premirrors. It also tries to mimic how one typically might setup
the premirror directory by simply copying the download directory.
(Bitbake rev: 9e913ade70474aaeb928814d4763e7105569d63a)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This unifies the three separate implementations of git() from
GitMakeShallowTest, GitShallowTest and GitLfsTest into one
implementation in FetcherTest. It also makes use of this in
FetcherLocalTest and FetcherNetworkTest.
(Bitbake rev: 8c8b418c3e6f8ab2535ad405e8006ee1012f3484)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes:
test_foo .. skipped 'Network tests being skipped'
test_foo .. skipped 'npm not installed, tests being skipped'
to:
test_foo .. skipped 'network test'
test_foo .. skipped 'npm not installed'
to avoid double "skipped" in the output.
(Bitbake rev: e551328a4595c220b94ab5002d978ddb4deeebcf)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in mirror variables with
"\n".
(Bitbake rev: ed22b078aae25e804ca5edf062e2c905d3e49426)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Both test_crate_url and test_crate_url_multi require Internet access.
(Bitbake rev: 34e9daed6a7f766bb6e27c7e02c94b545f4791d8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This imports the crate fetcher from OE-Core to resolve various module issues
and adds some very very basic tests of that new fetcher.
(Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These urls are no longer adding much to the test coverage but the intermittent
network issues connecting to them are painful. Drop the urls.
(Bitbake rev: bdf5739c5d831dc97a7d81568f94a0953c71017f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Internal bitbake shutdown timings must have changed recently as we're
seeing race issues on the autobuilder around the removal of the bitbake.lock
file. Improve the lockfile race code to cover bitbake's lockfile too
and use it in all the tests.
[YOCTO #14658]
[YOCTO #14652]
(Bitbake rev: bd1912bed64424f9fb28396b71bb49b6090ed087)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using "" as a target for .replace() is a really bad idea as it duplicates the replacement
for every character in the string. Add a testcase which triggered this and correct the
code to return the correct result.
(Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was clear with testing that we're asuming bitbake sets TOPDIR correctly
when running these tests. Remove that implict assumption and make it
explicit.
(Bitbake rev: c8f1eb377ceb1fc78cbfaed976107720ad78c075)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
vcs.pcre.org was a redirect to github which we use for subversion testing.
With the protocol changes at github and the removal of the redirect, use a
direct address for github.
(Bitbake rev: 6230ca71eb7eb2a6db162e28a01727d00af5299b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test to ensure that a parameter like protocol=git can be rewritten
to a different url and protocol.
(Bitbake rev: 69b4f9a09ff74378788cc2ec1ad58cd66b27ca59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.
[YOCTO #13823]
(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.
Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.
We could consider dropping destsuffix as a parameter as some future point.
Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.
(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test case test_fetch_premirror_use_downloadfilename_to_fetch to ensure
that 'downloadfilename' is used when fetching from premirror.
Although the other two previous test cases, test_fetch_premirror_specify_downloadfilename_regex_uri
and test_fetch_premirror_specify_downloadfilename_specific_uri already
implicitly contain such verification, we still need to add a very clear
case to ensure no regression.
(Bitbake rev: 20aabc3d53f69949810ecf02295725db947ffef8)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When downloadfilename is specified, it is used to fetch from premirror.
So fix the test cases accordingly.
(Bitbake rev: 3b4d2e3b5024324058360a2a28f33c34114218d0)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our minimum python version is 3.6 so these are now unneeded.
(Bitbake rev: 15f16bf973666807e94f3b356095f8265af0282a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen races where the socket may be gone but the server is still writing
out it's database. Handle that case too to avoid cleanup tracebacks.
[YOCTO #14440]
(Bitbake rev: b9e4fb843cb9d3a4d4404af093a781fab5520465)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some debugging, BB_CURRENTTASK is set too late to be useful as
it isn't present in some event handlers for example. There is no
other way to know which task is actually running so set the value
earlier.
(Bitbake rev: 7daaaaa27f55b5a458656857c6d61a51b34a62fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set a prefix so the temporary directories are identifable.
(Bitbake rev: c3440b82cbe9c317f9961d61e12ea37fc9541ce0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoids a warning from Python as \s isn't a valid escape.
(Bitbake rev: be39c2e926beebe78030fbe26c6737f08f960fcb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix:
lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence
for several lines of the fetch tests.
(Bitbake rev: 9d84fd557a3fcbae2cdd70b24e69325ad737a01e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds one new test that verifies the use of a specific filename
when defined in PREMIRRORS.
bb.tests.fetch.NPMTest:
- test_npm_premirrors_with_specified_filename
While testing bz#13039, it was found that test_npm_registry_alternate
fails with ENOTFOUND. This was corrected by using npmjs's public mirror.
The change to fetch2 for bz#13039 highlighted an issue with the
test_npm_premirrors test where the created file:// mirror was using the
downloadfilename rather than the tarball that is defined by the npm url.
(Bitbake rev: 5ba191a0407af9e652e3b86302dce3e952d6b587)
Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds three new tests which evaluate different use cases of the
downloadfilename property.
bb.tests.fetch.FetcherNetworkTest:
- test_fetch_specify_downloadfilename
- test_fetch_premirror_specify_downloadfilename_regex_uri
- test_fetch_premirror_specify_downloadfilename_specific_uri
(Bitbake rev: 61db3e96530d650e098436fd086f0182d32998f7)
Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.environment() is a context manager to alter os.environ inside
a specific block, restoring it after the block is closed.
(Bitbake rev: 9974848f67581ff7d76cef52a94f505af99b4932)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.
(Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting something like:
BAR:append:unusedoverride
should cause BAR to be None, not "" which was what the datastore was
returning. This caused problems when mixing variables like:
RDEPENDS:${PN}:inactiveoverride
RDEPENDS:${BPN}
since key expansion would report key overlap when there was none. This
is a bug in the datastore. Fix it and add a test too.
[YOCTO #14088]
(Bitbake rev: 699e36c270d863258502d315ed00a1b940bfbf96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The examples and tests use non-standard override names, convert these to
the new syntax by hand.
(Bitbake rev: a6c40eca1146c0160da7e4e0bd7ac52fef2029e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This runs the overrides conversion script in OE-Core over the bitbake code
base including the docs. A handful of things were excluded in toaster
and for the Changelog file.
(Bitbake rev: 47f8d3b24fd52381bf3b41e2f55a53e57841344c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that ":" is a valid character in variable key names, it needs to be
allowed by the variable expansion code too, to match.
(Bitbake rev: 019251649a38754d5877759d13b664e28dea77de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a library that implements file-like objects (similar to
gzip.GzipFile) that can stream to arbitrary compression programs. This
is utilized to implement a LZ4 and zstd compression API.
(Bitbake rev: 61c3acd058ea018696bd284b3922d0b458838d05)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
