7-Zip is a file archiver claiming the highest compression ratio.
This patch allows using 7-Zip commpressed files in bitbake recipes.
Two common formats are supported:
SRC_URI = "file://abc.tar.7z"
SRC_URI = "file://abc.7z"
(Bitbake rev: 7120f5bfaae54e91bc95da5667831424724ce613)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a mirror tarball is fetched, the original fetch method is called, which
unpacks the mirror tarball. After the original method is called, it checks the
localpath of the mirror tarball rather than the clone path, which isn't ideal,
particularly if the mirror tarball was removed due to being out of date. We
know the original fetch method will do what it needs to do to get its content
in the form it needs from the mirror tarball, so we can use its localpath
instead.
(Bitbake rev: 1732ad65d6c7d67b7d07cb30c074f5016adadbea)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This looks reasonable until you realise self.localpath is a function. Data
expansion of something which isn't a string is the original value so this
code just wastes CPU cycles and makes no sense. Remove it.
(Bitbake rev: 37214ea9bf484998b75dbc1200d53f1afc5257ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list. This permits us to re-evaluate the list
while excluding all matches that have previousily occured.
Without this fix, adding this test results in a failure:
RuntimeError: maximum recursion depth exceeded in cmp
(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When our clone exists, but is out of date, and the attempt to update it fails,
we don't necessarily want to remove the entire clone, particularly if it's
a large repository.
(Bitbake rev: 19af272ba5256653edeff6acbceeb09e3e478d61)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the main fetch method doesn't support checksums, the user will not be
defining them in the recipe, so we don't want to check them for
premirrors/mirrors either. This ensures that we never error due to missing
checksums on a git mirror tarball.
(Bitbake rev: 24c79bbed361b37f12d3351af13602e3d4386f4c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__.py: Add connection_cache param in Fetch __init__.
In order to pass connection cache object to checkstatus method.
[YOCTO #7796]
(Bitbake rev: 9fa6407e6cefe66c77467419a8040d6957a6bb01)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to pass connection cache object to checkstatus function
add fetch parameter.
(Bitbake rev: fbb9c6f5538084e125b58118a86968908e6f895b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchConnectionCache class acts as a container for socket connections
useful when implement connection cache into fetcher modules.
(Bitbake rev: 454da2cd17539ceb9caad6d76f034757e44ee12f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea here is to support package version numbering similar to gitpkgv in
meta-openembedded. This commit is the first step towards such functionality.
The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
per Richard's suggestion [1]. While writing this, I noticed that it would
become a copy of get_srcrev with only two lines changed. So to create something
more Pythonic than a boolean argument and conditionals around the calls to the
fetcher's sortable_revision, I just made the method to be called on the fetcher
an argument to the method. Defaulting to 'sortable_revision' prevents affecting
existing code.
Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
set the following in a recipe:
PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
and this would yield the same result as gitpkgv's GITPKGV variable.
See for the discussion leading to this change:
[1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
(Bitbake rev: 2f1f4483493cc290f5d2c07f9906e90eaea2f4c1)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
First, when building mirror urls we don't do any fetching so we should never
be calling clean functions.
Currently, if a mirror url fails, we don't process it further to see
if there are any mirrors of the mirror.
We should do this even when the mirror url fails, else we may miss out
on valid/useful mappings, particularly in the case of file:// urls.
(Bitbake rev: b7fd3ec9994f664b17fc86423e6e7afac07e897b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow GIT_SMART_HTTP to be passed through to the fetch command so that
servers that cannot use GIT_SMART_HTTP can be used by the fetcher.
(Bitbake rev: e5c97a85bed0436d48eeaac2e32962cfb5371d2f)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ALLOWED_NETWORKS is a list of hosts that the fetcher will be allowed
to use when BB_NO_NETWORK is not set.
If BB_NO_NETWORK is set, then networking is still disabled.
If BB_ALLOWED_NETWORKS is not set, the behaviour remains the same as
today.
If BB_NO_NETWORK is NOT set, and BB_ALLOWED_NETWORKS is configured, then
only the hosts in the list are usable by the fetcher.
eg:
BB_ALLOWED_NETWORKS="yoctoproject.org git.gnu.org"
The fetcher will be able to download from yoctoproject.org, git.gnu.org,
but not ftp.gnu.org or any other hostname that is not in the list.
There is also limited support for wildcards on the beginning of the
hosts, so BB_ALLOWED_NETWORKS="*.gnu.org" with match git.gnu.org and
ftp.gnu.org as well as foo.git.gnu.org
(Bitbake rev: c7263096ba31ba45daeeb9de90c1cb9ebef24a28)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting BB_ENV_EXTRAWHITE to include GIT_SSL_CAINFO is enough to
get "git ls-remote" to work for recipe parsing but it is not enough
to get the fetcher to clone properly.
This is because the fetcher has its own idea about what variables
should be exported in the environment for some operations.
It is desirable to use alternate CA Bundles for internal testing prior
to using public keys for https, so we should allow the GIT_SSL_CAINFO
to pass through.
(Bitbake rev: 40ff92282bbf32cf644b021bf7cbb1f393dbb856)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #5571] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571
The following workflow (whether accidentally or deliberately) would
previously not result in a checksum error, but would be helpful to do
so:
- Write a recipe with correct checksums
- Fetch the sources for this recipe using bitbake
- Change the checksums
Since the bitbake fetcher writes a done stamp after the initial download
and does not verify the checksums again (even if they are changed in the
recipe afterwards), the change of checksums is silently ignored.
Fix this without the overhead of computing the checksums from scratch on
every do_fetch by storing them in pickled format in the done stamp and
verifying that they still match those in the recipe.
(Bitbake rev: fbd4a0d422cf7f43db2f9eab2e47c41246a9031e)
Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the comments on get_srcrev() to better describe its function.
(Bitbake rev: b4d40f1ac7b32990c456cce261f99a5a157b5ae5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.
(Bitbake rev: 49dbcfbc56a206964acc5de761bba31be0283ba1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check if the 'subdir' parameter exists and assign it to 'destdir' so that
files are copied in ${WORKDIR}/destdir. This fixes urls that are of the form
file://a;subdir=b.
(Bitbake rev: 836a986b365eb9798563ec08d90b346596de7791)
Signed-off-by: Roxana Ciobanu <roxana.ciobanu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The clearcase fetcher is used to retrieve files from a ClearCase
(http://en.wikipedia.org/wiki/Rational_ClearCase) repository.
Usage in the recipe:
SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
SRCREV = "EXAMPLE_CLEARCASE_TAG"
PV = "${@d.getVar("SRCREV").replace("/", "+")}"
The fetcher uses the rcleartool or cleartool remote client, depending on which one is available.
Supported SRC_URI options are:
- vob
(required) The name of the clearcase VOB (with prepending "/")
- module
The module in the selected VOB (with prepending "/")
The module and vob parameters are combined to create
the following load rule in the view config spec:
load <vob><module>
- proto
http or https
Related variables:
CCASE_CUSTOM_CONFIG_SPEC
Write a config spec to this variable in your recipe to use it instead
of the default config spec generated by this fetcher.
Please note that the SRCREV loses its functionality if you specify
this variable. SRCREV is still used to label the archive after a fetch,
but it doesn't define what's fetched.
User credentials:
When using cleartool:
The login of cleartool is handled by the system. No special steps needed.
When using rcleartool:
In order to use rcleartool with authenticated users an `rcleartool login` is
necessary before using the fetcher.
(Bitbake rev: 76cff0aaad4cf10b325bceadd8ea90b3d75edcc2)
Signed-off-by: Dennis Meier <meier.dennis@siemens.com>
Reviewed-by: Roger Meier <r.meier@siemens.com>
Reviewed-by: Christian Liechti <christian.liechti@siemens.com>
Reviewed-by: Henrique Mendonca <henrique.mendonca@siemens.com>
Reviewed-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handle the following situations in a URL (e.g. in SRC_URI):
* Trailing semicolon in a URL - this is now ignored.
* Parameter specified with no value (no equals sign). This still
produces an error, but at least it is MalformedUrl with a proper
message rather than "ValueError: need more than 1 value to unpack".
(Bitbake rev: bfd13dfbc4c9f1dd8315002271791b1d9e274989)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The initial version of this patch was reverted in
0a94e568152de550dedc8135a766beb18bf064ab
However, it was later agreed upon that it was reverted due to
a misunderstanding during code review; the patch is ok.
This version of the patch also removes an unneeded initialization of
the 'checksum' variable outside the scopes it is being used in.
(Bitbake rev: 9ee19a3ca2f8e11a91f0289ea3486310c61d40f2)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 6dfb33fd7204e09815bde7ba5bf21941a73fe965.
It causes an unindeded change in behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup the fix done in f9416e76e272ba3249abff099f6f3a47fe82e03e.
Instead of adding continue statements we can just move the last
statements into the final else-clause.
(Bitbake rev: 6dfb33fd7204e09815bde7ba5bf21941a73fe965)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a directory matching a wildcard in SRC_URI when getting file
checksums, we should recurse into that instead of producing an error.
(Bitbake rev: ae87b7eb414e3d5eefd2effec7b30c22d2186b02)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Local fetcher's localpath is returning the parent directory for a
wildcard match; we need to handle this and add the wildcard
specification so that we checksum the correct files.
Fixes [YOCTO #6127].
(Bitbake rev: 4a90edd62c16cbf41b5b93280e155077564c774a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, if a file listed in SRC_URI was missing at the time of
parsing, and then was added and bitbake run again, it would not be
picked up because the recipe was cached without it. If we allow the file
to be added to the list of files to checksum, then it will be checked
for and found on the second run.
Fixes [YOCTO #4790].
(Bitbake rev: 71da822762cb298261cccdfa54b9c0fea02c3c5d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a while its been puzzling me why connman-gnome rebuilds as often as it
does. It turns out you can trigger this with a new checkout of the metadata.
The SRC_URI that is causing the problems is:
SRC_URI = "file://images/*"
and rather oddly the results in checksums for a file "." being added to
the tree, e.g.:
('.', 'ab48a68186f0e0f277c21ef4cb390b4b')
The problem is that when iterating files lists, the checksum variable can
become set yet we don't break the out from the for loop, which leads to
odd (and non-deterministic) entries being added into the file checksum list.
The exact item added probably depends on the order of items on the disk.
Before this change, bitbake-diffsigs on connman-gnome:do_fetch would report:
This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('.', 'ab48a68186f0e0f277c21ef4cb390b4b')]
Afterwards:
This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85')]
which is correct and deterministic without the "." entry.
(Bitbake rev: f9416e76e272ba3249abff099f6f3a47fe82e03e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting the username after already having set the password, the password
was unexpectedly reset. This change fixes this issue and introduces unit tests
to make sure it doesn't happen again.
(Bitbake rev: 25faef3a047f9c7564089463d7c96f6910b640cb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a case in meta-intel where a SRC_URI contains both a query string and
URI parameter:
https://edc.intel.com/Download.aspx?id=6190;downloadfilename=LIN_IEMGD_1_14_GOLD_2443.tgz
Python's urlparse thought the URI parameters were part of the query parameter
value, but in the bitbake context this is obviously not the case. As bitbake's
usage isn't really RFC compliant, we have to extract and remove the URI parameters
*before* urlparse sees the URI.
(Bitbake rev: c2f27ae4271985b48f957747b6ea372c8699cb49)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change introduces the .query property of the URI class. It is a
read/write dict of the parameters supplied in the query string of the
URI. E.g.:
http://example.com/?foo=bar => .query = {'foo': 'bar'}
(Bitbake rev: 1cb2b3c458c8c5521591d2c8f2e0058143fc77bb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The svk fetcher never appears to be used by anyone and the development
on svk appears to have stopped in 2010. We might as well drop support
for it.
(Bitbake rev: 8239264753977bd06ad5b1b574245d3842af489b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A typo was meaning that the mirror creation method wasn't being called
when it should have been. Fix the type to fix mirror tarball creation.
[YOCTO #5284]
(Bitbake rev: 348971d410bfd5d8b1757468d73e1d24ae78a594)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current message can be ambiguous, improve it (and also rename a
variable to clean up the rest of the function).
(Bitbake rev: 0c1bb7c0fce7b0f334311a2893ccb00385fa8d55)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.
Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.
Also add some unittests for this.
(Bitbake rev: e82a4ab48991035866da9914c8b75a9bfbc9a7fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently INVALID and None are checked as incorrect values under different
circumstances. This code standardises those checks to be consistent. We
should phase out the use of "INVALID".
(Bitbake rev: 86ef4e65ce18b71dc69643586bd2aa8f48703171)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher will try:
1) PREMIRROR
2) Upstream
3) MIRROR
If it fails to download from the Upstream, but succeeds from the MIRROR,
and ud.localpath != origud.localpath (for example, the git tarball),
then we will get the error (e.g.: xf86-video-omapfb):
ERROR: Function failed: Fetcher failure for URL: 'xxx'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /path/to/log.do_fetch.28024
It should not show the error and let the build go on since it succeeds.
(e.g.: xf86-video-omapfb)
[YOCTO #5686]
(Bitbake rev: c08ca1e4eeb04f78e1354780cf5a4c3855e49572)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* currently decode_url regexp parses branch=@foo as username so it ends like this:
- ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
+ ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
that there is at least one '/' as separator between netloc and path,
params, so it looks reasonable to prevent including '/' in username
(Bitbake rev: 2c82742114091cb55055328b54223686816582f2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This add a Git Annex backend which reuses the Git fetcher code; it
allows managing files with git, without checking the file contents
into git, being useful when dealing with files larger than git can
currently easily handle, whether due to limitations in memory, time,
or disk space.
(Bitbake rev: a61fc4db598e9d13c966712a6a0e4783e19448be)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
Everything should be removed, but the
0006-omapfb-port-to-new-xserver-video-API.patch.done still exists in the
DL_DIR, this is because the clean() in the fetch2/__init__.py skips
removing the local file, so that it will skip removing the .done.
The local file (file://) isn't needed to be removed since it is not
downloaded into DL_DIR, but the .done should be removed, this patch will
remove the .done, and it doesn't remove anything else since the clean()
in local.py does nothing.
[YOCTO #5687]
(Bitbake rev: 2bc99b9dfa532430a13c39fca4e5ef3a2206b3b8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
