mirror of
https://git.yoctoproject.org/poky
synced 2026-04-17 18:32:12 +02:00
bitbake: user-manual-fetching.xml: Re-write of "File Download Support" chapter.
Basic re-write to clean up text and flow. (Bitbake rev: 0f82dc4c22ce015bef87fd5aae0b6fa3e429016e) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark
committed by
Richard Purdie
Richard Purdie
parent
b4b3bf56f6
commit
f3caa2b27e
@@ -2,84 +2,110 @@
|
||||
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
|
||||
|
||||
<chapter>
|
||||
<title>File download support</title>
|
||||
<title>File Download Support</title>
|
||||
|
||||
<para>
|
||||
BitBake's <filename>fetch</filename> and
|
||||
<filename>fetch2</filename> modules support downloading
|
||||
files.
|
||||
This chapter provides an overview of the fetching process
|
||||
and also presents sections on each of the fetchers BitBake
|
||||
supports.
|
||||
<note>
|
||||
The original <filename>fetch</filename> code, for all
|
||||
practical purposes, has been replaced by
|
||||
<filename>fetch2</filename> code.
|
||||
Consequently, the information in this chapter does not
|
||||
apply to <filename>fetch</filename>.
|
||||
</note>
|
||||
</para>
|
||||
|
||||
<section id='file-download-overview'>
|
||||
<title>Overview</title>
|
||||
|
||||
<para>
|
||||
BitBake provides support to download files.
|
||||
This procedure is called fetching and is handled by the
|
||||
fetch and fetch2 modules.
|
||||
At this point, the original fetch code is considered to
|
||||
be replaced by fetch2 and this manual is only related
|
||||
to the fetch2 codebase.
|
||||
When BitBake starts to execute, the very first thing
|
||||
it does is to fetch the source files needed.
|
||||
This section overviews the process.
|
||||
For some additional information on fetching source files, see the
|
||||
"<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
|
||||
section.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The <filename>SRC_URI</filename> is normally used to
|
||||
tell BitBake which files to fetch.
|
||||
The next sections describe the available fetchers and
|
||||
their options.
|
||||
Each fetcher honors a set of variables and per
|
||||
URI parameters separated by a “;” consisting of a key and
|
||||
a value.
|
||||
The semantics of the variables and parameters are
|
||||
defined by the fetcher.
|
||||
BitBake tries to have consistent semantics between the
|
||||
different fetchers.
|
||||
When BitBake goes looking for source files, it follows a search
|
||||
order:
|
||||
<orderedlist>
|
||||
<listitem><para><emphasis>Pre-mirror Sites:</emphasis>
|
||||
BitBake first uses pre-mirrors to try and find source
|
||||
files.
|
||||
These locations are defined using the
|
||||
<link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
|
||||
variable.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>Source URI:</emphasis>
|
||||
If pre-mirrors fail, BitBake uses
|
||||
<link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>Mirror Sites:</emphasis>
|
||||
If fetch failures occur using <filename>SRC_URI</filename>,
|
||||
BitBake next uses mirror location as defined by the
|
||||
<link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
|
||||
variable.
|
||||
</para></listitem>
|
||||
</orderedlist>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The overall fetch process first attempts to fetch from
|
||||
<filename>PREMIRRORS</filename>.
|
||||
If these fail, the original <filename>SRC_URI</filename>
|
||||
is attempted.
|
||||
If that fails, BitBake falls back to
|
||||
<filename>MIRRORS</filename>.
|
||||
Because cross-URLs are supported, it is possible to mirror
|
||||
a Git repository on an HTTP server as a tarball.
|
||||
Here are some examples that show commonly used mirror
|
||||
definitions:
|
||||
<literallayout class='monospaced'>
|
||||
PREMIRRORS ?= "\
|
||||
bzr://.*/.* http://somemirror.org/sources/ \n \
|
||||
cvs://.*/.* http://somemirror.org/sources/ \n \
|
||||
git://.*/.* http://somemirror.org/sources/ \n \
|
||||
hg://.*/.* http://somemirror.org/sources/ \n \
|
||||
osc://.*/.* http://somemirror.org/sources/ \n \
|
||||
p4://.*/.* http://somemirror.org/sources/ \n \
|
||||
svk://.*/.* http://somemirror.org/sources/ \n \
|
||||
svn://.*/.* http://somemirror.org/sources/ \n"
|
||||
PREMIRRORS ?= "\
|
||||
bzr://.*/.* http://somemirror.org/sources/ \n \
|
||||
cvs://.*/.* http://somemirror.org/sources/ \n \
|
||||
git://.*/.* http://somemirror.org/sources/ \n \
|
||||
hg://.*/.* http://somemirror.org/sources/ \n \
|
||||
osc://.*/.* http://somemirror.org/sources/ \n \
|
||||
p4://.*/.* http://somemirror.org/sources/ \n \
|
||||
svk://.*/.* http://somemirror.org/sources/ \n \
|
||||
svn://.*/.* http://somemirror.org/sources/ \n"
|
||||
|
||||
MIRRORS =+ "\
|
||||
ftp://.*/.* http://somemirror.org/sources/ \n \
|
||||
http://.*/.* http://somemirror.org/sources/ \n \
|
||||
https://.*/.* http://somemirror.org/sources/ \n"
|
||||
MIRRORS =+ "\
|
||||
ftp://.*/.* http://somemirror.org/sources/ \n \
|
||||
http://.*/.* http://somemirror.org/sources/ \n \
|
||||
https://.*/.* http://somemirror.org/sources/ \n"
|
||||
</literallayout>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Non-local downloaded output is placed
|
||||
into the directory specified by the
|
||||
<filename>DL_DIR</filename> variable.
|
||||
For non local archive downloads, the code can verify
|
||||
sha256 and md5 checksums for the download to ensure
|
||||
the file has been downloaded correctly.
|
||||
These can be specified in the following forms
|
||||
for md5 and sha256 checksums, respectively:
|
||||
Any source files that are not local (i.e. downloaded from
|
||||
the Internet) are placed into the download directory,
|
||||
which is specified by
|
||||
<link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
For non-local archive downloads, the fetcher code can verify
|
||||
sha256 and md5 checksums to ensure
|
||||
the archives have been downloaded correctly.
|
||||
You can specify these checksums by using the
|
||||
<filename>SRC_URI</filename> variable with the appropriate
|
||||
varflags as follows:
|
||||
<literallayout class='monospaced'>
|
||||
SRC_URI[md5sum]
|
||||
SRC_URI[sha256sum]
|
||||
</literallayout>
|
||||
You can also specify them as parameters on the
|
||||
<filename>SRC_URI</filename>:
|
||||
You can also specify the checksums as parameters on the
|
||||
<filename>SRC_URI</filename> as shown below:
|
||||
<literallayout class='monospaced'>
|
||||
SRC_URI="http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
|
||||
</literallayout>
|
||||
If <filename>BB_STRICT_CHECKSUM</filename> is set, any download
|
||||
without a checksum will trigger an error message.
|
||||
In cases where multiple files are listed in
|
||||
If
|
||||
<link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
|
||||
is set, any download without a checksum triggers an error message.
|
||||
In cases where multiple files are listed using
|
||||
<filename>SRC_URI</filename>, the name parameter is used
|
||||
assign names to the URLs and these are then specified
|
||||
in the checksums using the following form:
|
||||
@@ -89,142 +115,226 @@ MIRRORS =+ "\
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='local-file-fetcher'>
|
||||
<title>Local file fetcher</title>
|
||||
<section id='bb-fetchers'>
|
||||
<title>Fetchers</title>
|
||||
|
||||
<para>
|
||||
The URN for the local file fetcher is file.
|
||||
The filename can be either absolute or relative.
|
||||
If the filename is relative,
|
||||
<filename>FILESPATH</filename> and failing that
|
||||
<filename>FILESDIR</filename> will be used to find the
|
||||
appropriate relative file.
|
||||
The metadata usually extend these variables to include
|
||||
variations of the values in <filename>OVERRIDES</filename>.
|
||||
Single files and complete directories can be specified.
|
||||
<literallayout class='monospaced'>
|
||||
As mentioned in the previous section, the
|
||||
<filename>SRC_URI</filename> is normally used to
|
||||
tell BitBake which files to fetch.
|
||||
And, the fetcher BitBake uses depends on the how
|
||||
<filename>SRC_URI</filename> is set.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
These next few sections describe the available fetchers and
|
||||
their options.
|
||||
Each fetcher honors a set of variables URI parameters,
|
||||
which are separated by semi-colon characters and consist
|
||||
of a key and a value.
|
||||
The semantics of the variables and parameters are
|
||||
defined by the fetcher.
|
||||
BitBake tries to have consistent semantics between the
|
||||
different fetchers.
|
||||
</para>
|
||||
|
||||
<section id='local-file-fetcher'>
|
||||
<title>Local file fetcher</title>
|
||||
|
||||
<para>
|
||||
The URN for the local file fetcher is file.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The filename can be either absolute or relative.
|
||||
If the filename is relative,
|
||||
<link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
|
||||
is used.
|
||||
Failing that,
|
||||
<link linkend='var-FILESDIR'><filename>FILESDIR</filename></link>
|
||||
is used to find the appropriate relative file.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The metadata usually extend these variables to include
|
||||
variations of the values in
|
||||
<link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
|
||||
Single files and complete directories can be specified.
|
||||
<literallayout class='monospaced'>
|
||||
SRC_URI = "file://relativefile.patch"
|
||||
SRC_URI = "file://relativefile.patch;this=ignored"
|
||||
SRC_URI = "file:///Users/ich/very_important_software"
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='cvs-fetcher'>
|
||||
<title>CVS fetcher</title>
|
||||
<section id='cvs-fetcher'>
|
||||
<title>CVS fetcher</title>
|
||||
|
||||
<para>
|
||||
The URN for the CVS fetcher is cvs.
|
||||
This fetcher honors the variables <filename>CVSDIR</filename>,
|
||||
<filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
|
||||
<filename>UPDATECOMMAND_cvs</filename>.
|
||||
<filename>DL_DIR</filename> specifies where a
|
||||
temporary checkout is saved.
|
||||
<filename>SRCDATE</filename> specifies which date to
|
||||
use when doing the fetching (the special value of "now"
|
||||
will cause the checkout to be updated on every build).
|
||||
<filename>FETCHCOMMAND</filename> and
|
||||
<filename>UPDATECOMMAND</filename> specify which executables
|
||||
to use for the CVS checkout or update.
|
||||
</para>
|
||||
<para>
|
||||
The URN for the CVS fetcher is cvs.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The supported parameters are module, tag, date,
|
||||
method, localdir, rshand scmdata.
|
||||
The module specifies which module to check out,
|
||||
the tag describes which CVS TAG should be used for
|
||||
the checkout.
|
||||
By default, the TAG is empty.
|
||||
A date can be specified to override the
|
||||
<filename>SRCDATE</filename> of the
|
||||
configuration to checkout a specific date.
|
||||
The special value of "now" will cause the checkout to be
|
||||
updated on every build.
|
||||
method is by default pserver.
|
||||
If ext is used the rsh parameter will be evaluated
|
||||
and <filename>CVS_RSH</filename> will be set.
|
||||
Finally, localdir is used to checkout into a special
|
||||
directory relative to <filename>CVSDIR</filename>.
|
||||
<literallayout class='monospaced'>
|
||||
<para>
|
||||
This fetcher honors the variables <filename>CVSDIR</filename>,
|
||||
<filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
|
||||
<filename>UPDATECOMMAND_cvs</filename>.
|
||||
The
|
||||
<link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
|
||||
variable specifies where a
|
||||
temporary checkout is saved.
|
||||
The
|
||||
<link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
|
||||
variable specifies which date to
|
||||
use when doing the fetching.
|
||||
The special value of "now" causes the checkout to be
|
||||
updated on every build.
|
||||
The <filename>FETCHCOMMAND</filename> and
|
||||
<filename>UPDATECOMMAND</filename> variables specify the executables
|
||||
to use for the CVS checkout or update.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The supported parameters are as follows:
|
||||
<itemizedlist>
|
||||
<listitem><para><emphasis>"module":</emphasis>
|
||||
Specifies the module to check out.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"tag":</emphasis>
|
||||
Describes which CVS TAG should be used for
|
||||
the checkout.
|
||||
By default, the TAG is empty.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"date":</emphasis>
|
||||
Specifies a date.
|
||||
If no "date" is specified, the
|
||||
<link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
|
||||
of the configuration is used to checkout a specific date.
|
||||
The special value of "now" causes the checkout to be
|
||||
updated on every build.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"method":</emphasis>
|
||||
By default <filename>pserver</filename>.
|
||||
If "method" is set to "ext", BitBake examines the "rsh"
|
||||
parameter and sets <filename>CVS_RSH</filename>.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"localdir":</emphasis>
|
||||
Used to checkout force into a special
|
||||
directory relative to <filename>CVSDIR</filename>.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"rsh"</emphasis>
|
||||
Used in conjunction with the "method" parameter.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"scmdata":</emphasis>
|
||||
I need a description for this.
|
||||
</para></listitem>
|
||||
</itemizedlist>
|
||||
Following are two examples using cvs:
|
||||
<literallayout class='monospaced'>
|
||||
SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
|
||||
SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='http-ftp-fetcher'>
|
||||
<title>HTTP/FTP fetcher</title>
|
||||
<section id='http-ftp-fetcher'>
|
||||
<title>HTTP/FTP fetcher</title>
|
||||
|
||||
<para>
|
||||
The URNs for the HTTP/FTP fetcher are http, https, and ftp.
|
||||
This fetcher honors the variables
|
||||
<filename>FETCHCOMMAND_wget</filename>.
|
||||
<filename>FETCHCOMMAND</filename> contains the command used
|
||||
for fetching.
|
||||
“${URI}” and “${FILES}” will be replaced by the URI and
|
||||
basename of the file to be fetched.
|
||||
<literallayout class='monospaced'>
|
||||
<para>
|
||||
The URNs for the HTTP/FTP fetcher are http, https, and ftp.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
This fetcher honors the variables
|
||||
<filename>FETCHCOMMAND_wget</filename>.
|
||||
The <filename>FETCHCOMMAND</filename> variable
|
||||
contains the command used for fetching.
|
||||
“${URI}” and “${FILES}” are replaced by the URI and
|
||||
the base name of the file to be fetched.
|
||||
<literallayout class='monospaced'>
|
||||
SRC_URI = "http://oe.handhelds.org/not_there.aac"
|
||||
SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
|
||||
SRC_URI = "ftp://you@oe.handheld.sorg/home/you/secret.plan"
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='svn-fetcher'>
|
||||
<title>SVN Fetcher</title>
|
||||
<section id='svn-fetcher'>
|
||||
<title>SVN Fetcher</title>
|
||||
|
||||
<para>
|
||||
The URN for the SVN fetcher is svn.
|
||||
</para>
|
||||
<para>
|
||||
The URN for the SVN fetcher is svn.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
This fetcher honors the variables
|
||||
<filename>FETCHCOMMAND_svn</filename>,
|
||||
<filename>SVNDIR</filename>,
|
||||
and <filename>SRCREV</filename>.
|
||||
<filename>FETCHCOMMAND</filename> contains the
|
||||
subversion command.
|
||||
<filename>SRCREV</filename> specifies which revision
|
||||
to use when doing the fetching.
|
||||
</para>
|
||||
<para>
|
||||
This fetcher honors the variables
|
||||
<filename>FETCHCOMMAND_svn</filename>,
|
||||
<filename>SVNDIR</filename>,
|
||||
and
|
||||
<link linkend='var-SRCREV'><filename>SRCREV</filename></link>.
|
||||
The <filename>FETCHCOMMAND</filename> variable contains the
|
||||
<filename>subversion</filename> command.
|
||||
The <filename>SRCREV</filename> variable specifies which revision
|
||||
to use when doing the fetching.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The supported parameters are proto, rev and scmdata.
|
||||
proto is the Subversion protocol, rev is the
|
||||
Subversion revision.
|
||||
If scmdata is set to “keep”, the “.svn” directories will
|
||||
be available during compile-time.
|
||||
<literallayout class='monospaced'>
|
||||
<para>
|
||||
The supported parameters are as follows:
|
||||
<itemizedlist>
|
||||
<listitem><para><emphasis>"proto":</emphasis>
|
||||
The Subversion protocol.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"rev":</emphasis>
|
||||
The Subversion revision.
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"scmdata":</emphasis>
|
||||
Set to "keep" causes the “.svn” directories
|
||||
to be available during compile-time.
|
||||
</para></listitem>
|
||||
</itemizedlist>
|
||||
Following are two examples using svn:
|
||||
<literallayout class='monospaced'>
|
||||
SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
|
||||
SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='git-fetcher'>
|
||||
<title>GIT Fetcher</title>
|
||||
<section id='git-fetcher'>
|
||||
<title>GIT Fetcher</title>
|
||||
|
||||
<para>
|
||||
The URN for the GIT Fetcher is git.
|
||||
</para>
|
||||
<para>
|
||||
The URN for the Git Fetcher is git.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The variable <filename>GITDIR</filename> will be used as the
|
||||
base directory where the Git tree is cloned to.
|
||||
</para>
|
||||
<para>
|
||||
The variable <filename>GITDIR</filename> is used as the
|
||||
base directory in which the Git tree is cloned.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The parameters are tag, protocol, and scmdata.
|
||||
The tag parameter is a Git tag, the default is “master”.
|
||||
The protocol tag is the Git protocol to use and defaults to “git”
|
||||
if a hostname is set, otherwise it is “file”.
|
||||
If scmdata is set to “keep”, the “.git” directory will be available
|
||||
during compile-time.
|
||||
<literallayout class='monospaced'>
|
||||
<para>
|
||||
The supported parameters are as follows:
|
||||
<itemizedlist>
|
||||
<listitem><para><emphasis>"tag":</emphasis>
|
||||
The Git tag.
|
||||
The default is "master".
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"protocol":</emphasis>
|
||||
The Git protocol.
|
||||
The default is "git" when a hostname is set.
|
||||
If a hostname is not set, the Git protocol is "file".
|
||||
</para></listitem>
|
||||
<listitem><para><emphasis>"scmdata":</emphasis>
|
||||
When set to “keep”, the “.git” directory is available
|
||||
during compile-time.
|
||||
</para></listitem>
|
||||
</itemizedlist>
|
||||
Following are two examples using git:
|
||||
<literallayout class='monospaced'>
|
||||
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
|
||||
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
|
||||
</literallayout>
|
||||
</para>
|
||||
</literallayout>
|
||||
</para>
|
||||
</section>
|
||||
</section>
|
||||
</chapter>
|
||||
|
||||
Reference in New Issue
Block a user