poky/bitbake/doc/user-manual/user-manual-fetching.xml

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<chapter>
<title>File Download Support</title>

    <para>
        BitBake's <filename>fetch</filename> and
        <filename>fetch2</filename> modules support downloading
        files.
        This chapter provides an overview of the fetching process
        and also presents sections on each of the fetchers BitBake
        supports.
        <note>
            The original <filename>fetch</filename> code, for all
            practical purposes, has been replaced by
            <filename>fetch2</filename> code.
            Consequently, the information in this chapter does not
            apply to <filename>fetch</filename>.
        </note>
    </para>

    <section id='file-download-overview'>
        <title>Overview</title>

        <para>
            When BitBake starts to execute, the very first thing
            it does is to fetch the source files needed.
            This section overviews the process.
        </para>

        <para>
            When BitBake goes looking for source files, it follows a search
            order:
            <orderedlist>
                <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
                    BitBake first uses pre-mirrors to try and find source
                    files.
                    These locations are defined using the
                    <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
                    variable.
                    </para></listitem>
                <listitem><para><emphasis>Source URI:</emphasis>
                    If pre-mirrors fail, BitBake uses
                    <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
                    </para></listitem>
                <listitem><para><emphasis>Mirror Sites:</emphasis>
                    If fetch failures occur using <filename>SRC_URI</filename>,
                    BitBake next uses mirror location as defined by the
                    <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
                    variable.
                    </para></listitem>
            </orderedlist>
        </para>

        <para>
            Because cross-URLs are supported, it is possible to mirror
            a Git repository on an HTTP server as a tarball.
            Here are some examples that show commonly used mirror
            definitions:
            <literallayout class='monospaced'>
     PREMIRRORS ?= "\
         bzr://.*/.*   http://somemirror.org/sources/ \n \
         cvs://.*/.*   http://somemirror.org/sources/ \n \
         git://.*/.*   http://somemirror.org/sources/ \n \
         hg://.*/.*    http://somemirror.org/sources/ \n \
         osc://.*/.*   http://somemirror.org/sources/ \n \
         p4://.*/.*    http://somemirror.org/sources/ \n \
         svk://.*/.*   http://somemirror.org/sources/ \n \
         svn://.*/.*   http://somemirror.org/sources/ \n"

     MIRRORS =+ "\
         ftp://.*/.*      http://somemirror.org/sources/ \n \
         http://.*/.*     http://somemirror.org/sources/ \n \
         https://.*/.*    http://somemirror.org/sources/ \n"
            </literallayout>
        </para>

        <para>
            Any source files that are not local (i.e. downloaded from
            the Internet) are placed into the download directory,
            which is specified by
            <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>.
        </para>

        <para>
            For non-local archive downloads, the fetcher code can verify
            sha256 and md5 checksums to ensure
            the archives have been downloaded correctly.
            You can specify these checksums by using the
            <filename>SRC_URI</filename> variable with the appropriate
            varflags as follows:
            <literallayout class='monospaced'>
     SRC_URI[md5sum]
     SRC_URI[sha256sum]
            </literallayout>
            You can also specify the checksums as parameters on the
            <filename>SRC_URI</filename> as shown below:
            <literallayout class='monospaced'>
     SRC_URI="http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
            </literallayout>
            If
            <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
            is set, any download without a checksum triggers an error message.
            In cases where multiple files are listed using
            <filename>SRC_URI</filename>, the name parameter is used
            assign names to the URLs and these are then specified
            in the checksums using the following form:
            <literallayout class='monospaced'>
     SRC_URI[name.sha256sum]
            </literallayout>
        </para>
    </section>

    <section id='bb-fetchers'>
        <title>Fetchers</title>

        <para>
            As mentioned in the previous section, the
            <filename>SRC_URI</filename> is normally used to
            tell BitBake which files to fetch.
            And, the fetcher BitBake uses depends on the how
            <filename>SRC_URI</filename> is set.
        </para>

        <para>
            These next few sections describe the available fetchers and
            their options.
            Each fetcher honors a set of variables URI parameters,
            which are separated by semi-colon characters and consist
            of a key and a value.
            The semantics of the variables and parameters are
            defined by the fetcher.
            BitBake tries to have consistent semantics between the
            different fetchers.
        </para>

        <section id='local-file-fetcher'>
            <title>Local file fetcher</title>

            <para>
                The URN for the local file fetcher is file.
            </para>

            <para>
                The filename can be either absolute or relative.
                If the filename is relative,
                <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
                is used.
                Failing that,
                <link linkend='var-FILESDIR'><filename>FILESDIR</filename></link>
                is used to find the appropriate relative file.
            </para>

            <para>
                The metadata usually extend these variables to include
                variations of the values in
                <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
                Single files and complete directories can be specified.
                <literallayout class='monospaced'>
     SRC_URI = "file://relativefile.patch"
     SRC_URI = "file://relativefile.patch;this=ignored"
     SRC_URI = "file:///Users/ich/very_important_software"
                </literallayout>
            </para>
        </section>

        <section id='cvs-fetcher'>
            <title>CVS fetcher</title>

            <para>
                The URN for the CVS fetcher is cvs.
            </para>

            <para>
                This fetcher honors the variables <filename>CVSDIR</filename>,
                <filename>SRCDATE</filename>, <filename>FETCHCOMMAND_cvs</filename>,
                <filename>UPDATECOMMAND_cvs</filename>.
                The
                <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
                variable specifies where a
                temporary checkout is saved.
                The
                <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
                variable specifies which date to
                use when doing the fetching.
                The special value of "now" causes the checkout to be
                updated on every build.
                The <filename>FETCHCOMMAND</filename> and
                <filename>UPDATECOMMAND</filename> variables specify the executables
                to use for the CVS checkout or update.
            </para>

            <para>
                The supported parameters are as follows:
                <itemizedlist>
                    <listitem><para><emphasis>"module":</emphasis>
                        Specifies the module to check out.
                        </para></listitem>
                    <listitem><para><emphasis>"tag":</emphasis>
                        Describes which CVS TAG should be used for
                        the checkout.
                        By default, the TAG is empty.
                        </para></listitem>
                    <listitem><para><emphasis>"date":</emphasis>
                        Specifies a date.
                        If no "date" is specified, the
                        <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
                        of the configuration is used to checkout a specific date.
                        The special value of "now" causes the checkout to be
                        updated on every build.
                        </para></listitem>
                    <listitem><para><emphasis>"method":</emphasis>
                        By default <filename>pserver</filename>.
                        If "method" is set to "ext", BitBake examines the "rsh"
                        parameter and sets <filename>CVS_RSH</filename>.
                        </para></listitem>
                    <listitem><para><emphasis>"localdir":</emphasis>
                        Used to checkout force into a special
                        directory relative to <filename>CVSDIR</filename>.
                        </para></listitem>
                    <listitem><para><emphasis>"rsh"</emphasis>
                        Used in conjunction with the "method" parameter.
                        </para></listitem>
                    <listitem><para><emphasis>"scmdata":</emphasis>
                        I need a description for this.
                        </para></listitem>
                </itemizedlist>
                Following are two examples using cvs:
                <literallayout class='monospaced'>
     SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
     SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
                </literallayout>
            </para>
        </section>

        <section id='http-ftp-fetcher'>
            <title>HTTP/FTP fetcher</title>

            <para>
                The URNs for the HTTP/FTP fetcher are http, https, and ftp.
            </para>

            <para>
                This fetcher honors the variables
                <filename>FETCHCOMMAND_wget</filename>.
                The <filename>FETCHCOMMAND</filename> variable
                contains the command used for fetching.
                “${URI}” and “${FILES}” are replaced by the URI and
                the base name of the file to be fetched.
                <literallayout class='monospaced'>
     SRC_URI = "http://oe.handhelds.org/not_there.aac"
     SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
     SRC_URI = "ftp://you@oe.handheld.sorg/home/you/secret.plan"
                </literallayout>
            </para>
        </section>

        <section id='svn-fetcher'>
            <title>SVN Fetcher</title>

            <para>
                The URN for the SVN fetcher is svn.
            </para>

            <para>
                This fetcher honors the variables
                <filename>FETCHCOMMAND_svn</filename>,
                <filename>SVNDIR</filename>,
                and
                <link linkend='var-SRCREV'><filename>SRCREV</filename></link>.
                The <filename>FETCHCOMMAND</filename> variable contains the
                <filename>subversion</filename> command.
                The <filename>SRCREV</filename> variable specifies which revision
                to use when doing the fetching.
            </para>

            <para>
                The supported parameters are as follows:
                <itemizedlist>
                    <listitem><para><emphasis>"proto":</emphasis>
                        The Subversion protocol.
                        </para></listitem>
                    <listitem><para><emphasis>"rev":</emphasis>
                        The Subversion revision.
                        </para></listitem>
                    <listitem><para><emphasis>"scmdata":</emphasis>
                        Set to "keep" causes the “.svn” directories
                        to be available during compile-time.
                        </para></listitem>
                </itemizedlist>
                Following are two examples using svn:
                <literallayout class='monospaced'>
     SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
     SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
                </literallayout>
            </para>
        </section>

        <section id='git-fetcher'>
            <title>GIT Fetcher</title>

            <para>
                The URN for the Git Fetcher is git.
            </para>

            <para>
                The variable <filename>GITDIR</filename> is used as the
                base directory in which the Git tree is cloned.
            </para>

            <para>
                The supported parameters are as follows:
                <itemizedlist>
                    <listitem><para><emphasis>"tag":</emphasis>
                        The Git tag.
                        The default is "master".
                        </para></listitem>
                    <listitem><para><emphasis>"protocol":</emphasis>
                        The Git protocol.
                        The default is "git" when a hostname is set.
                        If a hostname is not set, the Git protocol is "file".
                        </para></listitem>
                    <listitem><para><emphasis>"scmdata":</emphasis>
                        When set to “keep”, the “.git” directory is available
                        during compile-time.
                        </para></listitem>
                </itemizedlist>
                Following are two examples using git:
                <literallayout class='monospaced'>
     SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
     SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
                </literallayout>
            </para>
        </section>
    </section>
</chapter>