mirror of
https://git.yoctoproject.org/poky
synced 2026-02-08 18:02:12 +01:00
85408dfd36
I added more information about how BitBake actually runs a task. The information has to do with how tightly BB controls the execution environment of the build tasks to prevent contamination from the build machine from leaking into the task execution environment. This tight control actually causes some unexpected behavior during builds. For example, when a user exports and BB_ENV_EXTRAWHITE an environment item such as CCACHE_DIR, the effects of the environment item never make it to the BB task execution environment. They only make it to the data store. The user actually has to take some extra steps to export that environment item into the task execution environment. The added information I put into the "Running a Task" section describes these extra steps. Fixes [YOCTO #689] Reported-by: Wolfgang Denk <wd@denx.de> (From yocto-docs rev: f75a9d384c0d5ccaefe7ac2195917531b153cf5e) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
414 lines
21 KiB
XML
414 lines
21 KiB
XML
<!DOCTYPE appendix PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
|
|
|
|
<appendix id='ref-bitbake'>
|
|
|
|
<title>Reference: BitBake</title>
|
|
|
|
<para>
|
|
BitBake is a program written in Python that interprets the metadata that makes up the Yocto Project.
|
|
At some point, developers wonder what actually happens when you enter:
|
|
<literallayout class='monospaced'>
|
|
$ bitbake core-image-sato
|
|
</literallayout>
|
|
</para>
|
|
|
|
<para>
|
|
This appendix provides an overview of what happens behind the scenes from BitBake's perspective.
|
|
</para>
|
|
|
|
<note>
|
|
BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships.
|
|
As such, it has no real knowledge of what the tasks being executed actually do.
|
|
BitBake just considers a list of tasks with dependencies and handles metadata
|
|
that consists of variables in a certain format that get passed to the tasks.
|
|
</note>
|
|
|
|
<section id='ref-bitbake-parsing'>
|
|
<title>Parsing</title>
|
|
|
|
<para>
|
|
BitBake parses configuration files, classes, and <filename>.bb</filename> files.
|
|
</para>
|
|
|
|
<para>
|
|
The first thing BitBake does is look for the <filename>bitbake.conf</filename> file.
|
|
The Yocto Project keeps this file in the Yocto Project file's <filename>meta/conf/</filename>
|
|
directory.
|
|
BitBake finds it by examining the <filename>BBPATH</filename> environment
|
|
variable and looking for the <filename>meta/conf/</filename>
|
|
directory.
|
|
</para>
|
|
|
|
<para>
|
|
In the Yocto Project, <filename>bitbake.conf</filename> lists other configuration
|
|
files to include from a <filename>conf/</filename>
|
|
directory below the directories listed in <filename>BBPATH</filename>.
|
|
In general, the most important configuration file from a user's perspective
|
|
is <filename>local.conf</filename>, which contains a user's customized
|
|
settings for the Yocto Project build environment.
|
|
Other notable configuration files are the distribution
|
|
configuration file (set by the
|
|
<filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable)
|
|
and the machine configuration file
|
|
(set by the
|
|
<filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable).
|
|
The <filename>DISTRO</filename> and <filename>MACHINE</filename> environment
|
|
variables are both usually set in
|
|
the <filename>local.conf</filename> file.
|
|
Valid distribution
|
|
configuration files are available in the <filename>meta/conf/distro/</filename> directory
|
|
and valid machine configuration
|
|
files in the <filename>meta/conf/machine/</filename> directory.
|
|
Within the <filename>meta/conf/machine/include/</filename>
|
|
directory are various <filename>tune-*.inc</filename> configuration files that provide common
|
|
"tuning" settings specific to and shared between particular architectures and machines.
|
|
</para>
|
|
|
|
<para>
|
|
After the parsing of the configuration files, some standard classes are included.
|
|
The <filename>base.bbclass</filename> file is always included.
|
|
Other classes that are specified in the configuration using the
|
|
<filename><link linkend='var-INHERIT'>INHERIT</link></filename>
|
|
variable are also included.
|
|
Class files are searched for in a <filename>classes</filename> subdirectory
|
|
under the paths in <filename>BBPATH</filename> in the same way as
|
|
configuration files.
|
|
</para>
|
|
|
|
<para>
|
|
After classes are included, the variable
|
|
<filename><link linkend='var-BBFILES'>BBFILES</link></filename>
|
|
is set, usually in
|
|
<filename>local.conf</filename>, and defines the list of places to search for
|
|
<filename>.bb</filename> files.
|
|
By default, the <filename>BBFILES</filename> variable specifies the
|
|
<filename>meta/recipes-*/</filename> directory within Poky.
|
|
Adding extra content to <filename>BBFILES</filename> is best achieved through the use of
|
|
BitBake layers as described in the
|
|
<link linkend='usingpoky-changes-layers'>BitBake Layers</link> section.
|
|
</para>
|
|
|
|
<para>
|
|
BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and
|
|
stores the values of various variables.
|
|
In summary, for each <filename>.bb</filename>
|
|
file the configuration plus the base class of variables are set, followed
|
|
by the data in the <filename>.bb</filename> file
|
|
itself, followed by any inherit commands that
|
|
<filename>.bb</filename> file might contain.
|
|
</para>
|
|
|
|
<para>
|
|
Because parsing <filename>.bb</filename> files is a time
|
|
consuming process, a cache is kept to speed up subsequent parsing.
|
|
This cache is invalid if the timestamp of the <filename>.bb</filename>
|
|
file itself changes, or if the timestamps of any of the include,
|
|
configuration or class files the <filename>.bb</filename>
|
|
file depends on changes.
|
|
</para>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-providers'>
|
|
<title>Preferences and Providers</title>
|
|
|
|
<para>
|
|
Once all the <filename>.bb</filename> files have been
|
|
parsed, BitBake starts to build the target (<filename>core-image-sato</filename>
|
|
in the previous section's example) and looks for providers of that target.
|
|
Once a provider is selected, BitBake resolves all the dependencies for
|
|
the target.
|
|
In the case of <filename>core-image-sato</filename>, it would lead to
|
|
<filename>task-base.bb</filename>,
|
|
which in turn leads to packages like <filename>Contacts</filename>,
|
|
<filename>Dates</filename> and <filename>BusyBox</filename>.
|
|
These packages in turn depend on <filename>eglibc</filename> and the toolchain.
|
|
</para>
|
|
|
|
<para>
|
|
Sometimes a target might have multiple providers.
|
|
A common example is "virtual/kernel", which is provided by each kernel package.
|
|
Each machine often selects the best kernel provider by using a line similar to the
|
|
following in the machine configuration file:
|
|
</para>
|
|
|
|
<literallayout class='monospaced'>
|
|
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
|
|
</literallayout>
|
|
|
|
<para>
|
|
The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename>
|
|
is the provider with the same name as the target.
|
|
</para>
|
|
|
|
<para>
|
|
Understanding how providers are chosen is made complicated by the fact
|
|
that multiple versions might exist.
|
|
BitBake defaults to the highest version of a provider.
|
|
Version comparisons are made using the same method as Debian.
|
|
You can use the
|
|
<filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
|
|
variable to specify a particular version (usually in the distro configuration).
|
|
You can influence the order by using the
|
|
<filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename>
|
|
variable.
|
|
By default, files have a preference of "0".
|
|
Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
|
|
package unlikely to be used unless it is explicitly referenced.
|
|
Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
|
|
<filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
|
|
<filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
|
|
versions until they have undergone sufficient testing to be considered stable.
|
|
</para>
|
|
|
|
<para>
|
|
In summary, BitBake has created a list of providers, which is prioritized, for each target.
|
|
</para>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-dependencies'>
|
|
<title>Dependencies</title>
|
|
|
|
<para>
|
|
Each target BitBake builds consists of multiple tasks such as
|
|
<filename>fetch</filename>, <filename>unpack</filename>,
|
|
<filename>patch</filename>, <filename>configure</filename>,
|
|
and <filename>compile</filename>.
|
|
For best performance on multi-core systems, BitBake considers each task as an independent
|
|
entity with its own set of dependencies.
|
|
</para>
|
|
|
|
<para>
|
|
Dependencies are defined through several variables.
|
|
You can find information about variables BitBake uses in the
|
|
<ulink url='http://bitbake.berlios.de/manual/'>BitBake manual</ulink>.
|
|
At a basic level, it is sufficient to know that BitBake uses the
|
|
<filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and
|
|
<filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> variables when
|
|
calculating dependencies.
|
|
</para>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-tasklist'>
|
|
<title>The Task List</title>
|
|
|
|
<para>
|
|
Based on the generated list of providers and the dependency information,
|
|
BitBake can now calculate exactly what tasks it needs to run and in what
|
|
order it needs to run them.
|
|
The build now starts with BitBake forking off threads up to the limit set in the
|
|
<filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable.
|
|
BitBake continues to fork threads as long as there are tasks ready to run,
|
|
those tasks have all their dependencies met, and the thread threshold has not been
|
|
exceeded.
|
|
</para>
|
|
|
|
<para>
|
|
It is worth noting that you can greatly speed up the build time by properly setting
|
|
the <filename>BB_NUMBER_THREADS</filename> variable.
|
|
See the
|
|
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html#building-image'>
|
|
Building an Image</ulink> section in the
|
|
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html'>
|
|
Yocto Project Quick Start</ulink> for more information.
|
|
</para>
|
|
|
|
<para>
|
|
As each task completes, a timestamp is written to the directory specified by the
|
|
<filename><link linkend='var-STAMPS'>STAMPS</link></filename> variable (usually
|
|
<filename>build/tmp/stamps/*/</filename>).
|
|
On subsequent runs, BitBake looks at the <filename>STAMPS</filename> directory and does not rerun
|
|
tasks that are already completed unless a timestamp is found to be invalid.
|
|
Currently, invalid timestamps are only considered on a per
|
|
<filename>.bb</filename> file basis.
|
|
So, for example, if the configure stamp has a timestamp greater than the
|
|
compile timestamp for a given target, then the compile task would rerun.
|
|
Running the compile task again, however, has no effect on other providers
|
|
that depend on that target.
|
|
This behavior could change or become configurable in future versions of BitBake.
|
|
</para>
|
|
|
|
<note>
|
|
Some tasks are marked as "nostamp" tasks.
|
|
No timestamp file is created when these tasks are run.
|
|
Consequently, "nostamp" tasks are always rerun.
|
|
</note>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-runtask'>
|
|
<title>Running a Task</title>
|
|
|
|
<para>
|
|
Tasks can either be a shell task or a Python task.
|
|
For shell tasks, BitBake writes a shell script to
|
|
<filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script.
|
|
The generated shell script contains all the exported variables, and the shell functions
|
|
with all variables expanded.
|
|
Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
|
|
Looking at the expanded shell functions in the run file and the output in the log files
|
|
is a useful debugging technique.
|
|
</para>
|
|
|
|
<para>
|
|
For Python tasks, BitBake executes the task internally and logs information to the
|
|
controlling terminal.
|
|
Future versions of BitBake will write the functions to files similar to the way
|
|
shell tasks are handled.
|
|
Logging will be handled in way similar to shell tasks as well.
|
|
</para>
|
|
|
|
<para>
|
|
Once all the tasks have been completed BitBake exits.
|
|
</para>
|
|
|
|
<para>
|
|
When running a task, BitBake tightly controls the execution environment
|
|
of the build tasks to make sure unwanted contamination from the build machine
|
|
cannot influence the build.
|
|
Consequently, if you do want something to get passed into the build
|
|
task's environment, you must take a few steps:
|
|
<orderedlist>
|
|
<listitem><para>Tell BitBake to load what you want from the environment
|
|
into the data store.
|
|
You can do so through the <filename>BB_ENV_WHITELIST</filename>
|
|
variable.
|
|
For example, assume you want to prevent the build system from
|
|
accessing your <filename>$HOME/.ccache</filename> directory.
|
|
The following command tells BitBake to load
|
|
<filename>CCACHE_DIR</filename> from the environment into the data
|
|
store:
|
|
<literallayout class='monospaced'>
|
|
export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR"
|
|
</literallayout></para></listitem>
|
|
<listitem><para>Tell BitBake to export what you have loaded into the
|
|
environment store to the task environment of every running task.
|
|
Loading something from the environment into the data store
|
|
(previous step) only makes it available in the datastore.
|
|
To export it to the task environment of every running task,
|
|
use a command similar to the following in your
|
|
<filename>local.conf</filename> or distro configuration file:
|
|
<literallayout class='monospaced'>
|
|
export CCACHE_DIR
|
|
</literallayout></para></listitem>
|
|
</orderedlist>
|
|
</para>
|
|
|
|
<note>
|
|
A side effect of the previous steps is that BitBake records the variable
|
|
as a dependency of the build process in things like the shared state
|
|
checksums.
|
|
If doing so results in unnecessary rebuilds of tasks, you can whitelist the
|
|
variable so that the shared state code ignores the dependency when it creates
|
|
checksums.
|
|
For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
|
|
example in <xref linkend='checksums'>Checksums (Signatures)</xref>.
|
|
</note>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-commandline'>
|
|
<title>BitBake Command Line</title>
|
|
|
|
<para>
|
|
Following is the BitBake help output:
|
|
</para>
|
|
|
|
<screen>
|
|
$ bitbake --help
|
|
Usage: bitbake [options] [package ...]
|
|
|
|
Executes the specified task (default is 'build') for a given set of BitBake files.
|
|
It expects that BBFILES is defined, which is a space separated list of files to
|
|
be executed. BBFILES does support wildcards.
|
|
Default BBFILES are the .bb files in the current directory.
|
|
|
|
Options:
|
|
--version show program's version number and exit
|
|
-h, --help show this help message and exit
|
|
-b BUILDFILE, --buildfile=BUILDFILE
|
|
execute the task against this .bb file, rather than a
|
|
package from BBFILES. Does not handle any
|
|
dependencies.
|
|
-k, --continue continue as much as possible after an error. While the
|
|
target that failed, and those that depend on it,
|
|
cannot be remade, the other dependencies of these
|
|
targets can be processed all the same.
|
|
-a, --tryaltconfigs continue with builds by trying to use alternative
|
|
providers where possible.
|
|
-f, --force force run of specified cmd, regardless of stamp status
|
|
-c CMD, --cmd=CMD Specify task to execute. Note that this only executes
|
|
the specified task for the providee and the packages
|
|
it depends on, i.e. 'compile' does not implicitly call
|
|
stage for the dependencies (IOW: use only if you know
|
|
what you are doing). Depending on the base.bbclass a
|
|
listtasks tasks is defined and will show available
|
|
tasks
|
|
-r PREFILE, --read=PREFILE
|
|
read the specified file before bitbake.conf
|
|
-R POSTFILE, --postread=POSTFILE
|
|
read the specified file after bitbake.conf
|
|
-v, --verbose output more chit-chat to the terminal
|
|
-D, --debug Increase the debug level. You can specify this more
|
|
than once.
|
|
-n, --dry-run don't execute, just go through the motions
|
|
-S, --dump-signatures
|
|
don't execute, just dump out the signature
|
|
construction information
|
|
-p, --parse-only quit after parsing the BB files (developers only)
|
|
-s, --show-versions show current and preferred versions of all packages
|
|
-e, --environment show the global or per-package environment (this is
|
|
what used to be bbread)
|
|
-g, --graphviz emit the dependency trees of the specified packages in
|
|
the dot syntax
|
|
-I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
|
|
Assume these dependencies don't exist and are already
|
|
provided (equivalent to ASSUME_PROVIDED). Useful to
|
|
make dependency graphs more appealing
|
|
-l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
|
|
Show debug logging for the specified logging domains
|
|
-P, --profile profile the command and print a report
|
|
-u UI, --ui=UI userinterface to use
|
|
-t SERVERTYPE, --servertype=SERVERTYPE
|
|
Choose which server to use, none, process or xmlrpc
|
|
--revisions-changed Set the exit code depending on whether upstream
|
|
floating revisions have changed or not
|
|
</screen>
|
|
</section>
|
|
|
|
<section id='ref-bitbake-fetchers'>
|
|
<title>Fetchers</title>
|
|
|
|
<para>
|
|
BitBake also contains a set of "fetcher" modules that allow
|
|
retrieval of source code from various types of sources.
|
|
For example, BitBake can get source code from a disk with the metadata, from websites,
|
|
from remote shell accounts or from Source Code Management (SCM) systems
|
|
like <filename>cvs/subversion/git</filename>.
|
|
</para>
|
|
|
|
<para>
|
|
Fetchers are usually triggered by entries in
|
|
<filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>.
|
|
You can find information about the options and formats of entries for specific
|
|
fetchers in the <ulink url='http://bitbake.berlios.de/manual/'>BitBake manual</ulink>.
|
|
</para>
|
|
|
|
<para>
|
|
One useful feature for certain Source Code Manager (SCM) fetchers is the ability to
|
|
"auto-update" when the upstream SCM changes version.
|
|
Since this ability requires certain functionality from the SCM, not all
|
|
systems support it.
|
|
Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update".
|
|
This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename>
|
|
variable.
|
|
See the
|
|
<link linkend='platdev-appdev-srcrev'>Development Within Yocto Project for a Package that Uses
|
|
an External SCM</link> section for more information.
|
|
</para>
|
|
|
|
</section>
|
|
|
|
</appendix>
|
|
<!--
|
|
vim: expandtab tw=80 ts=4 spell spelllang=en_gb
|
|
-->
|