kernel-dev: Updated kernel configuration section

A lot of rewriting here in this section to get it up to speed.
Also, moved that final section on determining hardware and
non-hardware features into an appendix where it belonged.

(From yocto-docs rev: 752e80d6ae8f81a0de7743b11b010d0ef36b314b)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark
2017-09-26 14:33:56 -07:00
committed by Richard Purdie
parent fcdfe09d9c
commit b8dfb03764
2 changed files with 241 additions and 239 deletions

View File

@@ -10,8 +10,8 @@
work with the Yocto Project Linux kernel. work with the Yocto Project Linux kernel.
These tasks include preparing your host development system for These tasks include preparing your host development system for
kernel development, preparing a layer, modifying an existing recipe, kernel development, preparing a layer, modifying an existing recipe,
patching the kernel, iterative development, working with your own sources, patching the kernel, configuring the kernel, iterative development,
and incorporating out-of-tree modules. working with your own sources, and incorporating out-of-tree modules.
<note> <note>
The examples presented in this chapter work with the Yocto Project The examples presented in this chapter work with the Yocto Project
2.4 Release and forward. 2.4 Release and forward.
@@ -1447,34 +1447,6 @@
</para> </para>
</section> </section>
<section id='configuring-the-kernel'> <section id='configuring-the-kernel'>
<title>Configuring the Kernel</title> <title>Configuring the Kernel</title>
@@ -1516,17 +1488,22 @@
</para> </para>
<para> <para>
To use the <filename>menuconfig</filename> tool in the Yocto Project development To use the <filename>menuconfig</filename> tool in the Yocto
environment, you must launch it using BitBake. Project development environment, you must launch it using
BitBake.
Thus, the environment must be set up using the Thus, the environment must be set up using the
<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink> <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>
script found in the script found in the
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
You must also be sure of the state of your build in the You must also be sure of the state of your build's configuration
in the
<ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
The following commands run <filename>menuconfig</filename> The following commands initialize the BitBake environment,
assuming the Source Directory's top-level folder is run the
<filename>~/poky</filename>: <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-kernel_configme'><filename>do_kernel_configme</filename></ulink>
task, and launch <filename>menuconfig</filename>.
These commands assume the Source Directory's top-level folder
is <filename>~/poky</filename>:
<literallayout class='monospaced'> <literallayout class='monospaced'>
$ cd poky $ cd poky
$ source oe-init-build-env $ source oe-init-build-env
@@ -1542,79 +1519,81 @@
</para> </para>
<para> <para>
Consider an example that configures the <filename>linux-yocto-3.14</filename> Consider an example that configures the "CONFIG_SMP" setting
kernel. for the <filename>linux-yocto-4.12</filename> kernel.
The OpenEmbedded build system recognizes this kernel as <note>
<filename>linux-yocto</filename>. The OpenEmbedded build system recognizes this kernel as
Thus, the following commands from the shell in which you previously sourced the <filename>linux-yocto</filename> through Metadata (e.g.
environment initialization script cleans the shared state cache and the <ulink url='&YOCTO_DOCS_REF_URL;#var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></ulink><filename>_linux-yocto ?= "12.4%"</filename>).
<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> </note>
directory and then runs <filename>menuconfig</filename>: Once <filename>menuconfig</filename> launches, use the
<literallayout class='monospaced'> interface to navigate through the selections to find the
$ bitbake linux-yocto -c menuconfig configuration settings in which you are interested.
</literallayout> For this example, you deselect "CONFIG_SMP" by clearing the
"Symmetric Multi-Processing Support" option.
Using the interface, you can find the option under
"Processor Type and Features".
To deselect "CONFIG_SMP", use the arrow keys to
highlight "Symmetric Multi-Processing Support" and enter "N"
to clear the asterisk.
When you are finished, exit out and save the change.
</para> </para>
<para> <para>
Once <filename>menuconfig</filename> launches, use the interface Saving the selections updates the <filename>.config</filename>
to navigate through the selections to find the configuration settings in configuration file.
which you are interested. This is the file that the OpenEmbedded build system uses to
For example, consider the <filename>CONFIG_SMP</filename> configuration setting. configure the kernel during the build.
You can find it at <filename>Processor Type and Features</filename> under
the configuration selection <filename>Symmetric Multi-processing Support</filename>.
After highlighting the selection, use the arrow keys to select or deselect
the setting.
When you are finished with all your selections, exit out and save them.
</para>
<para>
Saving the selections updates the <filename>.config</filename> configuration file.
This is the file that the OpenEmbedded build system uses to configure the
kernel during the build.
You can find and examine this file in the Build Directory in You can find and examine this file in the Build Directory in
<filename>tmp/work/</filename>. <filename>tmp/work/</filename>.
The actual <filename>.config</filename> is located in the area where the The actual <filename>.config</filename> is located in the
specific kernel is built. area where the specific kernel is built.
For example, if you were building a Linux Yocto kernel based on the For example, if you were building a Linux Yocto kernel based
Linux 3.14 kernel and you were building a QEMU image targeted for on the <filename>linux-yocto-4.12</filename> kernel and you
were building a QEMU image targeted for
<filename>x86</filename> architecture, the <filename>x86</filename> architecture, the
<filename>.config</filename> file would be located here: <filename>.config</filename> file would be:
<literallayout class='monospaced'> <literallayout class='monospaced'>
poky/build/tmp/work/qemux86-poky-linux/linux-yocto-3.14.11+git1+84f... poky/build/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+gitAUTOINC+eda4d18...
...656ed30-r1/linux-qemux86-standard-build ...967-r0/linux-qemux86-standard-build/.config
</literallayout> </literallayout>
<note> <note>
The previous example directory is artificially split and many of the characters The previous example directory is artificially split and
in the actual filename are omitted in order to make it more readable. many of the characters in the actual filename are omitted
Also, depending on the kernel you are using, the exact pathname in order to make it more readable.
for <filename>linux-yocto-3.14...</filename> might differ. Also, depending on the kernel you are using, the exact
pathname might differ.
</note> </note>
</para> </para>
<para> <para>
Within the <filename>.config</filename> file, you can see the kernel settings. Within the <filename>.config</filename> file, you can see the
For example, the following entry shows that symmetric multi-processor support kernel settings.
is not set: For example, the following entry shows that symmetric
multi-processor support is not set:
<literallayout class='monospaced'> <literallayout class='monospaced'>
# CONFIG_SMP is not set # CONFIG_SMP is not set
</literallayout> </literallayout>
</para> </para>
<para> <para>
A good method to isolate changed configurations is to use a combination of the A good method to isolate changed configurations is to use a
<filename>menuconfig</filename> tool and simple shell commands. combination of the <filename>menuconfig</filename> tool and
Before changing configurations with <filename>menuconfig</filename>, copy the simple shell commands.
existing <filename>.config</filename> and rename it to something else, Before changing configurations with
use <filename>menuconfig</filename> to make <filename>menuconfig</filename>, copy the existing
as many changes as you want and save them, then compare the renamed configuration <filename>.config</filename> and rename it to something else,
use <filename>menuconfig</filename> to make as many changes as
you want and save them, then compare the renamed configuration
file against the newly created file. file against the newly created file.
You can use the resulting differences as your base to create configuration fragments You can use the resulting differences as your base to create
to permanently save in your kernel layer. configuration fragments to permanently save in your kernel
layer.
<note> <note>
Be sure to make a copy of the <filename>.config</filename> and don't just Be sure to make a copy of the <filename>.config</filename>
rename it. file and do not just rename it.
The build system needs an existing <filename>.config</filename> The build system needs an existing
from which to work. <filename>.config</filename> file from which to work.
</note> </note>
</para> </para>
</section> </section>
@@ -1647,7 +1626,8 @@
<filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename> <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename>
directory in your layer's directory in your layer's
<filename>recipes-kernel/linux</filename> directory, and rename <filename>recipes-kernel/linux</filename> directory, and rename
the copied file to "defconfig". the copied file to "defconfig" (e.g.
<filename>~/meta-mylayer/recipes-kernel/linux/linux-yocto/defconfig</filename>).
Then, add the following lines to the linux-yocto Then, add the following lines to the linux-yocto
<filename>.bbappend</filename> file in your layer: <filename>.bbappend</filename> file in your layer:
<literallayout class='monospaced'> <literallayout class='monospaced'>
@@ -1678,8 +1658,7 @@
"<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#changing-the-configuration'>Changing the Configuration</ulink>" "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#changing-the-configuration'>Changing the Configuration</ulink>"
and and
"<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#generating-configuration-files'>Generating Configuration Files</ulink>" "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#generating-configuration-files'>Generating Configuration Files</ulink>"
sections, both in the Yocto Project Linux Kernel Development sections.
Manual.
</para> </para>
</section> </section>
@@ -1687,21 +1666,39 @@
<title>Creating Configuration Fragments</title> <title>Creating Configuration Fragments</title>
<para> <para>
Configuration fragments are simply kernel options that appear in a file Configuration fragments are simply kernel options that
placed where the OpenEmbedded build system can find and apply them. appear in a file placed where the OpenEmbedded build system
Syntactically, the configuration statement is identical to what would appear can find and apply them.
in the <filename>.config</filename> file, which is in the The build system applies configuration fragments after
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: applying configurations from a <filename>defconfig</filename>
<literallayout class='monospaced'> file.
tmp/work/<replaceable>arch</replaceable>-poky-linux/linux-yocto-<replaceable>release_specific_string</replaceable>/linux-<replaceable>arch</replaceable>-<replaceable>build_type</replaceable> Thus, the final kernel configuration is a combination of the
</literallayout> configurations in the <filename>defconfig</filename>
file and then any configuration fragments you provide.
The build system applies fragments on top of and
after applying the existing defconfig file configurations.
</para>
<para>
Syntactically, the configuration statement is identical to
what would appear in the <filename>.config</filename> file,
which is in the
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
<note>
For more information about where the
<filename>.config</filename> file is located, see the
example in the
"<link linkend='using-menuconfig'>Using <filename>menuconfig</filename></link>"
section.
</note>
</para> </para>
<para> <para>
It is simple to create a configuration fragment. It is simple to create a configuration fragment.
For example, issuing the following from the shell creates a configuration fragment For example, issuing the following from the shell creates a
file named <filename>my_smp.cfg</filename> that enables multi-processor support configuration fragment file named
within the kernel: <filename>my_smp.cfg</filename> that enables multi-processor
support within the kernel:
<literallayout class='monospaced'> <literallayout class='monospaced'>
$ echo "CONFIG_SMP=y" >> my_smp.cfg $ echo "CONFIG_SMP=y" >> my_smp.cfg
</literallayout> </literallayout>
@@ -1715,29 +1712,34 @@
<para> <para>
Where do you put your configuration fragment files? Where do you put your configuration fragment files?
You can place these files in the same area pointed to by You can place these files in an area pointed to by
<ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>. <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
as directed by your <filename>bblayers.conf</filename> file,
which is located in your layer.
The OpenEmbedded build system picks up the configuration and The OpenEmbedded build system picks up the configuration and
adds it to the kernel's configuration. adds it to the kernel's configuration.
For example, suppose you had a set of configuration options For example, suppose you had a set of configuration options
in a file called <filename>myconfig.cfg</filename>. in a file called <filename>myconfig.cfg</filename>.
If you put that file inside a directory named If you put that file inside a directory named
<filename>linux-yocto</filename> that resides in the same <filename>linux-yocto</filename> that resides in the same
directory as the kernel's append file and then add a directory as the kernel's append file within your layer
<filename>SRC_URI</filename> statement such as the following and then add the following statements to the kernel's append
to the kernel's append file, those configuration options file, those configuration options will be picked up and applied
will be picked up and applied when the kernel is built. when the kernel is built:
<literallayout class='monospaced'> <literallayout class='monospaced'>
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://myconfig.cfg" SRC_URI += "file://myconfig.cfg"
</literallayout> </literallayout>
</para> </para>
<para> <para>
As mentioned earlier, you can group related configurations into multiple files and As mentioned earlier, you can group related configurations
name them all in the <filename>SRC_URI</filename> statement as well. into multiple files and name them all in the
For example, you could group separate configurations specifically for Ethernet and graphics <filename>SRC_URI</filename> statement as well.
into their own files and add those by using a <filename>SRC_URI</filename> statement like the For example, you could group separate configurations
following in your append file: specifically for Ethernet and graphics into their own files
and add those by using a <filename>SRC_URI</filename> statement
like the following in your append file:
<literallayout class='monospaced'> <literallayout class='monospaced'>
SRC_URI += "file://myconfig.cfg \ SRC_URI += "file://myconfig.cfg \
file://eth.cfg \ file://eth.cfg \
@@ -1797,7 +1799,11 @@
</para></listitem> </para></listitem>
<listitem><para>Separately run the <listitem><para>Separately run the
<filename>do_kernel_configme</filename> and <filename>do_kernel_configme</filename> and
<filename>do_kernel_configcheck</filename> tasks. <filename>do_kernel_configcheck</filename> tasks:
<literallayout class='monospaced'>
$ bitbake linux-yocto -c kernel_configme -f
$ bitbake linux-yocto -c kernel_configcheck -f
</literallayout>
</para></listitem> </para></listitem>
<listitem><para>Take the resulting list of files from the <listitem><para>Take the resulting list of files from the
<filename>do_kernel_configcheck</filename> task <filename>do_kernel_configcheck</filename> task
@@ -1834,135 +1840,14 @@
</para> </para>
<para> <para>
Iteratively working through steps two through four eventually yields Iteratively working through steps two through four eventually
a minimal, streamlined configuration file. yields a minimal, streamlined configuration file.
Once you have the best <filename>.config</filename>, you can build the Linux Once you have the best <filename>.config</filename>, you can
Yocto kernel. build the Linux Yocto kernel.
</para>
</section>
<section id='determining-hardware-and-non-hardware-features-for-the-kernel-configuration-audit-phase'>
<title>Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase</title>
<para>
This section describes part of the kernel configuration audit
phase that most developers can ignore.
During this part of the audit phase, the contents of the final
<filename>.config</filename> file are compared against the
fragments specified by the system.
These fragments can be system fragments, distro fragments,
or user specified configuration elements.
Regardless of their origin, the OpenEmbedded build system
warns the user if a specific option is not included in the
final kernel configuration.
</para>
<para>
In order to not overwhelm the user with configuration warnings,
by default the system only reports on missing "hardware"
options because a missing hardware option could mean a boot
failure or that important hardware is not available.
</para>
<para>
To determine whether or not a given option is "hardware" or
"non-hardware", the kernel Metadata contains files that
classify individual or groups of options as either hardware
or non-hardware.
To better show this, consider a situation where the
Yocto Project kernel cache contains the following files:
<literallayout class='monospaced'>
kernel-cache/features/drm-psb/hardware.cfg
kernel-cache/features/kgdb/hardware.cfg
kernel-cache/ktypes/base/hardware.cfg
kernel-cache/bsp/mti-malta32/hardware.cfg
kernel-cache/bsp/fsl-mpc8315e-rdb/hardware.cfg
kernel-cache/bsp/qemu-ppc32/hardware.cfg
kernel-cache/bsp/qemuarma9/hardware.cfg
kernel-cache/bsp/mti-malta64/hardware.cfg
kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg
kernel-cache/bsp/common-pc/hardware.cfg
kernel-cache/bsp/common-pc-64/hardware.cfg
kernel-cache/features/rfkill/non-hardware.cfg
kernel-cache/ktypes/base/non-hardware.cfg
kernel-cache/features/aufs/non-hardware.kcf
kernel-cache/features/ocf/non-hardware.kcf
kernel-cache/ktypes/base/non-hardware.kcf
kernel-cache/ktypes/base/hardware.kcf
kernel-cache/bsp/qemu-ppc32/hardware.kcf
</literallayout>
The following list provides explanations for the various
files:
<itemizedlist>
<listitem><para><filename>hardware.kcf</filename>:
Specifies a list of kernel Kconfig files that contain
hardware options only.
</para></listitem>
<listitem><para><filename>non-hardware.kcf</filename>:
Specifies a list of kernel Kconfig files that contain
non-hardware options only.
</para></listitem>
<listitem><para><filename>hardware.cfg</filename>:
Specifies a list of kernel
<filename>CONFIG_</filename> options that are hardware,
regardless of whether or not they are within a Kconfig
file specified by a hardware or non-hardware
Kconfig file (i.e. <filename>hardware.kcf</filename> or
<filename>non-hardware.kcf</filename>).
</para></listitem>
<listitem><para><filename>non-hardware.cfg</filename>:
Specifies a list of kernel
<filename>CONFIG_</filename> options that are
not hardware, regardless of whether or not they are
within a Kconfig file specified by a hardware or
non-hardware Kconfig file (i.e.
<filename>hardware.kcf</filename> or
<filename>non-hardware.kcf</filename>).
</para></listitem>
</itemizedlist>
Here is a specific example using the
<filename>kernel-cache/bsp/mti-malta32/hardware.cfg</filename>:
<literallayout class='monospaced'>
CONFIG_SERIAL_8250
CONFIG_SERIAL_8250_CONSOLE
CONFIG_SERIAL_8250_NR_UARTS
CONFIG_SERIAL_8250_PCI
CONFIG_SERIAL_CORE
CONFIG_SERIAL_CORE_CONSOLE
CONFIG_VGA_ARB
</literallayout>
The kernel configuration audit automatically detects these
files (hence the names must be exactly the ones discussed here),
and uses them as inputs when generating warnings about the
final <filename>.config</filename> file.
</para>
<para>
A user-specified kernel Metadata repository, or recipe space
feature, can use these same files to classify options that are
found within its <filename>.cfg</filename> files as hardware
or non-hardware, to prevent the OpenEmbedded build system from
producing an error or warning when an option is not in the
final <filename>.config</filename> file.
</para> </para>
</section> </section>
</section> </section>
<section id='using-an-iterative-development-process'> <section id='using-an-iterative-development-process'>
<title>Using an Iterative Development Process</title> <title>Using an Iterative Development Process</title>

View File

@@ -479,6 +479,123 @@
section for a detailed example that modifies the kernel. section for a detailed example that modifies the kernel.
</para> </para>
</section> </section>
<section id='determining-hardware-and-non-hardware-features-for-the-kernel-configuration-audit-phase'>
<title>Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase</title>
<para>
This section describes part of the kernel configuration audit
phase that most developers can ignore.
For general information on kernel configuration including
<filename>menuconfig</filename>, <filename>defconfig</filename>
files, and configuration fragments, see the
"<link linkend='configuring-the-kernel'>Configuring the Kernel</link>"
section.
</para>
<para>
During this part of the audit phase, the contents of the final
<filename>.config</filename> file are compared against the
fragments specified by the system.
These fragments can be system fragments, distro fragments,
or user-specified configuration elements.
Regardless of their origin, the OpenEmbedded build system
warns the user if a specific option is not included in the
final kernel configuration.
</para>
<para>
By default, in order to not overwhelm the user with
configuration warnings, the system only reports missing
"hardware" options as they could result in a boot
failure or indicate that important hardware is not available.
</para>
<para>
To determine whether or not a given option is "hardware" or
"non-hardware", the kernel Metadata contains files that
classify individual or groups of options as either hardware
or non-hardware.
To better show this, consider a situation where the
Yocto Project kernel cache contains the following files:
<literallayout class='monospaced'>
kernel-cache/features/drm-psb/hardware.cfg
kernel-cache/features/kgdb/hardware.cfg
kernel-cache/ktypes/base/hardware.cfg
kernel-cache/bsp/mti-malta32/hardware.cfg
kernel-cache/bsp/fsl-mpc8315e-rdb/hardware.cfg
kernel-cache/bsp/qemu-ppc32/hardware.cfg
kernel-cache/bsp/qemuarma9/hardware.cfg
kernel-cache/bsp/mti-malta64/hardware.cfg
kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg
kernel-cache/bsp/common-pc/hardware.cfg
kernel-cache/bsp/common-pc-64/hardware.cfg
kernel-cache/features/rfkill/non-hardware.cfg
kernel-cache/ktypes/base/non-hardware.cfg
kernel-cache/features/aufs/non-hardware.kcf
kernel-cache/features/ocf/non-hardware.kcf
kernel-cache/ktypes/base/non-hardware.kcf
kernel-cache/ktypes/base/hardware.kcf
kernel-cache/bsp/qemu-ppc32/hardware.kcf
</literallayout>
The following list provides explanations for the various
files:
<itemizedlist>
<listitem><para>
<filename>hardware.kcf</filename>:
Specifies a list of kernel Kconfig files that contain
hardware options only.
</para></listitem>
<listitem><para>
<filename>non-hardware.kcf</filename>:
Specifies a list of kernel Kconfig files that contain
non-hardware options only.
</para></listitem>
<listitem><para>
<filename>hardware.cfg</filename>:
Specifies a list of kernel <filename>CONFIG_</filename>
options that are hardware, regardless of whether or not
they are within a Kconfig file specified by a hardware
or non-hardware Kconfig file (i.e.
<filename>hardware.kcf</filename> or
<filename>non-hardware.kcf</filename>).
</para></listitem>
<listitem><para>
<filename>non-hardware.cfg</filename>:
Specifies a list of kernel <filename>CONFIG_</filename>
options that are not hardware, regardless of whether or
not they are within a Kconfig file specified by a
hardware or non-hardware Kconfig file (i.e.
<filename>hardware.kcf</filename> or
<filename>non-hardware.kcf</filename>).
</para></listitem>
</itemizedlist>
Here is a specific example using the
<filename>kernel-cache/bsp/mti-malta32/hardware.cfg</filename>:
<literallayout class='monospaced'>
CONFIG_SERIAL_8250
CONFIG_SERIAL_8250_CONSOLE
CONFIG_SERIAL_8250_NR_UARTS
CONFIG_SERIAL_8250_PCI
CONFIG_SERIAL_CORE
CONFIG_SERIAL_CORE_CONSOLE
CONFIG_VGA_ARB
</literallayout>
The kernel configuration audit automatically detects these
files (hence the names must be exactly the ones discussed here),
and uses them as inputs when generating warnings about the
final <filename>.config</filename> file.
</para>
<para>
A user-specified kernel Metadata repository, or recipe space
feature, can use these same files to classify options that are
found within its <filename>.cfg</filename> files as hardware
or non-hardware, to prevent the OpenEmbedded build system from
producing an error or warning when an option is not in the
final <filename>.config</filename> file.
</para>
</section>
</appendix> </appendix>
<!-- <!--
vim: expandtab tw=80 ts=4 vim: expandtab tw=80 ts=4