mirror of
https://git.yoctoproject.org/poky
synced 2026-04-27 03:32:12 +02:00
c387f0c254
(From yocto-docs rev: 0aeb7a94abcef3cb3850c753dd0a243f381e6675) Signed-off-by: Quentin Schulz <foss@0leil.net> Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
625 lines
37 KiB
XML
625 lines
37 KiB
XML
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
|
|
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
|
|
<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
|
|
|
|
<chapter id='test-manual-intro'>
|
|
|
|
<title>The Yocto Project Test Environment Manual</title>
|
|
<section id='test-welcome'>
|
|
<title>Welcome</title>
|
|
|
|
<para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in
|
|
progress. The manual contains information about the testing environment used by the
|
|
Yocto Project to make sure each major and minor release works as intended. All the
|
|
project's testing infrastructure and processes are publicly visible and available so
|
|
that the community can see what testing is being performed, how it's being done and the
|
|
current status of the tests and the project at any given time. It is intended that Other
|
|
organizations can leverage off the process and testing environment used by the Yocto
|
|
Project to create their own automated, production test environment, building upon the
|
|
foundations from the project core. </para>
|
|
|
|
<para> Currently, the Yocto Project Test Environment Manual has no projected release date.
|
|
This manual is a work-in-progress and is being initially loaded with information from
|
|
the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
<emphasis><filename>yocto-autobuilder2</filename>:</emphasis> This <ulink
|
|
url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md"
|
|
><filename>README.md</filename></ulink> is the main README which
|
|
detials how to set up the Yocto Project Autobuilder. The
|
|
<filename>yocto-autobuilder2</filename> repository represents the Yocto
|
|
Project's console UI plugin to Buildbot and the configuration necessary to
|
|
configure Buildbot to perform the testing the project requires. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis><filename>yocto-autobuilder-helper</filename>:</emphasis> This
|
|
<ulink
|
|
url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README"
|
|
><filename>README</filename></ulink> and repository contains Yocto
|
|
Project Autobuilder Helper scripts and configuration. The
|
|
<filename>yocto-autobuilder-helper</filename> repository contains the
|
|
"glue" logic that defines which tests to run and how to run them. As a
|
|
result, it can be used by any Continuous Improvement (CI) system to run
|
|
builds, support getting the correct code revisions, configure builds and
|
|
layers, run builds, and collect results. The code is independent of any CI
|
|
system, which means the code can work Buildbot, Jenkins, or others. This
|
|
repository has a branch per release of the project defining the tests to run
|
|
on a per release basis.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</section>
|
|
|
|
<section id='test-yocto-project-autobuilder-overview'>
|
|
<title>Yocto Project Autobuilder Overview</title>
|
|
|
|
<para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and
|
|
procedures used by the Yocto Project to test released software across supported hardware
|
|
in an automated and regular fashion. Basically, during the development of a Yocto
|
|
Project release, the Autobuilder tests if things work. The Autobuilder builds all test
|
|
targets and runs all the tests. </para>
|
|
|
|
<para>The Yocto Project uses now uses standard upstream <ulink
|
|
url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive
|
|
its integration and testing. Buildbot Nine has a plug-in interface that the Yocto
|
|
Project customizes using code from the <filename>yocto-autobuilder2</filename>
|
|
repository, adding its own console UI plugin. The resulting UI plug-in allows you to
|
|
visualize builds in a way suited to the project's needs.</para>
|
|
|
|
<para>A <filename>helper</filename> layer provides configuration and job management through
|
|
scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The
|
|
<filename>helper</filename> layer contains the bulk of the build configuration
|
|
information and is release-specific, which makes it highly customizable on a per-project
|
|
basis. The layer is CI system-agnostic and contains a number of Helper scripts that can
|
|
generate build configurations from simple JSON files. <note>
|
|
<para>The project uses Buildbot for historical reasons but also because many of the
|
|
project developers have knowledge of python. It is possible to use the outer
|
|
layers from another Continuous Integration (CI) system such as <ulink
|
|
url="https://en.wikipedia.org/wiki/Jenkins_(software)">Jenkins</ulink>
|
|
instead of Buildbot. </para>
|
|
</note>
|
|
</para>
|
|
|
|
<para> The following figure shows the Yocto Project Autobuilder stack with a topology that
|
|
includes a controller and a cluster of workers: <imagedata
|
|
fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center"
|
|
scalefit="1"/>
|
|
</para>
|
|
</section>
|
|
|
|
<section id='test-project-tests'>
|
|
<title>Yocto Project Tests - Types of Testing Overview</title>
|
|
|
|
<para>The Autobuilder tests different elements of the project by using thefollowing types of
|
|
tests: <itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Build Testing:</emphasis> Tests whether specific configurations
|
|
build by varying <ulink url="&YOCTO_DOCS_REF_URL;#var-MACHINE"
|
|
><filename>MACHINE</filename></ulink>, <ulink
|
|
url="&YOCTO_DOCS_REF_URL;#var-DISTRO"
|
|
><filename>DISTRO</filename></ulink>, other configuration options, and
|
|
the specific target images being built (or world). Used to trigger builds of
|
|
all the different test configurations on the Autobuilder. Builds usually
|
|
cover many different targets for different architectures, machines, and
|
|
distributions, as well as different configurations, such as different init
|
|
systems. The Autobuilder tests literally hundreds of configurations and
|
|
targets. <itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Sanity Checks During the Build Process:</emphasis>
|
|
Tests initiated through the <ulink
|
|
url="&YOCTO_DOCS_REF_URL;#ref-classes-insane"
|
|
><filename>insane</filename></ulink> class. These checks
|
|
ensure the output of the builds are correct. For example, does
|
|
the ELF architecture in the generated binaries match the target
|
|
system? ARM binaries would not work in a MIPS system! </para>
|
|
</listitem>
|
|
</itemizedlist></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Build Performance Testing:</emphasis> Tests whether or not
|
|
commonly used steps during builds work efficiently and avoid regressions.
|
|
Tests to time commonly used usage scenarios are run through
|
|
<filename>oe-build-perf-test</filename>. These tests are run on isolated
|
|
machines so that the time measurements of the tests are accurate and no
|
|
other processes interfere with the timing results. The project currently
|
|
tests performance on two different distributions, Fedora and Ubuntu, to
|
|
ensure we have no single point of failure and can ensure the different
|
|
distros work effectively. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>eSDK Testing:</emphasis> Image tests initiated through the
|
|
following command:
|
|
<literallayout class="monospaced">
|
|
$ bitbake <replaceable>image</replaceable> -c testsdkext
|
|
</literallayout>
|
|
The tests utilize the <filename>testsdkext</filename> class and the
|
|
<filename>do_testsdkext</filename> task. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Feature Testing:</emphasis> Various scenario-based tests are run
|
|
through the <ulink url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
|
|
>OpenEmbedded Self-Test</ulink> (oe-selftest). We test oe-selftest on
|
|
each of the main distrubutions we support. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Image Testing:</emphasis> Image tests initiated through the
|
|
following command:
|
|
<literallayout class="monospaced">
|
|
$ bitbake <replaceable>image</replaceable> -c testimage
|
|
</literallayout>
|
|
The tests utilize the <ulink
|
|
url="&YOCTO_DOCS_REF_URL;#ref-classes-testimage*"
|
|
><filename>testimage*</filename></ulink> classes and the <ulink
|
|
url="&YOCTO_DOCS_REF_URL;#ref-tasks-testimage"
|
|
><filename>do_testimage</filename></ulink> task. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Layer Testing:</emphasis> The Autobuilder has the possibility to
|
|
test whether specific layers work with the test of the system. The layers
|
|
tested may be selected by members of the project. Some key community layers
|
|
are also tested periodically.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Package Testing:</emphasis> A Package Test (ptest) runs tests
|
|
against packages built by the OpenEmbedded build system on the target
|
|
machine. See the "<ulink
|
|
url="&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest">Testing Packages
|
|
With ptest</ulink>" section in the Yocto Project Development Tasks
|
|
Manual and the "<ulink url="&YOCTO_WIKI_URL;/wiki/Ptest">Ptest</ulink>" Wiki
|
|
page for more information on Ptest. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>SDK Testing:</emphasis> Image tests initiated through the
|
|
following command:
|
|
<literallayout class="monospaced">
|
|
$ bitbake <replaceable>image</replaceable> -c testsdk
|
|
</literallayout>
|
|
The tests utilize the <ulink url="&YOCTO_DOCS_REF_URL;#ref-classes-testsdk"
|
|
><filename>testsdk</filename></ulink> class and the
|
|
<filename>do_testsdk</filename> task. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Unit Testing:</emphasis> Unit tests on various components of the
|
|
system run through <filename>oe-selftest</filename> and <ulink
|
|
url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
|
|
><filename>bitbake-selftest</filename></ulink>. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
<emphasis>Automatic Upgrade Helper:</emphasis> This target tests whether new
|
|
versions of software are available and whether we can automatically upgrade
|
|
to those new versions. If so, this target emails the maintainers with a
|
|
patch to let them know this is possible.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</section>
|
|
|
|
<section id='test-test-mapping'>
|
|
<title>How Tests Map to Areas of Code</title>
|
|
|
|
<para>
|
|
Tests map into the codebase as follows:
|
|
<itemizedlist>
|
|
<listitem><para>
|
|
<emphasis>bitbake-selftest</emphasis>: </para>
|
|
<para>These tests are self-contained and test BitBake as well as its APIs, which
|
|
include the fetchers. The tests are located in
|
|
<filename>bitbake/lib/*/tests</filename>. </para>
|
|
<para>From within the BitBake repository, run the following:
|
|
<literallayout class="monospaced">
|
|
$ bitbake-selftest
|
|
</literallayout>
|
|
</para>
|
|
<para>To skip tests that access the Internet, use the
|
|
<filename>BB_SKIP_NETTEST</filename> variable when running
|
|
"bitbake-selftest" as follows:
|
|
<literallayout class="monospaced">
|
|
$ BB_SKIP_NETTEST=yes bitbake-selftest
|
|
</literallayout></para>
|
|
<para>The default output is quiet and just prints a summary of what was run. To
|
|
see more information, there is a verbose
|
|
option:<literallayout class="monospaced">
|
|
$ bitbake-selftest -v
|
|
</literallayout></para>
|
|
<para>Use this option when you wish to skip tests that access the network, which
|
|
are mostly necessary to test the fetcher modules. To specify individual test
|
|
modules to run, append the test module name to the "bitbake-selftest"
|
|
command. For example, to specify the tests for the bb.data.module, run:
|
|
<literallayout class="monospaced">
|
|
$ bitbake-selftest bb.test.data.module
|
|
</literallayout>You
|
|
can also specify individual tests by defining the full name and module plus
|
|
the class path of the test, for example:
|
|
<literallayout class="monospaced">
|
|
$ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
|
|
</literallayout></para>
|
|
<para>The tests are based on <ulink
|
|
url="https://docs.python.org/3/library/unittest.html">Python
|
|
unittest</ulink>. </para></listitem>
|
|
<listitem><para>
|
|
<emphasis>oe-selftest</emphasis>: <itemizedlist>
|
|
<listitem>
|
|
<para>These tests use OE to test the workflows, which include
|
|
testing specific features, behaviors of tasks, and API unit
|
|
tests. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>The tests can take advantage of parallelism through the "-j"
|
|
option, which can specify a number of threads to spread the
|
|
tests across. Note that all tests from a given class of tests
|
|
will run in the same thread. To parallelize large numbers of
|
|
tests you can split the class into multiple units.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>The tests are based on Python unittest. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>The code for the tests resides in
|
|
<filename>meta/lib/oeqa/selftest/cases/</filename>. </para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>To run all the tests, enter the following command:
|
|
<literallayout class="monospaced">
|
|
$ oe-selftest -a
|
|
</literallayout>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>To run a specific test, use the following command form where
|
|
<replaceable>testname</replaceable> is the name of the
|
|
specific test:
|
|
<literallayout class="monospaced">
|
|
$ oe-selftest -r <replaceable>testname</replaceable>
|
|
</literallayout>
|
|
For example, the following command would run the tinfoil getVar
|
|
API
|
|
test:<literallayout class="monospaced">
|
|
$ oe-selftest -r tinfoil.TinfoilTests.test_getvar
|
|
</literallayout>It
|
|
is also possible to run a set of tests. For example the
|
|
following command will run all of the tinfoil
|
|
tests:<literallayout class="monospaced">
|
|
$ oe-selftest -r tinfoil
|
|
</literallayout></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para></listitem>
|
|
<listitem><para>
|
|
<emphasis>testimage:</emphasis>
|
|
<itemizedlist>
|
|
<listitem><para>
|
|
These tests build an image, boot it, and run tests
|
|
against the image's content.
|
|
</para></listitem>
|
|
<listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem>
|
|
<listitem><para>
|
|
You need to set the
|
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink>
|
|
variable as follows:
|
|
<literallayout class='monospaced'>
|
|
IMAGE_CLASSES += "testimage"
|
|
</literallayout>
|
|
</para></listitem>
|
|
<listitem><para>
|
|
Run the tests using the following command form:
|
|
<literallayout class='monospaced'>
|
|
$ bitbake <replaceable>image</replaceable> -c testimage
|
|
</literallayout>
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</para></listitem>
|
|
<listitem><para>
|
|
<emphasis>testsdk:</emphasis>
|
|
<itemizedlist>
|
|
<listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem>
|
|
<listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem>
|
|
<listitem><para>Run the test using the following command form:
|
|
<literallayout class="monospaced">
|
|
$ bitbake <replaceable>image</replaceable> -c testsdk
|
|
</literallayout>
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</para></listitem>
|
|
<listitem><para>
|
|
<emphasis>testsdk_ext:</emphasis>
|
|
<itemizedlist>
|
|
<listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem>
|
|
<listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem>
|
|
<listitem><para>To run the tests, use the following command form:
|
|
<literallayout class="monospaced">
|
|
$ bitbake <replaceable>image</replaceable> -c testsdkext
|
|
</literallayout>
|
|
</para></listitem>
|
|
</itemizedlist>
|
|
</para></listitem>
|
|
|
|
|
|
<listitem><para>
|
|
<emphasis>oe-build-perf-test:</emphasis>
|
|
<itemizedlist>
|
|
<listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem>
|
|
<listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem>
|
|
<listitem><para>To run the tests, use the following command form:
|
|
<literallayout class="monospaced">
|
|
$ oe-build-perf-test <replaceable>options</replaceable>
|
|
</literallayout>The
|
|
command takes a number of options, such as where to place the
|
|
test results. The Autobuilder Helper Scripts include the
|
|
<filename>build-perf-test-wrapper</filename> script with
|
|
examples of how to use the oe-build-perf-test from the command
|
|
line.</para>
|
|
<para>Use the <filename>oe-git-archive</filename> command to store
|
|
test results into a Git repository. </para>
|
|
<para>Use the <filename>oe-build-perf-report</filename> command to
|
|
generate text reports and HTML reports with graphs of the
|
|
performance data. For examples, see <link linkend=""
|
|
>http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>
|
|
and <link linkend=""
|
|
>http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>
|
|
<listitem>
|
|
<para>The tests are contained in
|
|
<filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para></listitem>
|
|
|
|
|
|
|
|
|
|
</itemizedlist>
|
|
</para>
|
|
</section>
|
|
|
|
<section id='test-examples'>
|
|
<title>Test Examples</title>
|
|
|
|
<para>This section provides example tests for each of the tests listed in the <link
|
|
linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>
|
|
<para>For oeqa tests, testcases for each area reside in the main test directory at
|
|
<filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>
|
|
<para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>
|
|
directory. </para>
|
|
|
|
<section id='bitbake-selftest-example'>
|
|
<title><filename>bitbake-selftest</filename></title>
|
|
|
|
<para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:
|
|
<literallayout class="monospaced">
|
|
class DataExpansions(unittest.TestCase):
|
|
def setUp(self):
|
|
self.d = bb.data.init()
|
|
self.d["foo"] = "value_of_foo"
|
|
self.d["bar"] = "value_of_bar"
|
|
self.d["value_of_foo"] = "value_of_'value_of_foo'"
|
|
|
|
def test_one_var(self):
|
|
val = self.d.expand("${foo}")
|
|
self.assertEqual(str(val), "value_of_foo")
|
|
</literallayout>
|
|
</para>
|
|
<para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class
|
|
of tests is created, derived from standard python unittest. The class has a common
|
|
<filename>setUp</filename> function which is shared by all the tests in the
|
|
class. A simple test is then added to test that when a variable is expanded, the
|
|
correct value is found.</para>
|
|
<para>Bitbake selftests are straightforward python unittest. Refer to the Python
|
|
unittest documentation for additional information on writing these tests at: <link
|
|
linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>
|
|
</section>
|
|
|
|
<section id='oe-selftest-example'>
|
|
<title><filename>oe-selftest</filename></title>
|
|
|
|
<para>These tests are more complex due to the setup required behind the scenes for full
|
|
builds. Rather than directly using Python's unittest, the code wraps most of the
|
|
standard objects. The tests can be simple, such as testing a command from within the
|
|
OE build environment using the following
|
|
example:<literallayout class="monospaced">
|
|
class BitbakeLayers(OESelftestTestCase):
|
|
def test_bitbakelayers_showcrossdepends(self):
|
|
result = runCmd('bitbake-layers show-cross-depends')
|
|
self.assertTrue('aspell' in result.output, msg = "No dependencies
|
|
were shown. bitbake-layers show-cross-depends output:
|
|
%s"% result.output)
|
|
</literallayout></para>
|
|
<para>This example, taken from
|
|
<filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
|
|
testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>
|
|
class, derived from <filename>unittest.TestCase</filename>, which runs the
|
|
<filename>bitbake-layers</filename> command and checks the output to ensure it
|
|
contains something we know should be here.</para>
|
|
<para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can
|
|
assist with common tasks, including:<itemizedlist>
|
|
<listitem>
|
|
<para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use
|
|
<filename>oeqa.utils.commands.get_bb_var()</filename> or use
|
|
<filename>oeqa.utils.commands.get_bb_vars()</filename> for more than
|
|
one variable</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><emphasis>Running a bitbake invocation for a build:</emphasis> Use
|
|
<filename>oeqa.utils.commands.bitbake()</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><emphasis>Running a command:</emphasis> Use
|
|
<filename>oeqa.utils.commandsrunCmd()</filename></para>
|
|
</listitem>
|
|
</itemizedlist></para>
|
|
<para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for
|
|
launching the <filename>runqemu</filename> command for testing things within a
|
|
running, virtualized image.</para>
|
|
<para>You can run these tests in parallel. Parallelism works per test class, so tests
|
|
within a given test class should always run in the same build, while tests in
|
|
different classes or modules may be split into different builds. There is no data
|
|
store available for these tests since the tests launch the
|
|
<filename>bitbake</filename> command and exist outside of its context. As a
|
|
result, common bitbake library functions (bb.*) are also unavailable.</para>
|
|
</section>
|
|
|
|
<section id='testimage-example'>
|
|
<title><filename>testimage</filename></title>
|
|
|
|
<para>These tests are run once an image is up and running, either on target hardware or
|
|
under QEMU. As a result, they are assumed to be running in a target image
|
|
environment, as opposed to a host build environment. A simple example from
|
|
<filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the
|
|
following:<literallayout class="monospaced">
|
|
class PythonTest(OERuntimeTestCase):
|
|
@OETestDepends(['ssh.SSHTest.test_ssh'])
|
|
@OEHasPackage(['python3-core'])
|
|
def test_python3(self):
|
|
cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,
|
|
jbeyq', 'rot13'))\""
|
|
status, output = self.target.run(cmd)
|
|
msg = 'Exit status was not 0. Output: %s' % output
|
|
self.assertEqual(status, 0, msg=msg)
|
|
</literallayout></para>
|
|
<para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>
|
|
class wraps <filename>unittest.TestCase</filename>. Within the test,
|
|
<filename>self.target</filename> represents the target system, where commands
|
|
can be run on it using the <filename>run()</filename> method. </para>
|
|
<para>To ensure certain test or package dependencies are met, you can use the
|
|
<filename>OETestDepends</filename> and <filename>OEHasPackage</filename>
|
|
decorators. For example, the test in this example would only make sense if
|
|
python3-core is installed in the image.</para>
|
|
</section>
|
|
|
|
<section id='testsdk_ext-example'>
|
|
<title><filename>testsdk_ext</filename></title>
|
|
|
|
<para>These tests are run against built extensible SDKs (eSDKs). The tests can assume
|
|
that the eSDK environment has already been setup. An example from
|
|
<filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
|
|
following:<literallayout class="monospaced">
|
|
class DevtoolTest(OESDKExtTestCase):
|
|
@classmethod
|
|
def setUpClass(cls):
|
|
myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
|
|
cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
|
|
shutil.copytree(myapp_src, cls.myapp_dst)
|
|
subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
|
|
subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
|
|
subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
|
|
|
|
@classmethod
|
|
def tearDownClass(cls):
|
|
shutil.rmtree(cls.myapp_dst)
|
|
def _test_devtool_build(self, directory):
|
|
self._run('devtool add myapp %s' % directory)
|
|
try:
|
|
self._run('devtool build myapp')
|
|
finally:
|
|
self._run('devtool reset myapp')
|
|
def test_devtool_build_make(self):
|
|
self._test_devtool_build(self.myapp_dst)
|
|
</literallayout>In
|
|
this example, the <filename>devtool</filename> command is tested to see whether a
|
|
sample application can be built with the <filename>devtool build</filename> command
|
|
within the eSDK.</para>
|
|
</section>
|
|
|
|
<section id='testsdk-example'>
|
|
<title><filename>testsdk</filename></title>
|
|
|
|
<para>These tests are run against built SDKs. The tests can assume that an SDK has
|
|
already been extracted and its environment file has been sourced. A simple example
|
|
from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
|
|
following:<literallayout class="monospaced">
|
|
class Python3Test(OESDKTestCase):
|
|
def setUp(self):
|
|
if not (self.tc.hasHostPackage("nativesdk-python3-core") or
|
|
self.tc.hasHostPackage("python3-core-native")):
|
|
raise unittest.SkipTest("No python3 package in the SDK")
|
|
|
|
def test_python3(self):
|
|
cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
|
|
output = self._run(cmd)
|
|
self.assertEqual(output, "Hello, world\n")
|
|
</literallayout>In
|
|
this example, if nativesdk-python3-core has been installed into the SDK, the code
|
|
runs the python3 interpreter with a basic command to check it is working correctly.
|
|
The test would only run if python3 is installed in the SDK.</para>
|
|
</section>
|
|
|
|
<section id='oe-build-perf-test-example'>
|
|
<title><filename>oe-build-perf-test</filename></title>
|
|
|
|
<para>The performance tests usually measure how long operations take and the resource
|
|
utilisation as that happens. An example from
|
|
<filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
|
|
following:<literallayout class="monospaced">
|
|
class Test3(BuildPerfTestCase):
|
|
|
|
def test3(self):
|
|
"""Bitbake parsing (bitbake -p)"""
|
|
# Drop all caches and parse
|
|
self.rm_cache()
|
|
oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
|
|
self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
|
|
'bitbake -p (no caches)')
|
|
# Drop tmp/cache
|
|
oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
|
|
self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
|
|
'bitbake -p (no tmp/cache)')
|
|
# Parse with fully cached data
|
|
self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
|
|
'bitbake -p (cached)')
|
|
</literallayout>This
|
|
example shows how three specific parsing timings are measured, with and without
|
|
various caches, to show how BitBake's parsing performance trends over time.</para>
|
|
</section>
|
|
</section>
|
|
<section id='test-writing-considerations'>
|
|
<title>Considerations When Writing Tests</title>
|
|
<para>When writing good tests, there are several things to keep in mind. Since things
|
|
running on the Autobuilder are accessed concurrently by multiple workers, consider the
|
|
following:</para>
|
|
<formalpara>
|
|
<title>Running "cleanall" is not permitted</title>
|
|
<para>This can delete files from DL_DIR which would potentially break other builds
|
|
running in parallel. If this is required, DL_DIR must be set to an isolated
|
|
directory.</para>
|
|
</formalpara>
|
|
<formalpara>
|
|
<title>Running "cleansstate" is not permitted</title>
|
|
<para>This can delete files from SSTATE_DIR which would potentially break other builds
|
|
running in parallel. If this is required, SSTATE_DIR must be set to an isolated
|
|
directory. Alternatively, you can use the "-f" option with the
|
|
<filename>bitbake</filename> command to "taint" tasks by changing the sstate
|
|
checksums to ensure sstate cache items will not be reused.</para>
|
|
</formalpara>
|
|
<formalpara>
|
|
<title>Tests should not change the metadata</title>
|
|
<para>This is particularly true for oe-selftests since these can run in parallel and
|
|
changing metadata leads to changing checksums, which confuses BitBake while running
|
|
in parallel. If this is necessary, copy layers to a temporary location and modify
|
|
them. Some tests need to change metadata, such as the devtool tests. To prevent the
|
|
metadate from changes, set up temporary copies of that data first.</para>
|
|
</formalpara>
|
|
</section>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</chapter>
|
|
<!--
|
|
vim: expandtab tw=80 ts=4
|
|
-->
|