We add the build control functionality to toaster.
* The bldcontrol application gains bbcontroller classes
that know how to manage a localhost build environment.
* The toaster UI now detects it is running under build
environment controller, and update the build controller
database and will shut down the bitbake server once
the build is complete.
* The toaster script can now run in standalone mode,
launching the build controller and the web interface instead
of just monitoring the build, as in the interactive mode.
* A fixture with the default build controller entry for
localhost is provided.
[YOCTO #5490]
[YOCTO #5491]
[YOCTO #5492]
[YOCTO #5493]
[YOCTO #5494]
[YOCTO #5537]
(Bitbake rev: 10988bd77c8c7cefad3b88744bc5d8a7e3c1f4cf)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add support to specify a connection token in the command line
and in the environment variable BBTOKEN.
When a client registers to a bitbake server, that client
will have exclusive access to the server. The client is identified
by a server-supplied token. If a client terminates, we cannot
reconnect to the server as the token is lost.
This patch adds the capability to specify the connection token
in the command line for xmlrpc clients. This allows us
to have bitbake work as an already-authenticated
client with the server and resume sending commands to a server.
(Bitbake rev: db5390940c0afbcdc9fbcf1225761968ae51d4a7)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We create the model classes that store information
about triggering builds, and the available build
environments.
We add a fixture with a default build environment
for build control, using a "build/" directory under
the poky checkout directory.
We enable the bldcontrol in toaster starting script
and in the toaster settings as to allow the actual database
to be kept in sync with the source code.
(Bitbake rev: d4bfe9059f765f11244b97e324c0131f32f8e400)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes toaster startup failing after the preparatory patches
for bldcontrol application have been merged.
I slipped a bldcontrol-specific line into the toaster startup
script and this prevents toaster from starting. We comment
out this line until the bldcontrol application have been merged.
(Bitbake rev: d095eec6fc958d0aeb514cdc206734617fd7c930)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'bitbake-layers show-appends' command can use the built-in
get_file_appends() method which correctly identifies bbappends that
include a '%' wildcard in their filename.
(Bitbake rev: 2732dbae67c1945b668c38cc4cc5678c4aafe3d6)
Signed-off-by: Ash Charles <ashcharles@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We disable bitbake self start to prevent race condition.
ToasterUI will shutdown the server when the build is done
if running in managed mode.
We fix usage of kill server flag in the bitbake binary.
(Bitbake rev: 30159dbda3a40fa596302f91c705cb5f148c97a9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The toaster starter script was designed to be sourced in
a build environment and set up the build recording environment
to be used in an interactive mode.
For the standalone web server mode, we modify the toaster
script to be run directly from the checked-out sources,
without a build environment set up, and run the web server
alone. In the standalone mode, the build environemnts and
all build activities are controled through the web interface.
(Bitbake rev: c1db4ccf27bedcbab2f03e7539fdb11b042c4fb9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add an option to the startup script, named "noweb" that
will start toaster without the embedded web server.
This is useful to start the system for build-only environments,
where the web server code is running on a different machine.
(Bitbake rev: c39838201301b9732581288a93783400bebe6591)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we use the XMLRPC client API to connect to a bitbake server,
we want to receive errors from the API instead of having the
API exiting without warning.
Thus the "bb.fatal" calls have been replaced with "bb.warn" calls,
and we re-raise the original exception for handling by the
original caller.
The bitbake starting script has been modified to properly test
for failures in calling the client API and handle them.
Additional error handling added in the client, as to prevent
fatal crashes.
(Bitbake rev: eb63f08c33644f64752aaae2146a000956ce894a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The configuration reading code should live in the
main bitbake entry point, and the server modules should
be supplied with correct configuration instead of attempting
to parse from configuration files.
This patch moves the endpoint address reading from XMLRPC
to the bitbake main script.
(Bitbake rev: ac5753274ff932e1d6f073ab4dab7bd6fe5355a1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
RRECOMMENDS must be satisfied at build time, and these could cross layer
boundaries, so report these if they exist.
(Bitbake rev: 5569b3dca61e6d962494ca65c7aad09b2eb2ae63)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Overlayed recipes caused this to show false positives because the
overlaying version appeared to be satisfying the overlayed version's
RDEPENDS; but you'd never be building both at the same time.
(Bitbake rev: b94318174fe7f92b9a20eabb0bc4055066cb3d51)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's not particularly useful to show globally inherited classes here
since they do not normally represent a dependency.
(Bitbake rev: d16948bb88fcf44d861985838030be7c08697963)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, show-cross-depends shows dependencies on OE-Core (i.e.
"meta") which is not particularly useful. Add an option to allow you to
hide those. For example, to hide all dependencies on OE-Core:
bitbake-layers show-cross-depends -i meta
Multiple layers can be specified by using commas as separators (no
spaces).
(Bitbake rev: 0e9062e65acbb05c1d9b3a9145eb866c3d562309)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If bitbake-diffsigs is run without arguments it will error:
Traceback (most recent call last):
File "./bin/bitbake-diffsigs", line 121, in <module>
if output:
NameError: name 'output' is not defined
Fix this by moving the check for output into the inner else-clause.
(Bitbake rev: 7d3545a66863ad7183a7650b2af57eee091c45ae)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Iterating through and calling setVar on this number of variables has significant
overhead in the profiling data. By not setting this, we save 3,000 calls
to setVar which gives a noticeable improvement to the speed of task execution.
The BBHASH variables have since been replaced by accessing that data through
the siggen code and going forward, that is the preferred way work with it.
(Bitbake rev: 92526eadd09d19938762290e0492076174367583)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connection may have failed before the event queue has been setup.
Handle this correctly in the exception handler.
(Bitbake rev: db4d80b5c2d32117cdf06333b9627202998b1512)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove trapping SIGCHLD due to a weird interaction
with the bash version used with Ubuntu 14.04 LTS.
(Bitbake rev: 59f2f33440449c586c23dd3a192698a37aaf0595)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the featureset didn't match the defaults, we'd pay the price of two
base configuration parses which showed up adversely in the performance
benchmarks. This also passes the feature set into the cooker creation so
in the common case we don't have to reset the server. This speeds up
both knotty and hob startup. If the featureset doesn't match, the system
will reset as before, this just streamlines the common case.
(Bitbake rev: 1249543c4dbf3edeac033d888497864cfc807a4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no easy way to make this change. We really need parameters for the -S
(dump signatures) handling code. Such a parameter can then be used within the
codebase to handle the signatures in different ways.
For now, "none" is the recommended default and "printdiff" will execute the
new (and more expensive) comparison algorithms.
(Bitbake rev: b9873588696507dfb6aade6821f6f75cb9a19e0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the capability to have the Toaster UI
detect when the Bitbake server exited and cleanly
trigger a clean shutdown of the system through the toaster
starting script.
(Bitbake rev: a9cfa3eacfc99550e1ad3f8bb61b2a0bc9b44332)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The children of the worker should have the default SIGTERM handler,
else they'll try and do cleanup which should only happen in the
parent leading to all kinds of bizarre build failures.
(Bitbake rev: a53c8d1f846d94082aa459996c4114f10970b8ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When passing -t with an incorrect server type, the error message was using a
variable that doesn't exist.
(Bitbake rev: 98b991287df06cd89955c1d0591fce3b5d4403d1)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove the checks for the stop command, since we assume
the user really means it wants to shutdown the system when
he issues the command.
[YOCTO #5376]
(Bitbake rev: 5d6594de8876be8ce4df56b846fee0fc687a6261)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if bitbake-worker handles a SIGTERM, it leaves the child
processes to complete or hang. It shouldn't do this so hook the SIGTERM
event and gracefully shutdown any children.
(Bitbake rev: 551406f3f9ee94de09d2da6e16fea054c6dbfdb7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you are just trying to fix one test at a time, it can be useful to be
able to specify an individual test(s) rather than running them all:
bitbake-selftest bb.tests.codeparser bb.tests.cow
You can even specify the test class or function to run, e.g.:
bitbake-selftest bb.tests.fetch.URITest
bitbake-selftest bb.tests.fetch.FetcherNetworkTest.test_fetch
(Bitbake rev: 4df9c72663e972437131a848e6ddcf3769ae1d2b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster start script lunches multiple process components
of the toaster system.
This patch adds logic into the startup script to
safely fail startup and do proper cleanup on any error
that may happen during system start.
Bitbake needs to return 0 if it will successfully lunches
the server-mode.
(Bitbake rev: f43d284e7ae752049711d8215a6020bee6966d45)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As Toaster advances, database schema alteration
will force users to delete old versions and lose
all data collected.
In order to prevent this, and to allow database
updates to happen without having to delete old data,
we use South to handle migrations for the ORM
application which stores the Toaster data.
[YOCTO #5559]
(Bitbake rev: 6936faed8d94f3a2ab4055049cd27d02d8229003)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to remain up to date with the relevant technologies,
Toaster is updated with this patch to Django 1.5. This also
makes headways to allow usage of emerging
Django-related technologies.
Changes include the startup script Django version check, usage
of TemplateView instead of deprecated simple function to do
redirects, and update to the new form of the _url_ template tag.
Support for Django 1.4.5 is now deprecated.
[YOCTO #5558]
(Bitbake rev: 2d37a1731a2b681bc976f3f391d65abb7745b6f9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is to build on the --status-only option for bitbake and
expose a mechanism where the oe init scripts can easily switch between
memres server and the non-memres server.
In the case of the standard oe init script the following
can shut down the server:
if [ -z "$BBSERVER" ] && [ -f bitbake.lock ] ; then
grep ":" bitbake.lock > /dev/null && BBSERVER=`cat bitbake.lock` bitbake --status-only
if [ $? = 0 ] ; then
echo "Shutting down bitbake memory resident server with bitbake -m"
BBSERVER=`cat bitbake.lock` bitbake -m
fi
fi
A similar function can be used to automatically detect if the server
is already running for the oe memres init script. This new
functionality allows for the memres init script to be started in a new
shell and connect up to an alaready running server without seeing the
error of trying to start the server multiple times.
(Bitbake rev: b1803958de8d7c3c3279841e38604a08dc2316cc)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the ability to dynamically select a port for the
bitbake memory resident server when the BBSERVER port is set to -1.
This allows for running multiple instances of the bitbake memory
resident server on the same system in different build directories.
The client portion of the bitbake instance can also request that the
server automatically start when using the auto port feature. This is
to deal with a bitbake instance that eventually times out and exits or
that has died for some unknown reason.
The new functionality allows for lazy startup of the server after
sourcing the init script for the memory resident functionality.
(Bitbake rev: d6abc07ff385357d312d8435b89e0a9c1f965433)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The --status-only option is purely to check the health of the server.
The idea is to use it from the oe environment script so as to know if
the server needs to be started or not.
(Bitbake rev: 46b26ef5c27bfa9c91680f494f02750305b8098c)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently tasks have no knowledge of which other tasks they depend
upon. This makes it impossible to do at least two things which would be
desirable/interesting:
a) Have the ability to create per recipe sysroots
b) Allow the aclocal files to be present only for the entries in
DEPENDS (directly and indirectly)
By exporting task data through this new variable, tasks can inspect
their dependencies and then take actions based upon this.
(Bitbake rev: 84f1dde717dac22435005b79d03ee0b80a3e8e62)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In recent versions of bitbake, it is not possible to initialise a
BBCooker object without having it load the configuration first. Thus we
should avoid creating the Tinfoil object here in bitbake-layers which
does that internally until we actually need to, so you can run
"bitbake-layers help" and not have to wait several seconds for the
output.
(Bitbake rev: 8f1e280fbbb6432d7bcc1fb4241f402668c6c5ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using the dry run option (-n), bitbake would still try and fire
a specific fakeroot worker. This is doomed to failure since it might
well not have been built.
Add in some checks to prevent the failures.
[YOCTO #5367]
(Bitbake rev: f34d0606f87ce9dacadeb78bac35879b74f10559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to use toaster, now you have to set INHERIT+="toaster buildhistory"
To keep it simple, I've done some changes in order to automate it. When toaster
is started, this line is added to a new file called toaster.conf.
This file is passed to the bitbake server with the --postread parameter.
Based on a patch by Cristiana Voicu <cristiana.voicu@intel.com>
(Bitbake rev: 029e868044989eda370340f8bf4200cfd2670fca)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the terminal where the server was started is closed,
the bitbake server should shutdown. Currently the system
is left in hanging state.
This patch uses "trap" command to make sure the servers
are closed on terminal exit.
[YOCTO #5376]
(Bitbake rev: 5f8b97010f7b465753b6ff6275d18426006ee14b)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the Toaster component to Bitbake.
Toaster is a module designed to record the progress of a
Bitbake build, and data about the resultant artifacts.
It contains a web-based interface and a REST API allowing
post-facto inspection of the build process and artifacts.
Features present in this build:
* toaster start script
* relational data model
* Django boilerplate code
* the REST API
* the Simple UI web interface
This patch has all the development history squashed together.
Code portions contributed by Calin Dragomir <calindragomir@gmail.com>.
(Bitbake rev: d24334a5e83d09b3ab227af485971bb768bf5412)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This utility doesn't take any special arguments, but it's nice if it at
least knows how to deal with no arguments, --help and errors properly.
(Bitbake rev: 0cabdf1d0cde6687bc1372675a0d6242587c87a0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Set up a logger independent of BitBake so we can log errors ourselves
* Handle common errors without printing a traceback
(Bitbake rev: 77b5f5b8dca4deebb06eeb06a8e7f2ccdbfff46f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Use OptionParser to parse the two options to -t rather than trying to
pick them out ourselves.
* Add a description shown with --help output
(Bitbake rev: daab42d19463b4108968fc88b207936e5ac84154)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adjust the task name automatically if the -t option is specified with
a task name that doesn't start with do_ (e.g. "configure" instead of
"do_configure").
(Bitbake rev: d182cbc63745303ef2dc9fa2cbbf5d87a68e0b52)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Getting an error message about --remote-server being set when really
BBSERVER was is confusing, clarify the message.
(Bitbake rev: d7b5938a30a9b0ed83f899a06a88786e8392f8bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When starting a server we don't want to show the debug event queue which
is reserved for when when errors have occurred. This patch copies the UI
code to ensure the user doesn't see confusing output.
(Bitbake rev: a886cda58415085981646fb9a024fa7641f55865)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
