We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
(Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We change the toaster starting script to make the
bitbake server listen on all interfaces on the local machine.
This is needed to be able to receive a controlling client
running on a remote machine.
(Bitbake rev: 137179eafca8d1a5a69b6302f8cc8961be3b45c4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We enhance the checksettings command to try to automatically
detect settings for running on localhost.
The checksettings will look for a nearby poky layer source,
for a nearby build directory, and will try to import settings
from "toasterconf.json" files found in the local layer.
On new configuration, it will also perform updates from the
layer source.
(Bitbake rev: 2aab77dfccb538e2b09829841ea6c464d40cafb1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we get no profiling oversight into either the main bitbake worker
process, or the overall parsing before task execution. This adds in extra
profiling hooks so we can truly capture all parts of bitbake's execution
into the profile data.
To do this we modify the 'magic' value passed to bitbake-worker to trigger
the profiling, before the configuration data is sent over to the worker.
(Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When processes terminate, we really want all of the child processes to
terminate too. This was not happening for worker processes which spawned their
own multiprocessing pools, leading to build hangs. This change ensures any
sigterm gets passed to the whole process group. In local tests, this resolved
some hanging process workloads I could generate. It does rely on signals
being delivered in a timely fashion and there is a multiprocessing bug we have
to work around there.
(Bitbake rev: 96f8ea07ace1379380fab2d78eb592fa40c867d4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SIGNATURE_HANDLER can be defined by the user, so we can't use choices.
(Bitbake rev: 0036b36e333a68c6d30a6bf5c21bfd4fd3e8441b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Toaster adds a number of specific requirements managed
by pip, we document these requirements in order to be able
to run bitbake and toaster under virtualenv. The target here
is to break the dependency on distro packages for specific
python libraries.
In order to start bitbake/Toaster in a distro-independent manner,
we use virtualenv and pip. We add venv to the .gitignore list
to make room for the virtualenv deployment in this directory.
Use this command sequence to setup the virtualenv:
$ virtualenv venv
$ . venv/bin/activate
$ cat toaster-requirements.txt | xargs pip install
bitbake and Toaster will then run normally under virtualenv.
We also fix bugs related to toaster starting in managed mode
through this script:
* toaster will not use bldcontrol application in interactive mode
* a proper message will be displayed if toaster detects a managed
start request in a build environment
(Bitbake rev: b5789c7ea7e6d1a4b67ec60fc0e4233ec543ba3a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add explicit absolute paths for a directory where
the layer sources will be checked out (sourcedir) and
where the build activities will take place.
Adding minimal checking when starting the application in
order to make sure that BuildEnvironment (BE) settings are
usable. This check is ran by the toaster script at startup.
Modify the localhost bbcontroller to use the BE settings
instead of trying to self-configure on checked out sources.
(Bitbake rev: d17500d3f73fdeeef5f11fb3773a65e927be3f02)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake -S had been updated which always reuires an argument, so
update the help info:
- Add the two args in the help message: none and printdiff
- Use type="choice" so that we can get more friendly error messages,
for example:
bitbake: error: option -S: invalid choice: 'printdiffX' (choose from 'none', 'printdiff')
- dump-signatures=DUMP_SIGNATURES -> dump-signatures=SIGNATURE_HANDLER
(Bitbake rev: 021b778fa4685bdde39e1a0f6c7c57632dcf792a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the new project and project page skeletons.
In the process, we add an identifier in the settings.py
to detect whenever Toaster is running in managed mode,
and a context processor to make this value available
to the template processor.
(Bitbake rev: 927a27c68e24cfe13f62ca5f0e60878b04fa4e24)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the build control functionality to toaster.
* The bldcontrol application gains bbcontroller classes
that know how to manage a localhost build environment.
* The toaster UI now detects it is running under build
environment controller, and update the build controller
database and will shut down the bitbake server once
the build is complete.
* The toaster script can now run in standalone mode,
launching the build controller and the web interface instead
of just monitoring the build, as in the interactive mode.
* A fixture with the default build controller entry for
localhost is provided.
[YOCTO #5490]
[YOCTO #5491]
[YOCTO #5492]
[YOCTO #5493]
[YOCTO #5494]
[YOCTO #5537]
(Bitbake rev: 10988bd77c8c7cefad3b88744bc5d8a7e3c1f4cf)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add support to specify a connection token in the command line
and in the environment variable BBTOKEN.
When a client registers to a bitbake server, that client
will have exclusive access to the server. The client is identified
by a server-supplied token. If a client terminates, we cannot
reconnect to the server as the token is lost.
This patch adds the capability to specify the connection token
in the command line for xmlrpc clients. This allows us
to have bitbake work as an already-authenticated
client with the server and resume sending commands to a server.
(Bitbake rev: db5390940c0afbcdc9fbcf1225761968ae51d4a7)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We create the model classes that store information
about triggering builds, and the available build
environments.
We add a fixture with a default build environment
for build control, using a "build/" directory under
the poky checkout directory.
We enable the bldcontrol in toaster starting script
and in the toaster settings as to allow the actual database
to be kept in sync with the source code.
(Bitbake rev: d4bfe9059f765f11244b97e324c0131f32f8e400)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes toaster startup failing after the preparatory patches
for bldcontrol application have been merged.
I slipped a bldcontrol-specific line into the toaster startup
script and this prevents toaster from starting. We comment
out this line until the bldcontrol application have been merged.
(Bitbake rev: d095eec6fc958d0aeb514cdc206734617fd7c930)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'bitbake-layers show-appends' command can use the built-in
get_file_appends() method which correctly identifies bbappends that
include a '%' wildcard in their filename.
(Bitbake rev: 2732dbae67c1945b668c38cc4cc5678c4aafe3d6)
Signed-off-by: Ash Charles <ashcharles@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We disable bitbake self start to prevent race condition.
ToasterUI will shutdown the server when the build is done
if running in managed mode.
We fix usage of kill server flag in the bitbake binary.
(Bitbake rev: 30159dbda3a40fa596302f91c705cb5f148c97a9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The toaster starter script was designed to be sourced in
a build environment and set up the build recording environment
to be used in an interactive mode.
For the standalone web server mode, we modify the toaster
script to be run directly from the checked-out sources,
without a build environment set up, and run the web server
alone. In the standalone mode, the build environemnts and
all build activities are controled through the web interface.
(Bitbake rev: c1db4ccf27bedcbab2f03e7539fdb11b042c4fb9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add an option to the startup script, named "noweb" that
will start toaster without the embedded web server.
This is useful to start the system for build-only environments,
where the web server code is running on a different machine.
(Bitbake rev: c39838201301b9732581288a93783400bebe6591)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we use the XMLRPC client API to connect to a bitbake server,
we want to receive errors from the API instead of having the
API exiting without warning.
Thus the "bb.fatal" calls have been replaced with "bb.warn" calls,
and we re-raise the original exception for handling by the
original caller.
The bitbake starting script has been modified to properly test
for failures in calling the client API and handle them.
Additional error handling added in the client, as to prevent
fatal crashes.
(Bitbake rev: eb63f08c33644f64752aaae2146a000956ce894a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The configuration reading code should live in the
main bitbake entry point, and the server modules should
be supplied with correct configuration instead of attempting
to parse from configuration files.
This patch moves the endpoint address reading from XMLRPC
to the bitbake main script.
(Bitbake rev: ac5753274ff932e1d6f073ab4dab7bd6fe5355a1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
RRECOMMENDS must be satisfied at build time, and these could cross layer
boundaries, so report these if they exist.
(Bitbake rev: 5569b3dca61e6d962494ca65c7aad09b2eb2ae63)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Overlayed recipes caused this to show false positives because the
overlaying version appeared to be satisfying the overlayed version's
RDEPENDS; but you'd never be building both at the same time.
(Bitbake rev: b94318174fe7f92b9a20eabb0bc4055066cb3d51)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's not particularly useful to show globally inherited classes here
since they do not normally represent a dependency.
(Bitbake rev: d16948bb88fcf44d861985838030be7c08697963)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, show-cross-depends shows dependencies on OE-Core (i.e.
"meta") which is not particularly useful. Add an option to allow you to
hide those. For example, to hide all dependencies on OE-Core:
bitbake-layers show-cross-depends -i meta
Multiple layers can be specified by using commas as separators (no
spaces).
(Bitbake rev: 0e9062e65acbb05c1d9b3a9145eb866c3d562309)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If bitbake-diffsigs is run without arguments it will error:
Traceback (most recent call last):
File "./bin/bitbake-diffsigs", line 121, in <module>
if output:
NameError: name 'output' is not defined
Fix this by moving the check for output into the inner else-clause.
(Bitbake rev: 7d3545a66863ad7183a7650b2af57eee091c45ae)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Iterating through and calling setVar on this number of variables has significant
overhead in the profiling data. By not setting this, we save 3,000 calls
to setVar which gives a noticeable improvement to the speed of task execution.
The BBHASH variables have since been replaced by accessing that data through
the siggen code and going forward, that is the preferred way work with it.
(Bitbake rev: 92526eadd09d19938762290e0492076174367583)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connection may have failed before the event queue has been setup.
Handle this correctly in the exception handler.
(Bitbake rev: db4d80b5c2d32117cdf06333b9627202998b1512)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove trapping SIGCHLD due to a weird interaction
with the bash version used with Ubuntu 14.04 LTS.
(Bitbake rev: 59f2f33440449c586c23dd3a192698a37aaf0595)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the featureset didn't match the defaults, we'd pay the price of two
base configuration parses which showed up adversely in the performance
benchmarks. This also passes the feature set into the cooker creation so
in the common case we don't have to reset the server. This speeds up
both knotty and hob startup. If the featureset doesn't match, the system
will reset as before, this just streamlines the common case.
(Bitbake rev: 1249543c4dbf3edeac033d888497864cfc807a4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no easy way to make this change. We really need parameters for the -S
(dump signatures) handling code. Such a parameter can then be used within the
codebase to handle the signatures in different ways.
For now, "none" is the recommended default and "printdiff" will execute the
new (and more expensive) comparison algorithms.
(Bitbake rev: b9873588696507dfb6aade6821f6f75cb9a19e0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the capability to have the Toaster UI
detect when the Bitbake server exited and cleanly
trigger a clean shutdown of the system through the toaster
starting script.
(Bitbake rev: a9cfa3eacfc99550e1ad3f8bb61b2a0bc9b44332)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The children of the worker should have the default SIGTERM handler,
else they'll try and do cleanup which should only happen in the
parent leading to all kinds of bizarre build failures.
(Bitbake rev: a53c8d1f846d94082aa459996c4114f10970b8ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When passing -t with an incorrect server type, the error message was using a
variable that doesn't exist.
(Bitbake rev: 98b991287df06cd89955c1d0591fce3b5d4403d1)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove the checks for the stop command, since we assume
the user really means it wants to shutdown the system when
he issues the command.
[YOCTO #5376]
(Bitbake rev: 5d6594de8876be8ce4df56b846fee0fc687a6261)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if bitbake-worker handles a SIGTERM, it leaves the child
processes to complete or hang. It shouldn't do this so hook the SIGTERM
event and gracefully shutdown any children.
(Bitbake rev: 551406f3f9ee94de09d2da6e16fea054c6dbfdb7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you are just trying to fix one test at a time, it can be useful to be
able to specify an individual test(s) rather than running them all:
bitbake-selftest bb.tests.codeparser bb.tests.cow
You can even specify the test class or function to run, e.g.:
bitbake-selftest bb.tests.fetch.URITest
bitbake-selftest bb.tests.fetch.FetcherNetworkTest.test_fetch
(Bitbake rev: 4df9c72663e972437131a848e6ddcf3769ae1d2b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster start script lunches multiple process components
of the toaster system.
This patch adds logic into the startup script to
safely fail startup and do proper cleanup on any error
that may happen during system start.
Bitbake needs to return 0 if it will successfully lunches
the server-mode.
(Bitbake rev: f43d284e7ae752049711d8215a6020bee6966d45)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As Toaster advances, database schema alteration
will force users to delete old versions and lose
all data collected.
In order to prevent this, and to allow database
updates to happen without having to delete old data,
we use South to handle migrations for the ORM
application which stores the Toaster data.
[YOCTO #5559]
(Bitbake rev: 6936faed8d94f3a2ab4055049cd27d02d8229003)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to remain up to date with the relevant technologies,
Toaster is updated with this patch to Django 1.5. This also
makes headways to allow usage of emerging
Django-related technologies.
Changes include the startup script Django version check, usage
of TemplateView instead of deprecated simple function to do
redirects, and update to the new form of the _url_ template tag.
Support for Django 1.4.5 is now deprecated.
[YOCTO #5558]
(Bitbake rev: 2d37a1731a2b681bc976f3f391d65abb7745b6f9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is to build on the --status-only option for bitbake and
expose a mechanism where the oe init scripts can easily switch between
memres server and the non-memres server.
In the case of the standard oe init script the following
can shut down the server:
if [ -z "$BBSERVER" ] && [ -f bitbake.lock ] ; then
grep ":" bitbake.lock > /dev/null && BBSERVER=`cat bitbake.lock` bitbake --status-only
if [ $? = 0 ] ; then
echo "Shutting down bitbake memory resident server with bitbake -m"
BBSERVER=`cat bitbake.lock` bitbake -m
fi
fi
A similar function can be used to automatically detect if the server
is already running for the oe memres init script. This new
functionality allows for the memres init script to be started in a new
shell and connect up to an alaready running server without seeing the
error of trying to start the server multiple times.
(Bitbake rev: b1803958de8d7c3c3279841e38604a08dc2316cc)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the ability to dynamically select a port for the
bitbake memory resident server when the BBSERVER port is set to -1.
This allows for running multiple instances of the bitbake memory
resident server on the same system in different build directories.
The client portion of the bitbake instance can also request that the
server automatically start when using the auto port feature. This is
to deal with a bitbake instance that eventually times out and exits or
that has died for some unknown reason.
The new functionality allows for lazy startup of the server after
sourcing the init script for the memory resident functionality.
(Bitbake rev: d6abc07ff385357d312d8435b89e0a9c1f965433)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The --status-only option is purely to check the health of the server.
The idea is to use it from the oe environment script so as to know if
the server needs to be started or not.
(Bitbake rev: 46b26ef5c27bfa9c91680f494f02750305b8098c)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently tasks have no knowledge of which other tasks they depend
upon. This makes it impossible to do at least two things which would be
desirable/interesting:
a) Have the ability to create per recipe sysroots
b) Allow the aclocal files to be present only for the entries in
DEPENDS (directly and indirectly)
By exporting task data through this new variable, tasks can inspect
their dependencies and then take actions based upon this.
(Bitbake rev: 84f1dde717dac22435005b79d03ee0b80a3e8e62)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
