bitbake: cooker, toaster: variable definition tracking

In order to track the file where a configuration
variable was defined, this patch bring these changes:

* a new feature is defined in CookerFeatures, named
BASEDATASTORE_TRACKING. When a UI requests BASEDATASTORE_TRACKING,
the base variable definition are tracked when configuration
is parsed.

* getAllKeysWithFlags now includes variable history in the
data dump

* toaster_ui.py will record the operation, file path
and line number where the variable was changes

* toaster Simple UI will display the file path
and line number for Configuration page

There is a change in the models to accomodate the recording
of variable change history.

[YOCTO #5227]

(Bitbake rev: 78e58fed82f2a71f052485de0052d7b9cca53ffd)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

README.hardware

@@ -75,14 +75,32 @@ Intel x86 based PCs and devices (genericx86)
 
 The genericx86 MACHINE is tested on the following platforms:
 
-  o Asus EeePC 901
-  o Acer Aspire One
-  o Toshiba NB305
-  o Intel Embedded Development Board 1-N450 (Black Sand)
+Intel Xeon/Core i-Series:
+  + Intel Romley Server: Sandy Bridge Xeon processor, C600 PCH (Patsburg), (Canoe Pass CRB)
+  + Intel Romley Server: Ivy Bridge Xeon processor, C600 PCH (Patsburg), (Intel SDP S2R3)
+  + Intel Crystal Forest Server: Sandy Bridge Xeon processor, DH89xx PCH (Cave Creek), (Stargo CRB)
+  + Intel Chief River Mobile: Ivy Bridge Mobile processor, QM77 PCH (Panther Point-M), (Emerald Lake II CRB, Sabino Canyon CRB)
+  + Intel Huron River Mobile: Sandy Bridge processor, QM67 PCH (Cougar Point), (Emerald Lake CRB, EVOC EC7-1817LNAR board)
+  + Intel Calpella Platform: Core i7 processor, QM57 PCH (Ibex Peak-M), (Red Fort CRB, Emerson MATXM CORE-411-B)
+  + Intel Nehalem/Westmere-EP Server: Xeon 56xx/55xx processors, 5520 chipset, ICH10R IOH (82801), (Hanlan Creek CRB)
+  + Intel Nehalem Workstation: Xeon 56xx/55xx processors, System SC5650SCWS (Greencity CRB)
+  + Intel Picket Post Server: Xeon 56xx/55xx processors (Jasper Forest), 3420 chipset (Ibex Peak), (Osage CRB)
+  + Intel Storage Platform: Sandy Bridge Xeon processor, C600 PCH (Patsburg), (Oak Creek Canyon CRB)
+  + Intel Shark Bay Client Platform: Haswell processor, LynxPoint PCH, (Walnut Canyon CRB, Lava Canyon CRB, Basking Ridge CRB, Flathead Creek CRB)
+  + Intel Shark Bay Ultrabook Platform: Haswell ULT processor, Lynx Point-LP PCH, (WhiteTip Mountain 1 CRB)
 
-and is likely to work on many unlisted Atom based devices. The MACHINE type
-supports ethernet, wifi, sound, and i915 graphics by default in addition to
-common PC input devices, busses, and so on.
+Intel Atom platforms:
+  + Intel embedded Menlow: Intel Atom Z510/530 CPU, System Controller Hub US15W (Portwell NANO-8044)
+  + Intel Luna Pier: Intel Atom N4xx/D5xx series CPU (aka: Pineview-D & -M), 82801HM I/O Hub (ICH8M), (Advantech AIMB-212, Moon Creek CRB)
+  + Intel Queens Bay platform: Intel Atom E6xx CPU (aka: Tunnel Creek), Topcliff EG20T I/O Hub (Emerson NITX-315, Crown Bay CRB, Minnow Board)
+  + Intel Fish River Island platform: Intel Atom E6xx CPU (aka: Tunnel Creek), Topcliff EG20T I/O Hub (Kontron KM2M806)
+  + Intel Cedar Trail platform: Intel Atom N2000 & D2000 series CPU (aka: Cedarview), NM10 Express Chipset (Norco kit BIS-6630, Cedar Rock CRB)
+
+and is likely to work on many unlisted Atom/Core/Xeon based devices. The MACHINE
+type supports ethernet, wifi, sound, and Intel/vesa graphics by default in
+addition to common PC input devices, busses, and so on.  Note that it does not
+included the binary-only graphic drivers used on some Atom platforms, for
+accelerated graphics on these machines please refer to meta-intel.
 
 Depending on the device, it can boot from a traditional hard-disk, a USB device,
 or over the network. Writing generated images to physical media is

bitbake/LICENSE

@@ -0,0 +1,10 @@
+BitBake is licensed under the GNU General Public License version 2.0. See COPYING for further details.
+
+The following external components are distributed with this software:
+
+* The Toaster Simple UI application is based upon the Django project template, the files of which are covered by the BSD license and are copyright (c) Django Software
+Foundation and individual contributors.
+
+* Twitter Bootstrap (including Glyphicons), redistributed under the Apache License 2.0.
+
+* jQuery is redistributed under the MIT license.

bitbake/bin/bitbake

@@ -41,7 +41,7 @@ from bb import ui
 from bb import server
 from bb import cookerdata
 
-__version__ = "1.19.1"
+__version__ = "1.20.0"
 logger = logging.getLogger("BitBake")
 
 # Python multiprocessing requires /dev/shm
@@ -75,18 +75,6 @@ def get_ui(config):
         sys.exit("FATAL: Invalid user interface '%s' specified.\n"
                  "Valid interfaces: depexp, goggle, ncurses, hob, knotty [default]." % interface)
 
-def gather_extra_cache_data():
-    extra = []
-    interfaces = ['depexp', 'goggle', 'ncurses', 'hob', 'knotty']
-    for i in interfaces:
-        try:
-            ui = __import__("bb.ui." + i, fromlist = [i])
-            if hasattr(ui, "extraCaches"):
-                extra = extra + ui.extraCaches
-            del ui
-        except:
-            pass
-    return extra
 
 # Display bitbake/OE warnings via the BitBake.Warnings logger, ignoring others"""
 warnlog = logging.getLogger("BitBake.Warnings")
@@ -112,59 +100,58 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
     def parseCommandLine(self):
         parser = optparse.OptionParser(
             version = "BitBake Build Tool Core version %s, %%prog version %s" % (bb.__version__, __version__),
-            usage = """%prog [options] [package ...]
+            usage = """%prog [options] [recipename/target ...]
 
-    Executes the specified task (default is 'build') for a given set of BitBake files.
-    It expects that BBFILES is defined, which is a space separated list of files to
-    be executed.  BBFILES does support wildcards.
-    Default BBFILES are the .bb files in the current directory.""")
+    Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
+    It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
+    will provide the layer, BBFILES and other configuration information.""")
 
-        parser.add_option("-b", "--buildfile", help = "execute the task against this .bb file, rather than a package from BBFILES. Does not handle any dependencies.",
+        parser.add_option("-b", "--buildfile", help = "Execute tasks from a specific .bb recipe directly. WARNING: Does not handle any dependencies from other recipes.",
                    action = "store", dest = "buildfile", default = None)
 
-        parser.add_option("-k", "--continue", help = "continue as much as possible after an error. While the target that failed, and those that depend on it, cannot be remade, the other dependencies of these targets can be processed all the same.",
+        parser.add_option("-k", "--continue", help = "Continue as much as possible after an error. While the target that failed and anything depending on it cannot be built, as much as possible will be built before stopping.",
                    action = "store_false", dest = "abort", default = True)
 
-        parser.add_option("-a", "--tryaltconfigs", help = "continue with builds by trying to use alternative providers where possible.",
+        parser.add_option("-a", "--tryaltconfigs", help = "Continue with builds by trying to use alternative providers where possible.",
                    action = "store_true", dest = "tryaltconfigs", default = False)
 
-        parser.add_option("-f", "--force", help = "force run of specified cmd, regardless of stamp status",
+        parser.add_option("-f", "--force", help = "Force the specified targets/task to run (invalidating any existing stamp file).",
                    action = "store_true", dest = "force", default = False)
 
-        parser.add_option("-c", "--cmd", help = "Specify task to execute. Note that this only executes the specified task for the providee and the packages it depends on, i.e. 'compile' does not implicitly call stage for the dependencies (IOW: use only if you know what you are doing). Depending on the base.bbclass a listtasks tasks is defined and will show available tasks",
+        parser.add_option("-c", "--cmd", help = "Specify the task to execute. The exact options available depend on the metadata. Some examples might be 'compile' or 'populate_sysroot' or 'listtasks' may give a list of the tasks available.",
                    action = "store", dest = "cmd")
 
-        parser.add_option("-C", "--clear-stamp", help = "Invalidate the stamp for the specified cmd such as 'compile' and run the default task for the specified target(s)",
+        parser.add_option("-C", "--clear-stamp", help = "Invalidate the stamp for the specified task such as 'compile' and then run the default task for the specified target(s).",
                     action = "store", dest = "invalidate_stamp")
 
-        parser.add_option("-r", "--read", help = "read the specified file before bitbake.conf",
+        parser.add_option("-r", "--read", help = "Read the specified file before bitbake.conf.",
                    action = "append", dest = "prefile", default = [])
 
-        parser.add_option("-R", "--postread", help = "read the specified file after bitbake.conf",
+        parser.add_option("-R", "--postread", help = "Read the specified file after bitbake.conf.",
                           action = "append", dest = "postfile", default = [])
 
-        parser.add_option("-v", "--verbose", help = "output more chit-chat to the terminal",
+        parser.add_option("-v", "--verbose", help = "Output more log message data to the terminal.",
                    action = "store_true", dest = "verbose", default = False)
 
         parser.add_option("-D", "--debug", help = "Increase the debug level. You can specify this more than once.",
                    action = "count", dest="debug", default = 0)
 
-        parser.add_option("-n", "--dry-run", help = "don't execute, just go through the motions",
+        parser.add_option("-n", "--dry-run", help = "Don't execute, just go through the motions.",
                    action = "store_true", dest = "dry_run", default = False)
 
-        parser.add_option("-S", "--dump-signatures", help = "don't execute, just dump out the signature construction information",
+        parser.add_option("-S", "--dump-signatures", help = "Don't execute, just dump out the signature construction information.",
                    action = "store_true", dest = "dump_signatures", default = False)
 
-        parser.add_option("-p", "--parse-only", help = "quit after parsing the BB files (developers only)",
+        parser.add_option("-p", "--parse-only", help = "Quit after parsing the BB recipes.",
                    action = "store_true", dest = "parse_only", default = False)
 
-        parser.add_option("-s", "--show-versions", help = "show current and preferred versions of all recipes",
+        parser.add_option("-s", "--show-versions", help = "Show current and preferred versions of all recipes.",
                    action = "store_true", dest = "show_versions", default = False)
 
-        parser.add_option("-e", "--environment", help = "show the global or per-package environment (this is what used to be bbread)",
+        parser.add_option("-e", "--environment", help = "Show the global or per-package environment complete with information about where variables were set/changed.",
                    action = "store_true", dest = "show_environment", default = False)
 
-        parser.add_option("-g", "--graphviz", help = "emit the dependency trees of the specified packages in the dot syntax, and the pn-buildlist to show the build list",
+        parser.add_option("-g", "--graphviz", help = "Save dependency tree information for the specified targets in the dot syntax.",
                     action = "store_true", dest = "dot_graph", default = False)
 
         parser.add_option("-I", "--ignore-deps", help = """Assume these dependencies don't exist and are already provided (equivalent to ASSUME_PROVIDED). Useful to make dependency graphs more appealing""",
@@ -173,34 +160,34 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
         parser.add_option("-l", "--log-domains", help = """Show debug logging for the specified logging domains""",
                     action = "append", dest = "debug_domains", default = [])
 
-        parser.add_option("-P", "--profile", help = "profile the command and print a report",
+        parser.add_option("-P", "--profile", help = "Profile the command and save reports.",
                    action = "store_true", dest = "profile", default = False)
 
-        parser.add_option("-u", "--ui", help = "userinterface to use",
+        parser.add_option("-u", "--ui", help = "The user interface to use (e.g. knotty, hob, depexp).",
                    action = "store", dest = "ui")
 
-        parser.add_option("-t", "--servertype", help = "Choose which server to use, process or xmlrpc",
+        parser.add_option("-t", "--servertype", help = "Choose which server to use, process or xmlrpc.",
                    action = "store", dest = "servertype")
 
-        parser.add_option("", "--revisions-changed", help = "Set the exit code depending on whether upstream floating revisions have changed or not",
+        parser.add_option("", "--revisions-changed", help = "Set the exit code depending on whether upstream floating revisions have changed or not.",
                    action = "store_true", dest = "revisions_changed", default = False)
 
-        parser.add_option("", "--server-only", help = "Run bitbake without UI,  the frontend can connect with bitbake server itself",
+        parser.add_option("", "--server-only", help = "Run bitbake without a UI, only starting a server (cooker) process.",
                    action = "store_true", dest = "server_only", default = False)
 
-        parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to",
+        parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to.",
                    action = "store", dest = "bind", default = False)
 
-        parser.add_option("", "--no-setscene", help = "Do not run any setscene tasks, forces builds",
+        parser.add_option("", "--no-setscene", help = "Do not run any setscene tasks. sstate will be ignored and everything needed, built.",
                    action = "store_true", dest = "nosetscene", default = False)
 
-        parser.add_option("", "--remote-server", help = "Connect to the specified server",
+        parser.add_option("", "--remote-server", help = "Connect to the specified server.",
                    action = "store", dest = "remote_server", default = False)
 
-        parser.add_option("-m", "--kill-server", help = "Terminate the remote server",
+        parser.add_option("-m", "--kill-server", help = "Terminate the remote server.",
                     action = "store_true", dest = "kill_server", default = False)
 
-        parser.add_option("", "--observe-only", help = "Connect to a server as an observing-only client",
+        parser.add_option("", "--observe-only", help = "Connect to a server as an observing-only client.",
                    action = "store_true", dest = "observe_only", default = False)
 
         options, targets = parser.parse_args(sys.argv)
@@ -275,8 +262,8 @@ def main():
         if not configParams.bind:
             sys.exit("FATAL: The '--server-only' option requires a name/address to bind to with the -B option.\n")
         if configParams.remote_server:
-            sys.exit("FATAL: The '--server-only' option conflicts with the '--remote-server' option. %s\n" % 
-                ("Please check your BBSERVER environment" if "BBSERVER" in os.environ else "" ))
+            sys.exit("FATAL: The '--server-only' option conflicts with %s.\n" % 
+                ("the BBSERVER environment variable" if "BBSERVER" in os.environ else "the '--remote-server' option" ))
 
     if configParams.bind and configParams.servertype != "xmlrpc":
         sys.exit("FATAL: If '-B' or '--bind' is defined, we must set the servertype as 'xmlrpc'.\n")
@@ -302,30 +289,28 @@ def main():
     # Clear away any spurious environment variables while we stoke up the cooker
     cleanedvars = bb.utils.clean_environment()
 
-    # Collect all the caches we need
-    if configParams.server_only:
-        configuration.extra_caches = gather_extra_cache_data()
-    else:
-        configuration.extra_caches = getattr(ui_module, "extraCaches", [])
-
     if not configParams.remote_server:
         # we start a server with a given configuration
         server = start_server(servermodule, configParams, configuration)
+        bb.event.ui_queue = []
     else:
-        # we start a stub server that is actually a XMLRPClient to
+        # we start a stub server that is actually a XMLRPClient that connects to a real server
         server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
 
-    logger.removeHandler(handler)
-
     if not configParams.server_only:
+        # Collect the feature set for the UI
+        featureset = getattr(ui_module, "featureSet", [])
+
         # Setup a connection to the server (cooker)
-        server_connection = server.establishConnection()
+        server_connection = server.establishConnection(featureset)
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:
             os.environ[k] = cleanedvars[k]
 
+        logger.removeHandler(handler)
+
         try:
             return ui_module.main(server_connection.connection, server_connection.events, configParams)
         finally:
@@ -339,6 +324,8 @@ def main():
 if __name__ == "__main__":
     try:
         ret = main()
+    except bb.BBHandledException:
+        ret = 1
     except Exception:
         ret = 1
         import traceback

bitbake/bin/bitbake-diffsigs

@@ -3,7 +3,7 @@
 # bitbake-diffsigs
 # BitBake task signature data comparison utility
 #
-# Copyright (C) 2012 Intel Corporation
+# Copyright (C) 2012-2013 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -30,7 +30,18 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), '
 import bb.tinfoil
 import bb.siggen
 
-logger = logging.getLogger('BitBake')
+def logger_create(name, output=sys.stderr):
+    logger = logging.getLogger(name)
+    console = logging.StreamHandler(output)
+    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
+    if output.isatty():
+        format.enable_color()
+    console.setFormatter(format)
+    logger.addHandler(console)
+    logger.setLevel(logging.INFO)
+    return logger
+
+logger = logger_create('bitbake-diffsigs')
 
 def find_compare_task(bbhandler, pn, taskname):
     """ Find the most recent signature files for the specified PN/task and compare them """
@@ -39,6 +50,9 @@ def find_compare_task(bbhandler, pn, taskname):
         logger.error('Metadata does not support finding signature data files')
         sys.exit(1)
 
+    if not taskname.startswith('do_'):
+        taskname = 'do_%s' % taskname
+
     filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data)
     latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:]
     if not latestfiles:
@@ -71,6 +85,7 @@ def find_compare_task(bbhandler, pn, taskname):
 
 
 parser = optparse.OptionParser(
+    description = "Compares siginfo/sigdata files written out by BitBake",
     usage = """
   %prog -t recipename taskname
   %prog sigdatafile1 sigdatafile2
@@ -78,25 +93,30 @@ parser = optparse.OptionParser(
 
 parser.add_option("-t", "--task",
         help = "find the signature data files for last two runs of the specified task and compare them",
-        action="store_true", dest="taskmode")
+        action="store", dest="taskargs", nargs=2, metavar='recipename taskname')
 
 options, args = parser.parse_args(sys.argv)
 
-if len(args) == 1:
-    parser.print_help()
+if options.taskargs:
+    tinfoil = bb.tinfoil.Tinfoil()
+    tinfoil.prepare(config_only = True)
+    find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1])
 else:
-    if options.taskmode:
-        tinfoil = bb.tinfoil.Tinfoil()
-        if len(args) < 3:
-            logger.error("Please specify a recipe and task name")
-            sys.exit(1)
-        tinfoil.prepare(config_only = True)
-        find_compare_task(tinfoil, args[1], args[2])
+    if len(args) == 1:
+        parser.print_help()
     else:
-        if len(args) == 2:
-            output = bb.siggen.dump_sigfile(sys.argv[1])
-        else:
-            output = bb.siggen.compare_sigfiles(sys.argv[1], sys.argv[2])
+        import cPickle
+        try:
+            if len(args) == 2:
+                output = bb.siggen.dump_sigfile(sys.argv[1])
+            else:
+                output = bb.siggen.compare_sigfiles(sys.argv[1], sys.argv[2])
+        except IOError as e:
+            logger.error(str(e))
+            sys.exit(1)
+        except cPickle.UnpicklingError, EOFError:
+            logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files')
+            sys.exit(1)
 
-        if output:
-            print '\n'.join(output)
+    if output:
+        print '\n'.join(output)

bitbake/bin/bitbake-dumpsig

@@ -1,11 +1,65 @@
 #!/usr/bin/env python
+
+# bitbake-dumpsig
+# BitBake task signature dump utility
+#
+# Copyright (C) 2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
 import os
 import sys
 import warnings
+import optparse
+import logging
+
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
 
 import bb.siggen
 
-output = bb.siggen.dump_sigfile(sys.argv[1])
-if output:
-    print '\n'.join(output)
+def logger_create(name, output=sys.stderr):
+    logger = logging.getLogger(name)
+    console = logging.StreamHandler(output)
+    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
+    if output.isatty():
+        format.enable_color()
+    console.setFormatter(format)
+    logger.addHandler(console)
+    logger.setLevel(logging.INFO)
+    return logger
+
+logger = logger_create('bitbake-dumpsig')
+
+parser = optparse.OptionParser(
+    description = "Dumps siginfo/sigdata files written out by BitBake",
+    usage = """
+  %prog sigdatafile""")
+
+options, args = parser.parse_args(sys.argv)
+
+if len(args) == 1:
+    parser.print_help()
+else:
+    import cPickle
+    try:
+        output = bb.siggen.dump_sigfile(args[1])
+    except IOError as e:
+        logger.error(str(e))
+        sys.exit(1)
+    except cPickle.UnpicklingError, EOFError:
+        logger.error('Invalid signature data - ensure you are specifying a sigdata/siginfo file')
+        sys.exit(1)
+
+    if output:
+        print '\n'.join(output)

bitbake/bin/bitbake-worker

@@ -148,6 +148,8 @@ def fork_off_task(cfg, data, workerdata, fn, task, taskname, appends, quieterror
 
             data.setVar("BB_WORKERCONTEXT", "1")
             data.setVar("BUILDNAME", workerdata["buildname"])
+            data.setVar("DATE", workerdata["date"])
+            data.setVar("TIME", workerdata["time"])
             bb.parse.siggen.set_taskdata(workerdata["hashes"], workerdata["hash_deps"], workerdata["sigchecksums"])
             ret = 0
             try:

bitbake/bin/toaster

@@ -0,0 +1,164 @@
+#!/bin/bash
+# (c) 2013 Intel Corp.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+
+# This script enables toaster event logging and
+# starts bitbake resident server
+# use as:  source toaster [start|stop]
+
+# Helper function to kill a background toaster development server
+
+function webserverKillAll()
+{
+	local pidfile
+	for pidfile in ${BUILDDIR}/.toastermain.pid; do
+		if [ -f ${pidfile} ]; then
+		while kill -0 $(< ${pidfile}) 2>/dev/null; do
+			kill -SIGTERM -$(< ${pidfile}) 2>/dev/null
+			sleep 1;
+		done;
+		rm  ${pidfile}
+		fi
+	done
+}
+
+function webserverStartAll()
+{
+        retval=0
+        python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1
+        if [ $retval -eq 1 ]; then
+                echo "Failed db sync, stopping system start" 1>&2
+        else
+            python $BBBASEDIR/lib/toaster/manage.py runserver 0.0.0.0:8000 </dev/null >${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
+        fi
+        return $retval
+}
+
+# Helper functions to add a special configuration file
+
+function addtoConfiguration()
+{
+        echo "#Created by toaster start script" > ${BUILDDIR}/conf/$2
+        echo $1 >> ${BUILDDIR}/conf/$2
+}
+
+# define the stop command
+function stop_system()
+{
+    if [ -f ${BUILDDIR}/.toasterui.pid ]; then
+        kill $(< ${BUILDDIR}/.toasterui.pid )
+        rm ${BUILDDIR}/.toasterui.pid
+    fi
+    BBSERVER=localhost:8200 bitbake -m
+    unset BBSERVER
+    webserverKillAll
+    # force stop any misbehaving bitbake server
+    lsof bitbake.lock | awk '{print $2}' | grep "[0-9]\+" | xargs -n1 -r kill
+}
+
+# We make sure we're running in the current shell and in a good environment
+
+if [ -z "$ZSH_NAME" ] && [ `basename \"$0\"` = `basename \"$BASH_SOURCE\"` ]; then
+    echo "Error: This script needs to be sourced. Please run as 'source toaster [start|stop]'" 1>&2;
+    exit 1
+fi
+
+if [ -z "$BUILDDIR" ] || [ -z `which bitbake` ]; then
+    echo "Error: Build environment is not setup or bitbake is not in path." 1>&2;
+    return 2
+fi
+
+BBBASEDIR=`dirname ${BASH_SOURCE}`/..
+
+
+# Verify prerequisites
+
+if ! echo "import django; print (1,4,5) == django.VERSION[0:3]" | python 2>/dev/null | grep True >/dev/null; then
+    echo -e "This program needs Django 1.4.5. Please install with\n\nsudo pip install django==1.4.5"
+    return 2
+fi
+
+
+
+# Determine the action. If specified by arguments, fine, if not, toggle it
+if [ "x$1" == "xstart" ] || [ "x$1" == "xstop" ]; then
+    CMD="$1"
+else
+    if [ -z "$BBSERVER" ]; then
+        CMD="start"
+    else
+        CMD="stop"
+    fi;
+fi
+
+NOTOASTERUI=0
+if [ "x$2" == "xnoui" ]; then
+    NOTOASTERUI=1
+fi
+
+echo "The system will $CMD."
+
+# Make sure it's safe to run by checking bitbake lock
+
+lock=1
+if [ -e $BUILDDIR/bitbake.lock ]; then
+    (flock -n 200 ) 200<$BUILDDIR/bitbake.lock || lock=0
+fi
+
+if [ ${CMD} == "start" ] && ( [ $lock -eq 0 ] || [ -e $BUILDDIR/.toastermain.pid ] ); then
+    echo "Error: bitbake lock state error. System is already on." 2>&1
+    return 3
+elif [ ${CMD} == "stop" ] && ( [ $lock -eq 1 ] || ! [ -e $BUILDDIR/.toastermain.pid ] ) ; then
+    echo "Error: bitbake lock state error. Trying to stop a stopped system ?
+If you think the system is hanged up, you can try to manually stop system with the commands
+
+# BBSERVER=localhost:8200 bitbake -m
+
+and
+
+# webserverKillAll
+" 2>&1
+    return 3
+fi
+
+
+
+
+
+# Execute the commands
+
+case $CMD in
+    start )
+        addtoConfiguration "INHERIT+=\"toaster buildhistory\"" toaster.conf
+        webserverStartAll || return 4
+        unset BBSERVER
+        bitbake --postread conf/toaster.conf --server-only -t xmlrpc -B localhost:8200
+        export BBSERVER=localhost:8200
+        if [ $NOTOASTERUI == 0 ]; then        # we start the TOASTERUI only if not inhibited
+            bitbake --observe-only -u toasterui >${BUILDDIR}/toaster_ui.log 2>&1 & echo $! >${BUILDDIR}/.toasterui.pid
+        fi
+        # stop system on terminal exit
+        trap stop_system SIGHUP
+    ;;
+    stop )
+        stop_system
+        trap '' SIGHUP
+    ;;
+esac
+
+echo "Successful ${CMD}."
+

bitbake/contrib/vim/ftdetect/bitbake.vim

@@ -14,7 +14,7 @@ endif
 au BufNewFile,BufRead *.{bb,bbappend,bbclass}	set filetype=bitbake
 
 " .inc
-au BufNewFile,BufRead *.inc 		set filetype=bitbake
+au BufNewFile,BufRead *.inc		set filetype=bitbake
 
 " .conf
 au BufNewFile,BufRead *.conf

bitbake/contrib/vim/ftplugin/bitbake.vim

@@ -1 +1,2 @@
 set sts=4 sw=4 et
+set cms=#%s

bitbake/contrib/vim/plugin/newbb.vim

@@ -15,7 +15,7 @@ if &compatible || v:version < 600
 endif
 
 fun! <SID>GetUserName()
-    let l:user_name = system("git-config --get user.name")
+    let l:user_name = system("git config --get user.name")
     if v:shell_error
         return "Unknow User"
     else
@@ -23,7 +23,7 @@ fun! <SID>GetUserName()
 endfun
 
 fun! <SID>GetUserEmail()
-    let l:user_email = system("git-config --get user.email")
+    let l:user_email = system("git config --get user.email")
     if v:shell_error
         return "unknow@user.org"
     else

bitbake/contrib/vim/syntax/bitbake.vim

@@ -44,8 +44,8 @@ syn match bbArrayBrackets       "[\[\]]" contained
 
 " BitBake strings
 syn match bbContinue            "\\$"
-syn region bbString             matchgroup=bbQuote start=+"+ skip=+\\$+ excludenl end=+"+ contained keepend contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
-syn region bbString             matchgroup=bbQuote start=+'+ skip=+\\$+ excludenl end=+'+ contained keepend contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
+syn region bbString             matchgroup=bbQuote start=+"+ skip=+\\$+ end=+"+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
+syn region bbString             matchgroup=bbQuote start=+'+ skip=+\\$+ end=+'+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
 
 " Vars definition
 syn match bbExport            "^export" nextgroup=bbIdentifier skipwhite
@@ -55,11 +55,11 @@ syn match bbVarDeref            "${[a-zA-Z0-9\-_\.\/\+]\+}" contained
 syn match bbVarEq               "\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)" contained nextgroup=bbVarValue
 syn match bbVarDef              "^\(export\s*\)\?\([a-zA-Z0-9\-_\.\/\+]\+\(_[${}a-zA-Z0-9\-_\.\/\+]\+\)\?\)\s*\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)\@=" contains=bbExportFlag,bbIdentifier,bbVarDeref nextgroup=bbVarEq
 syn match bbVarValue            ".*$" contained contains=bbString,bbVarDeref,bbVarPyValue
-syn region bbVarPyValue         start=+${@+ skip=+\\$+ excludenl end=+}+ contained contains=@python
+syn region bbVarPyValue         start=+${@+ skip=+\\$+ end=+}+ contained contains=@python
 
 " Vars metadata flags
 syn match bbVarFlagDef          "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\.]\+\]\)\@=" contains=bbIdentifier nextgroup=bbVarFlagFlag
-syn region bbVarFlagFlag        matchgroup=bbArrayBrackets start="\[" end="\]\s*\(=\)\@=" keepend excludenl contained contains=bbIdentifier nextgroup=bbVarEq
+syn region bbVarFlagFlag        matchgroup=bbArrayBrackets start="\[" end="\]\s*\(=\|+=\|=+\|?=\)\@=" contained contains=bbIdentifier nextgroup=bbVarEq
 
 " Includes and requires
 syn keyword bbInclude           inherit include require contained 
@@ -83,13 +83,16 @@ if exists("b:current_syntax")
   unlet b:current_syntax
 endif
 syn keyword bbShFakeRootFlag    fakeroot contained
-syn match bbShFuncDef           "^\(fakeroot\s*\)\?\([0-9A-Za-z_-]\+\)\(python\)\@<!\(\s*()\s*\)\({\)\@=" contains=bbShFakeRootFlag,bbFunction,bbDelimiter nextgroup=bbShFuncRegion skipwhite
-syn region bbShFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" keepend contained contains=@shell
+syn match bbShFuncDef           "^\(fakeroot\s*\)\?\([0-9A-Za-z_${}-]\+\)\(python\)\@<!\(\s*()\s*\)\({\)\@=" contains=bbShFakeRootFlag,bbFunction,bbVarDeref,bbDelimiter nextgroup=bbShFuncRegion skipwhite
+syn region bbShFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" contained contains=@shell
+
+" Python value inside shell functions
+syn region shDeref         start=+${@+ skip=+\\$+ excludenl end=+}+ contained contains=@python
 
 " BitBake python metadata
 syn keyword bbPyFlag            python contained
-syn match bbPyFuncDef           "^\(python\s\+\)\([0-9A-Za-z_-]\+\)\?\(\s*()\s*\)\({\)\@=" contains=bbPyFlag,bbFunction,bbDelimiter nextgroup=bbPyFuncRegion skipwhite 
-syn region bbPyFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" keepend contained contains=@python
+syn match bbPyFuncDef           "^\(python\s\+\)\([0-9A-Za-z_${}-]\+\)\?\(\s*()\s*\)\({\)\@=" contains=bbPyFlag,bbFunction,bbVarDeref,bbDelimiter nextgroup=bbPyFuncRegion skipwhite
+syn region bbPyFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" contained contains=@python
 
 " BitBake 'def'd python functions
 syn keyword bbPyDef             def contained

bitbake/doc/manual/usermanual.xml

@@ -137,6 +137,24 @@ share common metadata between many packages.</para></listitem>
                 <para>In this example, <varname>B</varname> is now <literal>bvaladditionaldata</literal> and <varname>C</varname> is <literal>testcval</literal>. In contrast to the above appending and prepending operators, no additional space
 will be introduced.</para>
             </section>
+            <section>
+                <title>Appending and Prepending (override style syntax)</title>
+                    <para><screen><varname>B</varname> = "bval"
+<varname>B_append</varname> = " additional data"
+<varname>C</varname> = "cval"
+<varname>C_prepend</varname> = "additional data "</screen></para>
+                 <para>This example results in <varname>B</varname> becoming <literal>bval additional data</literal>
+and <varname>C</varname> becoming <literal>additional data cval</literal>. Note the spaces in the append.
+Unlike the += operator, additional space is not automatically added.  You must take steps to add space
+yourself.</para>
+            </section>
+            <section>
+                <title>Removing (override style syntax)</title>
+                <para><screen><varname>FOO</varname> = "123 456 789 123456 123 456 123 456"
+<varname>FOO_remove</varname> = "123"
+<varname>FOO_remove</varname> = "456"</screen></para>
+                <para>In this example, <varname>FOO</varname> is now <literal>789 123456</literal>.</para>
+            </section>
             <section>
                 <title>Conditional metadata set</title>
                 <para>OVERRIDES is a <quote>:</quote> separated variable containing each item you want to satisfy conditions.  So, if you have a variable which is conditional on <quote>arm</quote>, and <quote>arm</quote> is in OVERRIDES, then the <quote>arm</quote> specific version of the variable is used rather than the non-conditional version.  Example:</para>
@@ -536,7 +554,7 @@ options:
                 <example>
                     <title>Generating dependency graphs</title>
                     <para>BitBake is able to generate dependency graphs using the dot syntax. These graphs can be converted
-to images using the <application>dot</application> application from <ulink url="http://www.graphviz.org">Graphviz</ulink>. 
+to images using the <application>dot</application> application from <ulink url="http://www.graphviz.org">Graphviz</ulink>.
 Two files will be written into the current working directory, <emphasis>depends.dot</emphasis> containing dependency information at the package level and <emphasis>task-depends.dot</emphasis> containing a breakdown of the dependencies at the task level. To stop depending on common depends, one can use the <prompt>-I depend</prompt> to omit these from the graph. This can lead to more readable graphs.  This way, <varname>DEPENDS</varname> from inherited classes such as base.bbclass can be removed from the graph.</para>
                     <screen><prompt>$ </prompt>bitbake -g blah</screen>
                     <screen><prompt>$ </prompt>bitbake -g -I virtual/whatever -I bloom blah</screen>

bitbake/lib/bb/__init__.py

@@ -21,7 +21,7 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-__version__ = "1.19.1"
+__version__ = "1.20.0"
 
 import sys
 if sys.version_info < (2, 7, 3):

bitbake/lib/bb/build.py

@@ -69,9 +69,12 @@ class FuncFailed(Exception):
 class TaskBase(event.Event):
     """Base class for task events"""
 
-    def __init__(self, t, d ):
+    def __init__(self, t, logfile, d):
         self._task = t
         self._package = d.getVar("PF", True)
+        self.taskfile = d.getVar("FILE", True)
+        self.taskname = self._task
+        self.logfile = logfile
         event.Event.__init__(self)
         self._message = "recipe %s: task %s: %s" % (d.getVar("PF", True), t, self.getDisplayName())
 
@@ -88,6 +91,9 @@ class TaskBase(event.Event):
 
 class TaskStarted(TaskBase):
     """Task execution started"""
+    def __init__(self, t, logfile, taskflags, d):
+        super(TaskStarted, self).__init__(t, logfile, d)
+        self.taskflags = taskflags
 
 class TaskSucceeded(TaskBase):
     """Task execution completed"""
@@ -96,16 +102,11 @@ class TaskFailed(TaskBase):
     """Task execution failed"""
 
     def __init__(self, task, logfile, metadata, errprinted = False):
-        self.logfile = logfile
         self.errprinted = errprinted
-        super(TaskFailed, self).__init__(task, metadata)
+        super(TaskFailed, self).__init__(task, logfile, metadata)
 
 class TaskFailedSilent(TaskBase):
     """Task execution failed (silently)"""
-    def __init__(self, task, logfile, metadata):
-        self.logfile = logfile
-        super(TaskFailedSilent, self).__init__(task, metadata)
-
     def getDisplayName(self):
         # Don't need to tell the user it was silent
         return "Failed"
@@ -113,7 +114,7 @@ class TaskFailedSilent(TaskBase):
 class TaskInvalid(TaskBase):
 
     def __init__(self, task, metadata):
-        super(TaskInvalid, self).__init__(task, metadata)
+        super(TaskInvalid, self).__init__(task, None, metadata)
         self._message = "No such task '%s'" % task
 
 
@@ -348,6 +349,14 @@ def _exec_task(fn, task, d, quieterr):
     if not tempdir:
         bb.fatal("T variable not set, unable to build")
 
+    # Change nice level if we're asked to
+    nice = localdata.getVar("BB_TASK_NICE_LEVEL", True)
+    if nice:
+        curnice = os.nice(0)
+        nice = int(nice) - curnice
+        newnice = os.nice(nice)
+        logger.debug(1, "Renice to %s " % newnice)
+
     bb.utils.mkdirhier(tempdir)
 
     # Determine the logfile to generate
@@ -416,7 +425,9 @@ def _exec_task(fn, task, d, quieterr):
     localdata.setVar('BB_LOGFILE', logfn)
     localdata.setVar('BB_RUNTASK', task)
 
-    event.fire(TaskStarted(task, localdata), localdata)
+    flags = localdata.getVarFlags(task)
+
+    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
     try:
         for func in (prefuncs or '').split():
             exec_func(func, localdata)
@@ -453,7 +464,7 @@ def _exec_task(fn, task, d, quieterr):
             logger.debug(2, "Zero size logfn %s, removing", logfn)
             bb.utils.remove(logfn)
             bb.utils.remove(loglink)
-    event.fire(TaskSucceeded(task, localdata), localdata)
+    event.fire(TaskSucceeded(task, logfn, localdata), localdata)
 
     if not localdata.getVarFlag(task, 'nostamp') and not localdata.getVarFlag(task, 'selfstamp'):
         make_stamp(task, localdata)

bitbake/lib/bb/cache.py

@@ -43,7 +43,7 @@ except ImportError:
     logger.info("Importing cPickle failed. "
                 "Falling back to a very slow implementation.")
 
-__cache_version__ = "146"
+__cache_version__ = "147"
 
 def getCacheFile(path, filename, data_hash):
     return os.path.join(path, filename + "." + data_hash)

bitbake/lib/bb/cache_extra.py

@@ -35,6 +35,12 @@ class HobRecipeInfo(RecipeInfoCommon):
     # such as (bb_cache.dat, bb_extracache_hob.dat) 
     cachefile = "bb_extracache_" + classname +".dat"        
 
+    # override this member with the list of extra cache fields
+    # that this class will provide
+    cachefields = ['summary', 'license', 'section',
+            'description', 'homepage', 'bugtracker',
+            'prevision', 'files_info']
+
     def __init__(self, filename, metadata):
 
         self.summary = self.getvar('SUMMARY', metadata)

bitbake/lib/bb/command.py

@@ -117,14 +117,14 @@ class Command:
             return False
 
     def finishAsyncCommand(self, msg=None, code=None):
-        if msg:
+        if msg or msg == "":
             bb.event.fire(CommandFailed(msg), self.cooker.event_data)
         elif code:
             bb.event.fire(CommandExit(code), self.cooker.event_data)
         else:
             bb.event.fire(CommandCompleted(), self.cooker.event_data)
         self.currentAsyncCommand = None
-
+        self.cooker.finishcommand()
 
 class CommandsSync:
     """
@@ -137,13 +137,22 @@ class CommandsSync:
         """
         Trigger cooker 'shutdown' mode
         """
-        command.cooker.shutdown()
+        command.cooker.shutdown(False)
 
-    def stateStop(self, command, params):
+    def stateForceShutdown(self, command, params):
         """
         Stop the cooker
         """
-        command.cooker.stop()
+        command.cooker.shutdown(True)
+
+    def getAllKeysWithFlags(self, command, params):
+        """
+        Returns a dump of the global state. Call with
+        variable flags to be retrieved as params.
+        """
+        flaglist = params[0]
+        return command.cooker.getAllKeysWithFlags(flaglist)
+    getAllKeysWithFlags.readonly = True
 
     def getVariable(self, command, params):
         """
@@ -232,6 +241,13 @@ class CommandsSync:
         op = params[3]
         command.cooker.modifyConfigurationVar(var, val, default_file, op)
 
+    def removeVarFile(self, command, params):
+        """
+        Remove a variable declaration from a file
+        """
+        var = params[0]
+        command.cooker.removeConfigurationVar(var)
+
     def createConfigFile(self, command, params):
         """
         Create an extra configuration file

bitbake/lib/bb/cooker.py

@@ -61,7 +61,7 @@ class CollectionError(bb.BBHandledException):
     """
 
 class state:
-    initial, parsing, running, shutdown, stop = range(5)
+    initial, parsing, running, shutdown, forceshutdown, stopped = range(6)
 
 
 class SkippedPackage:
@@ -79,6 +79,29 @@ class SkippedPackage:
         elif reason:
             self.skipreason = reason
 
+
+class CookerFeatures(object):
+    _feature_list = [HOB_EXTRA_CACHES, SEND_DEPENDS_TREE, BASEDATASTORE_TRACKING] = range(3)
+
+    def __init__(self):
+        self._features=set()
+
+    def setFeature(self, f):
+        # validate we got a request for a feature we support
+        if f not in CookerFeatures._feature_list:
+            return
+        self._features.add(f)
+
+    def __contains__(self, f):
+        return f in self._features
+
+    def __iter__(self):
+        return self._features.__iter__()
+
+    def next(self):
+        return self._features.next()
+
+
 #============================================================================#
 # BBCooker
 #============================================================================#
@@ -90,25 +113,10 @@ class BBCooker:
     def __init__(self, configuration):
         self.recipecache = None
         self.skiplist = {}
+        self.featureset = CookerFeatures()
 
         self.configuration = configuration
 
-        self.caches_array = []
-
-        caches_name_array = ['bb.cache:CoreRecipeInfo'] + configuration.extra_caches
-
-        # At least CoreRecipeInfo will be loaded, so caches_array will never be empty!
-        # This is the entry point, no further check needed!
-        for var in caches_name_array:
-            try:
-                module_name, cache_name = var.split(':')
-                module = __import__(module_name, fromlist=(cache_name,))
-                self.caches_array.append(getattr(module, cache_name)) 
-            except ImportError as exc:
-                logger.critical("Unable to import extra RecipeInfo '%s' from '%s': %s" % (cache_name, module_name, exc))
-                sys.exit("FATAL: Failed to import extra cache class '%s'." % cache_name)
-
-        self.data = None
         self.loadConfigurationData()
 
         # Take a lock so only one copy of bitbake can run against a given build
@@ -118,13 +126,6 @@ class BBCooker:
         if not self.lock:
             bb.fatal("Only one copy of bitbake should be run against a build directory")
 
-        #
-        # Special updated configuration we use for firing events
-        #
-        self.event_data = bb.data.createCopy(self.data)
-        bb.data.update_data(self.event_data)
-        bb.parse.init_parser(self.event_data)
-
         # TOSTOP must not be set or our children will hang when they output
         fd = sys.stdout.fileno()
         if os.isatty(fd):
@@ -141,11 +142,30 @@ class BBCooker:
         self.parser = None
 
     def initConfigurationData(self):
-        worker = False
-        if not self.configuration.server_register_idlecallback:
-            worker = True
 
-        self.databuilder = bb.cookerdata.CookerDataBuilder(self.configuration, worker)
+        self.state = state.initial
+
+        self.caches_array = []
+
+        all_extra_cache_names = []
+        # We hardcode all known cache types in a single place, here.
+        if CookerFeatures.HOB_EXTRA_CACHES in self.featureset:
+            all_extra_cache_names.append("bb.cache_extra:HobRecipeInfo")
+
+        caches_name_array = ['bb.cache:CoreRecipeInfo'] + all_extra_cache_names
+
+        # At least CoreRecipeInfo will be loaded, so caches_array will never be empty!
+        # This is the entry point, no further check needed!
+        for var in caches_name_array:
+            try:
+                module_name, cache_name = var.split(':')
+                module = __import__(module_name, fromlist=(cache_name,))
+                self.caches_array.append(getattr(module, cache_name))
+            except ImportError as exc:
+                logger.critical("Unable to import extra RecipeInfo '%s' from '%s': %s" % (cache_name, module_name, exc))
+                sys.exit("FATAL: Failed to import extra cache class '%s'." % cache_name)
+
+        self.databuilder = bb.cookerdata.CookerDataBuilder(self.configuration, False)
         self.data = self.databuilder.data
 
     def enableDataTracking(self):
@@ -157,16 +177,33 @@ class BBCooker:
         self.data.disableTracking()
 
     def loadConfigurationData(self):
+        if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
+            self.enableDataTracking()
+
         self.initConfigurationData()
         self.databuilder.parseBaseConfiguration()
         self.data = self.databuilder.data
         self.data_hash = self.databuilder.data_hash
 
+        #
+        # Special updated configuration we use for firing events
+        #
+        self.event_data = bb.data.createCopy(self.data)
+        bb.data.update_data(self.event_data)
+        bb.parse.init_parser(self.event_data)
+
+        if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
+            self.disableDataTracking()
+
+
     def modifyConfigurationVar(self, var, val, default_file, op):
         if op == "append":
             self.appendConfigurationVar(var, val, default_file)
         elif op == "set":
-            self.saveConfigurationVar(var, val, default_file)
+            self.saveConfigurationVar(var, val, default_file, "=")
+        elif op == "earlyAssign":
+            self.saveConfigurationVar(var, val, default_file, "?=")
+
 
     def appendConfigurationVar(self, var, val, default_file):
         #add append var operation to the end of default_file
@@ -180,7 +217,7 @@ class BBCooker:
         for c in contents:
             total += c
 
-        total += "#added by bitbake"
+        total += "#added by hob"
         total += "\n%s += \"%s\"\n" % (var, val)
 
         with open(default_file, 'w') as f:
@@ -191,7 +228,7 @@ class BBCooker:
         loginfo = {"op":append, "file":default_file, "line":total.count("\n")}
         self.data.appendVar(var, val, **loginfo)
 
-    def saveConfigurationVar(self, var, val, default_file):
+    def saveConfigurationVar(self, var, val, default_file, op):
 
         replaced = False
         #do not save if nothing changed
@@ -233,8 +270,8 @@ class BBCooker:
                     #check if the variable was saved before in the same way
                     #if true it replace the place where the variable was declared
                     #else it comments it
-                    if contents[begin_line-1]== "#added by bitbake\n":
-                        contents[begin_line] = "%s = \"%s\"\n" % (var, val)
+                    if contents[begin_line-1]== "#added by hob\n":
+                        contents[begin_line] = "%s %s \"%s\"\n" % (var, op, val)
                         replaced = True
                     else:
                         for ii in range(begin_line, end_line):
@@ -263,8 +300,8 @@ class BBCooker:
                 total += c
 
             #add the variable on a single line, to be easy to replace the second time
-            total += "\n#added by bitbake"
-            total += "\n%s = \"%s\"\n" % (var, val)
+            total += "\n#added by hob"
+            total += "\n%s %s \"%s\"\n" % (var, op, val)
 
             with open(default_file, 'w') as f:
                 f.write(total)
@@ -274,13 +311,50 @@ class BBCooker:
             loginfo = {"op":set, "file":default_file, "line":total.count("\n")}
             self.data.setVar(var, val, **loginfo)
 
+    def removeConfigurationVar(self, var):
+        conf_files = self.data.varhistory.get_variable_files(var)
+        topdir = self.data.getVar("TOPDIR")
+
+        for conf_file in conf_files:
+            if topdir in conf_file:
+                with open(conf_file, 'r') as f:
+                    contents = f.readlines()
+                f.close()
+
+                lines = self.data.varhistory.get_variable_lines(var, conf_file)
+                for line in lines:
+                    total = ""
+                    i = 0
+                    for c in contents:
+                        total += c
+                        i = i + 1
+                        if i==int(line):
+                            end_index = len(total)
+                    index = total.rfind(var, 0, end_index)
+
+                    begin_line = total.count("\n",0,index)
+
+                    #check if the variable was saved before in the same way
+                    if contents[begin_line-1]== "#added by hob\n":
+                        contents[begin_line-1] = contents[begin_line] = "\n"
+                    else:
+                        contents[begin_line] = "\n"
+                    #remove var from history
+                    self.data.varhistory.del_var_history(var, conf_file, line)
+
+                total = ""
+                for c in contents:
+                    total += c
+                with open(conf_file, 'w') as f:
+                    f.write(total)
+                f.close()
+
     def createConfigFile(self, name):
         path = os.getcwd()
         confpath = os.path.join(path, "conf", name)
         open(confpath, 'w').close()
 
     def parseConfiguration(self):
-
         # Set log file verbosity
         verboselogs = bb.utils.to_boolean(self.data.getVar("BB_VERBOSE_LOGS", "0"))
         if verboselogs:
@@ -348,13 +422,7 @@ class BBCooker:
             if pkgs_to_build[0] in set(ignore.split()):
                 bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0])
 
-            localdata = data.createCopy(self.data)
-            bb.data.update_data(localdata)
-            bb.data.expandKeys(localdata)
-
-            taskdata = bb.taskdata.TaskData(self.configuration.abort)
-            taskdata.add_provider(localdata, self.recipecache, pkgs_to_build[0])
-            taskdata.add_unresolved(localdata, self.recipecache)
+            taskdata, runlist, pkgs_to_build = self.buildTaskData(pkgs_to_build, None, self.configuration.abort)
 
             targetid = taskdata.getbuild_id(pkgs_to_build[0])
             fnid = taskdata.build_targets[targetid][0]
@@ -386,34 +454,44 @@ class BBCooker:
             if data.getVarFlag( e, 'python', envdata ):
                 logger.plain("\npython %s () {\n%s}\n", e, data.getVar(e, envdata, 1))
 
-    def prepareTreeData(self, pkgs_to_build, task):
+
+    def buildTaskData(self, pkgs_to_build, task, abort):
         """
         Prepare a runqueue and taskdata object for iteration over pkgs_to_build
         """
         bb.event.fire(bb.event.TreeDataPreparationStarted(), self.data)
 
-        # If we are told to do the None task then query the default task
-        if (task == None):
+        # A task of None means use the default task
+        if task is None:
             task = self.configuration.cmd
 
-        pkgs_to_build = self.checkPackages(pkgs_to_build)
+        fulltargetlist = self.checkPackages(pkgs_to_build)
 
         localdata = data.createCopy(self.data)
         bb.data.update_data(localdata)
         bb.data.expandKeys(localdata)
+        taskdata = bb.taskdata.TaskData(abort, skiplist=self.skiplist)
+
+        current = 0
+        runlist = []
+        for k in fulltargetlist:
+            taskdata.add_provider(localdata, self.recipecache, k)
+            current += 1
+            runlist.append([k, "do_%s" % task])
+            bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(fulltargetlist)), self.data)
+        taskdata.add_unresolved(localdata, self.recipecache)
+        bb.event.fire(bb.event.TreeDataPreparationCompleted(len(fulltargetlist)), self.data)
+        return taskdata, runlist, fulltargetlist
+
+    def prepareTreeData(self, pkgs_to_build, task):
+        """
+        Prepare a runqueue and taskdata object for iteration over pkgs_to_build
+        """
+
         # We set abort to False here to prevent unbuildable targets raising
         # an exception when we're just generating data
-        taskdata = bb.taskdata.TaskData(False, skiplist=self.skiplist)
+        taskdata, runlist, pkgs_to_build = self.buildTaskData(pkgs_to_build, task, False)
 
-        runlist = []
-        current = 0
-        for k in pkgs_to_build:
-            taskdata.add_provider(localdata, self.recipecache, k)
-            runlist.append([k, "do_%s" % task])
-            current += 1
-            bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(pkgs_to_build)), self.data)
-        taskdata.add_unresolved(localdata, self.recipecache)
-        bb.event.fire(bb.event.TreeDataPreparationCompleted(len(pkgs_to_build)), self.data)
         return runlist, taskdata
     
     ######## WARNING : this function requires cache_extra to be enabled ########
@@ -426,7 +504,10 @@ class BBCooker:
         runlist, taskdata = self.prepareTreeData(pkgs_to_build, task)
         rq = bb.runqueue.RunQueue(self, self.data, self.recipecache, taskdata, runlist)
         rq.rqdata.prepare()
+        return self.buildDependTree(rq, taskdata)
 
+
+    def buildDependTree(self, rq, taskdata):
         seen_fnids = []
         depend_tree = {}
         depend_tree["depends"] = {}
@@ -436,6 +517,7 @@ class BBCooker:
         depend_tree["packages"] = {}
         depend_tree["rdepends-pkg"] = {}
         depend_tree["rrecs-pkg"] = {}
+        depend_tree["layer-priorities"] = self.recipecache.bbfile_config_priorities
 
         for task in xrange(len(rq.rqdata.runq_fnid)):
             taskname = rq.rqdata.runq_task[task]
@@ -447,6 +529,20 @@ class BBCooker:
                 depend_tree["pn"][pn] = {}
                 depend_tree["pn"][pn]["filename"] = fn
                 depend_tree["pn"][pn]["version"] = version
+                depend_tree["pn"][pn]["inherits"] = self.recipecache.inherits.get(fn, None)
+
+                # if we have extra caches, list all attributes they bring in
+                extra_info = []
+                for cache_class in self.caches_array:
+                    if type(cache_class) is type and issubclass(cache_class, bb.cache.RecipeInfoCommon) and hasattr(cache_class, 'cachefields'):
+                        cachefields = getattr(cache_class, 'cachefields', [])
+                        extra_info = extra_info + cachefields
+
+                # for all attributes stored, add them to the dependency tree
+                for ei in extra_info:
+                    depend_tree["pn"][pn][ei] = vars(self.recipecache)[ei][fn]
+
+
             for dep in rq.rqdata.runq_depends[task]:
                 depfn = taskdata.fn_index[rq.rqdata.runq_fnid[dep]]
                 deppn = self.recipecache.pkg_fn[depfn]
@@ -509,35 +605,30 @@ class BBCooker:
         depend_tree["rdepends-pkg"] = {}
         depend_tree["rrecs-pkg"] = {}
 
+        # if we have extra caches, list all attributes they bring in
+        extra_info = []
+        for cache_class in self.caches_array:
+            if type(cache_class) is type and issubclass(cache_class, bb.cache.RecipeInfoCommon) and hasattr(cache_class, 'cachefields'):
+                cachefields = getattr(cache_class, 'cachefields', [])
+                extra_info = extra_info + cachefields
+
         for task in xrange(len(tasks_fnid)):
             fnid = tasks_fnid[task]
             fn = taskdata.fn_index[fnid]
             pn = self.recipecache.pkg_fn[fn]
-            version  = "%s:%s-%s" % self.recipecache.pkg_pepvpr[fn]
-            summary = self.recipecache.summary[fn]
-            lic = self.recipecache.license[fn]
-            section = self.recipecache.section[fn]
-            description = self.recipecache.description[fn]
-            homepage = self.recipecache.homepage[fn]
-            bugtracker = self.recipecache.bugtracker[fn]
-            files_info = self.recipecache.files_info[fn]
-            rdepends = self.recipecache.rundeps[fn]
-            rrecs = self.recipecache.runrecs[fn]
-            prevision = self.recipecache.prevision[fn]
-            inherits = self.recipecache.inherits.get(fn, None)
+
             if pn not in depend_tree["pn"]:
                 depend_tree["pn"][pn] = {}
                 depend_tree["pn"][pn]["filename"] = fn
+                version  = "%s:%s-%s" % self.recipecache.pkg_pepvpr[fn]
                 depend_tree["pn"][pn]["version"] = version
-                depend_tree["pn"][pn]["summary"] = summary
-                depend_tree["pn"][pn]["license"] = lic
-                depend_tree["pn"][pn]["section"] = section
-                depend_tree["pn"][pn]["description"] = description
-                depend_tree["pn"][pn]["inherits"] = inherits
-                depend_tree["pn"][pn]["homepage"] = homepage
-                depend_tree["pn"][pn]["bugtracker"] = bugtracker
-                depend_tree["pn"][pn]["files_info"] = files_info
-                depend_tree["pn"][pn]["revision"] = prevision
+                rdepends = self.recipecache.rundeps[fn]
+                rrecs = self.recipecache.runrecs[fn]
+                depend_tree["pn"][pn]["inherits"] = self.recipecache.inherits.get(fn, None)
+
+                # for all extra attributes stored, add them to the dependency tree
+                for ei in extra_info:
+                    depend_tree["pn"][pn][ei] = vars(self.recipecache)[ei][fn]
 
             if fnid not in seen_fnids:
                 seen_fnids.append(fnid)
@@ -1000,7 +1091,6 @@ class BBCooker:
 
         self.buildSetVars()
 
-        self.recipecache = bb.cache.CacheData(self.caches_array)
         infos = bb.cache.Cache.parse(fn, self.collection.get_file_appends(fn), \
                                      self.data,
                                      self.caches_array)
@@ -1047,7 +1137,7 @@ class BBCooker:
 
         def buildFileIdle(server, rq, abort):
 
-            if abort or self.state == state.stop:
+            if abort or self.state == state.forceshutdown:
                 rq.finish_runqueue(True)
             elif self.state == state.shutdown:
                 rq.finish_runqueue(False)
@@ -1076,15 +1166,8 @@ class BBCooker:
         Attempt to build the targets specified
         """
 
-        # If we are told to do the NULL task then query the default task
-        if (task == None):
-            task = self.configuration.cmd
-
-        universe = ('universe' in targets)
-        targets = self.checkPackages(targets)
-
         def buildTargetsIdle(server, rq, abort):
-            if abort or self.state == state.stop:
+            if abort or self.state == state.forceshutdown:
                 rq.finish_runqueue(True)
             elif self.state == state.shutdown:
                 rq.finish_runqueue(False)
@@ -1108,27 +1191,35 @@ class BBCooker:
 
         self.buildSetVars()
 
+        taskdata, runlist, fulltargetlist = self.buildTaskData(targets, task, self.configuration.abort)
+
         buildname = self.data.getVar("BUILDNAME")
-        bb.event.fire(bb.event.BuildStarted(buildname, targets), self.data)
-
-        localdata = data.createCopy(self.data)
-        bb.data.update_data(localdata)
-        bb.data.expandKeys(localdata)
-
-        taskdata = bb.taskdata.TaskData(self.configuration.abort, skiplist=self.skiplist)
-
-        runlist = []
-        for k in targets:
-            taskdata.add_provider(localdata, self.recipecache, k)
-            runlist.append([k, "do_%s" % task])
-        taskdata.add_unresolved(localdata, self.recipecache)
+        bb.event.fire(bb.event.BuildStarted(buildname, fulltargetlist), self.data)
 
         rq = bb.runqueue.RunQueue(self, self.data, self.recipecache, taskdata, runlist)
-        if universe:
+        if 'universe' in targets:
             rq.rqdata.warn_multi_bb = True
 
         self.configuration.server_register_idlecallback(buildTargetsIdle, rq)
 
+
+    def getAllKeysWithFlags(self, flaglist):
+        dump = {}
+        for k in self.data.keys():
+            try:
+                v = self.data.getVar(k, True)
+                if not k.startswith("__") and not isinstance(v, bb.data_smart.DataSmart):
+                    dump[k] = {
+    'v' : v ,
+    'history' : self.data.varhistory.variable(k),
+                    }
+                    for d in flaglist:
+                        dump[k][d] = self.data.getVarFlag(k, d)
+            except Exception as e:
+                print(e)
+        return dump
+
+
     def generateNewImage(self, image, base_image, package_queue, timestamp, description):
         '''
         Create a new image with a "require"/"inherit" base_image statement
@@ -1173,9 +1264,9 @@ class BBCooker:
         if self.state == state.running:
             return
 
-        if self.state in (state.shutdown, state.stop):
+        if self.state in (state.shutdown, state.forceshutdown):
             self.parser.shutdown(clean=False, force = True)
-            sys.exit(1)
+            raise bb.BBHandledException()
 
         if self.state != state.parsing:
             self.parseConfiguration ()
@@ -1197,7 +1288,7 @@ class BBCooker:
         if not self.parser.parse_next():
             collectlog.debug(1, "parsing complete")
             if self.parser.error:
-                sys.exit(1)
+                raise bb.BBHandledException()
             self.show_appends_with_no_recipes()
             self.handlePrefProviders()
             self.recipecache.bbfile_priority = self.collection.collection_priorities(self.recipecache.pkg_fn)
@@ -1208,6 +1299,9 @@ class BBCooker:
 
     def checkPackages(self, pkgs_to_build):
 
+        # Return a copy, don't modify the original
+        pkgs_to_build = pkgs_to_build[:]
+
         if len(pkgs_to_build) == 0:
             raise NothingToBuild
 
@@ -1243,18 +1337,19 @@ class BBCooker:
         prserv.serv.auto_shutdown(self.data)
         bb.event.fire(CookerExit(), self.event_data)
 
-    def shutdown(self):
-        self.state = state.shutdown
+    def shutdown(self, force = False):
+        if force:
+            self.state = state.forceshutdown
+        else:
+            self.state = state.shutdown
 
-    def stop(self):
-        self.state = state.stop
+    def finishcommand(self):
+        self.state = state.initial
 
     def initialize(self):
-        self.state = state.initial
         self.initConfigurationData()
 
     def reset(self):
-        self.state = state.initial
         self.loadConfigurationData()
 
 def server_main(cooker, func, *args):
@@ -1482,7 +1577,7 @@ class Feeder(multiprocessing.Process):
                 continue
 
 class Parser(multiprocessing.Process):
-    def __init__(self, jobs, results, quit, init):
+    def __init__(self, jobs, results, quit, init, profile):
         self.jobs = jobs
         self.results = results
         self.quit = quit
@@ -1490,8 +1585,28 @@ class Parser(multiprocessing.Process):
         multiprocessing.Process.__init__(self)
         self.context = bb.utils.get_context().copy()
         self.handlers = bb.event.get_class_handlers().copy()
+        self.profile = profile
 
     def run(self):
+
+        if not self.profile:
+            self.realrun()
+            return
+
+        try:
+            import cProfile as profile
+        except:
+            import profile
+        prof = profile.Profile()
+        try:
+            profile.Profile.runcall(prof, self.realrun)
+        finally:
+            logfile = "profile-parse-%s.log" % multiprocessing.current_process().name
+            prof.dump_stats(logfile)
+            bb.utils.process_profilelog(logfile)
+            print("Raw profiling information saved to %s and processed statistics to %s.processed" % (logfile, logfile))
+
+    def realrun(self):
         if self.init:
             self.init()
 
@@ -1592,7 +1707,7 @@ class CookerParser(object):
             self.feeder = Feeder(self.willparse, self.jobs, self.feeder_quit)
             self.feeder.start()
             for i in range(0, self.num_processes):
-                parser = Parser(self.jobs, self.result_queue, self.parser_quit, init)
+                parser = Parser(self.jobs, self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
                 parser.start()
                 self.processes.append(parser)
 

bitbake/lib/bb/cookerdata.py

@@ -127,7 +127,6 @@ class CookerConfiguration(object):
         self.dump_signatures = False
         self.dry_run = False
         self.tracking = False
-        self.extra_caches = []
 
         self.env = {}
 

bitbake/lib/bb/data.py

@@ -59,7 +59,7 @@ def init():
 def init_db(parent = None):
     """Return a new object representing the Bitbake data,
     optionally based on an existing object"""
-    if parent:
+    if parent is not None:
         return parent.createCopy()
     else:
         return _dict_type()
@@ -148,7 +148,7 @@ def expandKeys(alterdata, readdata = None):
         readdata = alterdata
 
     todolist = {}
-    for key in keys(alterdata):
+    for key in alterdata:
         if not '${' in key:
             continue
 
@@ -285,20 +285,24 @@ def update_data(d):
     """Performs final steps upon the datastore, including application of overrides"""
     d.finalize(parent = True)
 
-def build_dependencies(key, keys, shelldeps, vardepvals, d):
+def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
     deps = set()
-    vardeps = d.getVarFlag(key, "vardeps", True)
     try:
         if key[-1] == ']':
             vf = key[:-1].split('[')
             value = d.getVarFlag(vf[0], vf[1], False)
-        else:
-            value = d.getVar(key, False)
+            parser = d.expandWithRefs(value, key)
+            deps |= parser.references
+            deps = deps | (keys & parser.execs)
+            return deps, value
+        varflags = d.getVarFlags(key, ["vardeps", "vardepvalue", "vardepsexclude"]) or {}
+        vardeps = varflags.get("vardeps")
+        value = d.getVar(key, False)
 
-        if key in vardepvals:
-           value =  d.getVarFlag(key, "vardepvalue", True)
-        elif d.getVarFlag(key, "func"):
-            if d.getVarFlag(key, "python"):
+        if "vardepvalue" in varflags:
+           value = varflags.get("vardepvalue")
+        elif varflags.get("func"):
+            if varflags.get("python"):
                 parsedvar = d.expandWithRefs(value, key)
                 parser = bb.codeparser.PythonParser(key, logger)
                 if parsedvar.value and "\t" in parsedvar.value:
@@ -320,19 +324,16 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
             deps = deps | (keys & parser.execs)
 
         # Add varflags, assuming an exclusion list is set
-        varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
         if varflagsexcl:
             varfdeps = []
-            varflags = d.getVarFlags(key)
-            if varflags:
-                for f in varflags:
-                    if f not in varflagsexcl:
-                        varfdeps.append('%s[%s]' % (key, f))
+            for f in varflags:
+                if f not in varflagsexcl:
+                    varfdeps.append('%s[%s]' % (key, f))
             if varfdeps:
                 deps |= set(varfdeps)
 
         deps |= set((vardeps or "").split())
-        deps -= set((d.getVarFlag(key, "vardepsexclude", True) or "").split())
+        deps -= set(varflags.get("vardepsexclude", "").split())
     except Exception as e:
         raise bb.data_smart.ExpansionError(key, None, e)
     return deps, value
@@ -341,16 +342,16 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
 
 def generate_dependencies(d):
 
-    keys = set(key for key in d.keys() if not key.startswith("__"))
-    shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
-    vardepvals = set(key for key in keys if d.getVarFlag(key, "vardepvalue"))
+    keys = set(key for key in d if not key.startswith("__"))
+    shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
+    varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
 
     deps = {}
     values = {}
 
     tasklist = d.getVar('__BBTASKS') or []
     for task in tasklist:
-        deps[task], values[task] = build_dependencies(task, keys, shelldeps, vardepvals, d)
+        deps[task], values[task] = build_dependencies(task, keys, shelldeps, varflagsexcl, d)
         newdeps = deps[task]
         seen = set()
         while newdeps:
@@ -359,7 +360,7 @@ def generate_dependencies(d):
             newdeps = set()
             for dep in nextdeps:
                 if dep not in deps:
-                    deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, vardepvals, d)
+                    deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, varflagsexcl, d)
                 newdeps |=  deps[dep]
             newdeps -= seen
         #print "For %s: %s" % (task, str(deps[task]))

bitbake/lib/bb/data_smart.py

@@ -40,7 +40,7 @@ logger = logging.getLogger("BitBake.Data")
 
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
 __setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>.*))?$')
-__expand_var_regexp__ = re.compile(r"\${[^{}]+}")
+__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t ]+}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
 
 def infer_caller_details(loginfo, parent = False, varval = True):
@@ -94,9 +94,13 @@ class VariableParse:
             if self.varname and key:
                 if self.varname == key:
                     raise Exception("variable %s references itself!" % self.varname)
-            var = self.d.getVar(key, True)
+            if key in self.d.expand_cache:
+                varparse = self.d.expand_cache[key]
+                var = varparse.value
+            else:
+                var = self.d.getVar(key, True)
+            self.references.add(key)
             if var is not None:
-                self.references.add(key)
                 return var
             else:
                 return match.group()
@@ -277,9 +281,13 @@ class VariableHistory(object):
                 lines.append(line)
         return lines
 
-    def del_var_history(self, var):
+    def del_var_history(self, var, f=None, line=None):
+        """If file f and line are not given, the entire history of var is deleted"""
         if var in self.variables:
-            self.variables[var] = []
+            if f and line:
+                self.variables[var] = [ x for x in self.variables[var] if x['file']!=f and x['line']!=line]
+            else:
+                self.variables[var] = []
 
 class DataSmart(MutableMapping):
     def __init__(self, special = COWDictBase.copy(), seen = COWDictBase.copy() ):
@@ -505,12 +513,7 @@ class DataSmart(MutableMapping):
             self._seen_overrides[override].add( var )
 
     def getVar(self, var, expand=False, noweakdefault=False):
-        value = self.getVarFlag(var, "_content", False, noweakdefault)
-
-        # Call expand() separately to make use of the expand cache
-        if expand and value:
-            return self.expand(value, var)
-        return value
+        return self.getVarFlag(var, "_content", expand, noweakdefault)
 
     def renameVar(self, key, newkey, **loginfo):
         """
@@ -578,17 +581,30 @@ class DataSmart(MutableMapping):
         if flag == "defaultval" and '_' in var:
             self._setvar_update_overrides(var)
 
+        if flag == "unexport" or flag == "export":
+            if not "__exportlist" in self.dict:
+                self._makeShadowCopy("__exportlist")
+            if not "_content" in self.dict["__exportlist"]:
+                self.dict["__exportlist"]["_content"] = set()
+            self.dict["__exportlist"]["_content"].add(var)
+
     def getVarFlag(self, var, flag, expand=False, noweakdefault=False):
         local_var = self._findVar(var)
         value = None
-        if local_var:
+        if local_var is not None:
             if flag in local_var:
                 value = copy.copy(local_var[flag])
             elif flag == "_content" and "defaultval" in local_var and not noweakdefault:
                 value = copy.copy(local_var["defaultval"])
         if expand and value:
-            value = self.expand(value, None)
-        if value and flag == "_content" and local_var and "_removeactive" in local_var:
+            # Only getvar (flag == _content) hits the expand cache
+            cachename = None
+            if flag == "_content":
+                cachename = var
+            else:
+                cachename = var + "[" + flag + "]"
+            value = self.expand(value, cachename)
+        if value is not None and flag == "_content" and local_var is not None and "_removeactive" in local_var:
             filtered = filter(lambda v: v not in local_var["_removeactive"],
                               value.split(" "))
             value = " ".join(filtered)
@@ -636,16 +652,17 @@ class DataSmart(MutableMapping):
             self.varhistory.record(**loginfo)
             self.dict[var][i] = flags[i]
 
-    def getVarFlags(self, var):
+    def getVarFlags(self, var, expand = False, internalflags=False):
         local_var = self._findVar(var)
         flags = {}
 
         if local_var:
             for i in local_var:
-                if i.startswith("_"):
+                if i.startswith("_") and not internalflags:
                     continue
                 flags[i] = local_var[i]
-
+                if expand and i in expand:
+                    flags[i] = self.expand(flags[i], var + "[" + i + "]")
         if len(flags) == 0:
             return None
         return flags
@@ -751,13 +768,16 @@ class DataSmart(MutableMapping):
         for key in keys:
             if key in config_whitelist:
                 continue
+
             value = d.getVar(key, False) or ""
             data.update({key:value})
 
-            varflags = d.getVarFlags(key)
+            varflags = d.getVarFlags(key, internalflags = True)
             if not varflags:
                 continue
             for f in varflags:
+                if f == "_content":
+                    continue
                 data.update({'%s[%s]' % (key, f):varflags[f]})
 
         for key in ["__BBTASKS", "__BBANONFUNCS", "__BBHANDLERS"]:

bitbake/lib/bb/event.py

@@ -589,6 +589,16 @@ class PackageInfo(Event):
         Event.__init__(self)
         self._pkginfolist = pkginfolist
 
+class MetadataEvent(Event):
+    """
+    Generic event that target for OE-Core classes
+    to report information during asynchrous execution
+    """
+    def __init__(self, eventtype, eventdata):
+        Event.__init__(self)
+        self.type = eventtype
+        self.data = eventdata
+
 class SanityCheck(Event):
     """
     Event to issue sanity check

bitbake/lib/bb/monitordisk.py

@@ -225,7 +225,7 @@ class diskMonitor:
                         self.preFreeS[k] = freeSpace
 
                     if action == "STOPTASKS" and not self.checked[k]:
-                        logger.error("No new tasks can be excuted since the disk space monitor action is \"STOPTASKS\"!")
+                        logger.error("No new tasks can be executed since the disk space monitor action is \"STOPTASKS\"!")
                         self.checked[k] = True
                         rq.finish_runqueue(False)
                         bb.event.fire(bb.event.DiskFull(dev, 'disk', freeSpace, path), self.configuration)
@@ -253,7 +253,7 @@ class diskMonitor:
                         self.preFreeI[k] = freeInode
 
                     if action  == "STOPTASKS" and not self.checked[k]:
-                        logger.error("No new tasks can be excuted since the disk space monitor action is \"STOPTASKS\"!")
+                        logger.error("No new tasks can be executed since the disk space monitor action is \"STOPTASKS\"!")
                         self.checked[k] = True
                         rq.finish_runqueue(False)
                         bb.event.fire(bb.event.DiskFull(dev, 'inode', freeInode, path), self.configuration)

bitbake/lib/bb/providers.py

@@ -341,7 +341,7 @@ def filterProvidersRunTime(providers, item, cfgData, dataCache):
     if numberPreferred > 1:
         logger.error("Trying to resolve runtime dependency %s resulted in conflicting PREFERRED_PROVIDER entries being found.\nThe providers found were: %s\nThe PREFERRED_PROVIDER entries resulting in this conflict were: %s", item, preferred, preferred_vars)
 
-    logger.debug(1, "sorted providers for %s are: %s", item, eligible)
+    logger.debug(1, "sorted runtime providers for %s are: %s", item, eligible)
 
     return eligible, numberPreferred
 

bitbake/lib/bb/runqueue.py

@@ -217,6 +217,15 @@ class RunQueueData:
             ret.extend([nam])
         return ret
 
+    def get_task_name(self, task):
+        return self.runq_task[task]
+
+    def get_task_file(self, task):
+        return self.taskData.fn_index[self.runq_fnid[task]]
+
+    def get_task_hash(self, task):
+        return self.runq_hash[task]
+
     def get_user_idstring(self, task, task_name_suffix = ""):
         fn = self.taskData.fn_index[self.runq_fnid[task]]
         taskname = self.runq_task[task] + task_name_suffix
@@ -696,6 +705,14 @@ class RunQueueData:
                     prov_list[prov].append(fn)
         for prov in prov_list:
             if len(prov_list[prov]) > 1 and prov not in self.multi_provider_whitelist:
+                seen_pn = []
+                # If two versions of the same PN are being built its fatal, we don't support it.
+                for fn in prov_list[prov]:
+                    pn = self.dataCache.pkg_fn[fn]
+                    if pn not in seen_pn:
+                        seen_pn.append(pn)
+                    else:
+                        bb.fatal("Multiple versions of %s are due to be built (%s). Only one version of a given PN should be built in any given build. You likely need to set PREFERRED_VERSION_%s to select the correct version or don't depend on multiple versions." % (pn, " ".join(prov_list[prov]), pn))
                 msg = "Multiple .bb files are due to be built which each provide %s (%s)." % (prov, " ".join(prov_list[prov]))
                 if self.warn_multi_bb:
                     logger.warn(msg)
@@ -703,7 +720,6 @@ class RunQueueData:
                     msg += "\n This usually means one provides something the other doesn't and should."
                     logger.error(msg)
 
-
         # Create a whitelist usable by the stamp checks
         stampfnwhitelist = []
         for entry in self.stampwhitelist.split():
@@ -853,6 +869,8 @@ class RunQueue:
             "logdefaultdomain" : bb.msg.loggerDefaultDomains,
             "prhost" : self.cooker.prhost,
             "buildname" : self.cfgData.getVar("BUILDNAME", True),
+            "date" : self.cfgData.getVar("DATE", True),
+            "time" : self.cfgData.getVar("TIME", True),
         }
 
         worker.stdin.write("<cookerconfig>" + pickle.dumps(self.cooker.configuration) + "</cookerconfig>")
@@ -990,6 +1008,11 @@ class RunQueue:
             else:
                 self.state = runQueueSceneInit
 
+                # we are ready to run, see if any UI client needs the dependency info
+                if bb.cooker.CookerFeatures.SEND_DEPENDS_TREE in self.cooker.featureset:
+                    depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
+                    bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
+
         if self.state is runQueueSceneInit:
             if self.cooker.configuration.dump_signatures:
                 self.dump_signatures()
@@ -1316,9 +1339,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
         if self.rqdata.taskData.abort:
             self.rq.state = runQueueCleanUp
 
-    def task_skip(self, task):
+    def task_skip(self, task, reason):
         self.runq_running[task] = 1
         self.runq_buildable[task] = 1
+        bb.event.fire(runQueueTaskSkipped(task, self.stats, self.rq, reason), self.cfgData)
         self.task_completeoutright(task)
         self.stats.taskCompleted()
         self.stats.taskSkipped()
@@ -1343,13 +1367,13 @@ class RunQueueExecuteTasks(RunQueueExecute):
             if task in self.rq.scenequeue_covered:
                 logger.debug(2, "Setscene covered task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "covered")
                 return True
 
             if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
                 logger.debug(2, "Stamp current task %s (%s)", task,
                                 self.rqdata.get_user_idstring(task))
-                self.task_skip(task)
+                self.task_skip(task, "existing")
                 return True
 
             taskdep = self.rqdata.dataCache.task_deps[fn]
@@ -1636,6 +1660,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
 
     def task_complete(self, task):
         self.stats.taskCompleted()
+        bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
         self.task_completeoutright(task)
 
     def task_fail(self, task, result):
@@ -1770,6 +1795,9 @@ class runQueueEvent(bb.event.Event):
     def __init__(self, task, stats, rq):
         self.taskid = task
         self.taskstring = rq.rqdata.get_user_idstring(task)
+        self.taskname = rq.rqdata.get_task_name(task)
+        self.taskfile = rq.rqdata.get_task_file(task)
+        self.taskhash = rq.rqdata.get_task_hash(task)
         self.stats = stats.copy()
         bb.event.Event.__init__(self)
 
@@ -1781,6 +1809,9 @@ class sceneQueueEvent(runQueueEvent):
         runQueueEvent.__init__(self, task, stats, rq)
         realtask = rq.rqdata.runq_setscene[task]
         self.taskstring = rq.rqdata.get_user_idstring(realtask, "_setscene")
+        self.taskname = rq.rqdata.get_task_name(realtask) + "_setscene"
+        self.taskfile = rq.rqdata.get_task_file(realtask)
+        self.taskhash = rq.rqdata.get_task_hash(task)
 
 class runQueueTaskStarted(runQueueEvent):
     """
@@ -1819,6 +1850,19 @@ class runQueueTaskCompleted(runQueueEvent):
     Event notifing a task completed
     """
 
+class sceneQueueTaskCompleted(sceneQueueEvent):
+    """
+    Event notifing a setscene task completed
+    """
+
+class runQueueTaskSkipped(runQueueEvent):
+    """
+    Event notifing a task was skipped
+    """
+    def __init__(self, task, stats, rq, reason):
+        runQueueEvent.__init__(self, task, stats, rq)
+        self.reason = reason
+
 class runQueuePipe():
     """
     Abstraction for a pipe between a worker thread and the server

bitbake/lib/bb/server/__init__.py

@@ -89,7 +89,7 @@ class BitBakeBaseServer(object):
     def detach(self):
         return
 
-    def establishConnection(self):
+    def establishConnection(self, featureset):
         raise   "Must redefine the %s.establishConnection()" % self.__class__.__name__
 
     def endSession(self):

bitbake/lib/bb/server/process.py

@@ -31,7 +31,7 @@ import sys
 import time
 import select
 from Queue import Empty
-from multiprocessing import Event, Process, util, Queue, Pipe, queues
+from multiprocessing import Event, Process, util, Queue, Pipe, queues, Manager
 
 from . import BitBakeBaseServer, BitBakeBaseServerConnection, BaseImplServer
 
@@ -78,12 +78,13 @@ class ProcessServer(Process, BaseImplServer):
     profile_filename = "profile.log"
     profile_processed_filename = "profile.log.processed"
 
-    def __init__(self, command_channel, event_queue):
+    def __init__(self, command_channel, event_queue, featurelist):
         BaseImplServer.__init__(self)
-        Process.__init__(self)
+        Process.__init__(self, args=(featurelist))
         self.command_channel = command_channel
         self.event_queue = event_queue
         self.event = EventAdapter(event_queue)
+        self.featurelist = featurelist
         self.quit = False
 
         self.keep_running = Event()
@@ -94,6 +95,14 @@ class ProcessServer(Process, BaseImplServer):
         for event in bb.event.ui_queue:
             self.event_queue.put(event)
         self.event_handle.value = bb.event.register_UIHhandler(self)
+
+        # process any feature changes based on what UI requested
+        original_featureset = list(self.cooker.featureset)
+        while len(self.featurelist)> 0:
+            self.cooker.featureset.setFeature(self.featurelist.pop())
+        if (original_featureset != list(self.cooker.featureset)):
+            self.cooker.reset()
+
         bb.cooker.server_main(self.cooker, self.main)
 
     def main(self):
@@ -113,7 +122,7 @@ class ProcessServer(Process, BaseImplServer):
         self.event_queue.close()
         bb.event.unregister_UIHhandler(self.event_handle.value)
         self.command_channel.close()
-        self.cooker.stop()
+        self.cooker.shutdown(True)
         self.idle_commands(.1)
 
     def idle_commands(self, delay, fds = []):
@@ -198,13 +207,17 @@ class BitBakeServer(BitBakeBaseServer):
         #
         self.ui_channel, self.server_channel = Pipe()
         self.event_queue = ProcessEventQueue(0)
-        self.serverImpl = ProcessServer(self.server_channel, self.event_queue)
+        manager = Manager()
+        self.featurelist = manager.list()
+        self.serverImpl = ProcessServer(self.server_channel, self.event_queue, self.featurelist)
 
     def detach(self):
         self.serverImpl.start()
         return
 
-    def establishConnection(self):
+    def establishConnection(self, featureset):
+        for f in featureset:
+            self.featurelist.append(f)
         self.connection = BitBakeProcessServerConnection(self.serverImpl, self.ui_channel, self.event_queue)
         signal.signal(signal.SIGTERM, lambda i, s: self.connection.terminate())
         return self.connection

bitbake/lib/bb/server/xmlrpc.py

@@ -89,12 +89,23 @@ class BitBakeServerCommands():
         self.server = server
         self.has_client = False
 
-    def registerEventHandler(self, host, port):
+    def registerEventHandler(self, host, port, featureset = []):
         """
         Register a remote UI Event Handler
         """
         s, t = _create_server(host, port)
 
+        # we don't allow connections if the cooker is running
+        if (self.cooker.state in [bb.cooker.state.parsing, bb.cooker.state.running]):
+            return None
+
+        original_featureset = list(self.cooker.featureset)
+        for f in featureset:
+            self.cooker.featureset.setFeature(f)
+
+        if (original_featureset != list(self.cooker.featureset)):
+            self.cooker.reset()
+
         self.event_handle = bb.event.register_UIHhandler(s)
         return self.event_handle
 
@@ -169,51 +180,6 @@ class BitBakeXMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
         self.end_headers()
         self.wfile.write(response)
 
-class BitBakeUIEventServer(threading.Thread):
-    class EventAdapter():
-        """
-        Adapter to wrap our event queue since the caller (bb.event) expects to
-        call a send() method, but our actual queue only has put()
-        """
-        def __init__(self, notify):
-            self.queue = []
-            self.notify = notify
-            self.qlock = threading.Lock()
-
-        def send(self, event):
-            self.qlock.acquire()
-            self.queue.append(event)
-            self.qlock.release()
-            self.notify.set()
-
-        def get(self):
-            self.qlock.acquire()
-            if len(self.queue) == 0:
-                self.qlock.release()
-                return None
-            e = self.queue.pop(0)
-            if len(self.queue) == 0:
-                self.notify.clear()
-            self.qlock.release()
-            return e
-
-    def __init__(self, connection):
-        self.connection = connection
-        self.notify = threading.Event()
-        self.event = BitBakeUIEventServer.EventAdapter(self.notify)
-        self.quit = False
-        threading.Thread.__init__(self)
-
-    def terminateServer(self):
-        self.quit = True
-
-    def run(self):
-        while not self.quit:
-            self.notify.wait(0.1)
-            evt = self.event.get()
-            if evt:
-                self.connection.event.sendpickle(pickle.dumps(evt))
-
 
 class XMLRPCProxyServer(BaseImplServer):
     """ not a real working server, but a stub for a proxy server connection
@@ -308,11 +274,12 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
         self.connection_token = token
 
 class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
-    def __init__(self, serverImpl, clientinfo=("localhost", 0), observer_only = False):
+    def __init__(self, serverImpl, clientinfo=("localhost", 0), observer_only = False, featureset = []):
         self.connection, self.transport = _create_server(serverImpl.host, serverImpl.port)
         self.clientinfo = clientinfo
         self.serverImpl = serverImpl
         self.observer_only = observer_only
+        self.featureset = featureset
 
     def connect(self):
         if not self.observer_only:
@@ -322,7 +289,8 @@ class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
         if token is None:
             return None
         self.transport.set_connection_token(token)
-        self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo)
+
+        self.events = uievent.BBUIEventQueue(self.connection, self.clientinfo, self.featureset)
         for event in bb.event.ui_queue:
             self.events.queue_event(event)
         return self
@@ -346,14 +314,15 @@ class BitBakeXMLRPCServerConnection(BitBakeBaseServerConnection):
 
 class BitBakeServer(BitBakeBaseServer):
     def initServer(self, interface = ("localhost", 0)):
+        self.interface = interface
         self.serverImpl = XMLRPCServer(interface)
 
     def detach(self):
         daemonize.createDaemon(self.serverImpl.serve_forever, "bitbake-cookerdaemon.log")
         del self.cooker
 
-    def establishConnection(self):
-        self.connection = BitBakeXMLRPCServerConnection(self.serverImpl)
+    def establishConnection(self, featureset):
+        self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, self.interface, False, featureset)
         return self.connection.connect()
 
     def set_connection_token(self, token):
@@ -363,12 +332,13 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
 
     def __init__(self, observer_only = False):
         self.observer_only = observer_only
+        # if we need extra caches, just tell the server to load them all
         pass
 
     def saveConnectionDetails(self, remote):
         self.remote = remote
 
-    def establishConnection(self):
+    def establishConnection(self, featureset):
         # The format of "remote" must be "server:port"
         try:
             [host, port] = self.remote.split(":")
@@ -384,9 +354,12 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             s.close()
         except:
             return None
-        self.serverImpl = XMLRPCProxyServer(host, port)
-        self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only)
-        return self.connection.connect()
+        try:
+            self.serverImpl = XMLRPCProxyServer(host, port)
+            self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
+            return self.connection.connect()
+        except Exception as e:
+            bb.fatal("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
 
     def endSession(self):
         self.connection.removeClient()

bitbake/lib/bb/siggen.py

@@ -91,8 +91,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
         basehash = {}
 
         for task in tasklist:
-            data = d.getVar(task, False)
-            lookupcache[task] = data
+            data = lookupcache[task]
 
             if data is None:
                 bb.error("Task %s from %s seems to be empty?!" % (task, fn))
@@ -115,16 +114,8 @@ class SignatureGeneratorBasic(SignatureGenerator):
             alldeps = sorted(seen)
             for dep in alldeps:
                 data = data + dep
-                if dep in lookupcache:
-                    var = lookupcache[dep]
-                elif dep[-1] == ']':
-                    vf = dep[:-1].split('[')
-                    var = d.getVarFlag(vf[0], vf[1], False)
-                    lookupcache[dep] = var
-                else:
-                    var = d.getVar(dep, False)
-                    lookupcache[dep] = var
-                if var:
+                var = lookupcache[dep]
+                if var is not None:
                     data = data + str(var)
             self.basehash[fn + "." + task] = hashlib.md5(data).hexdigest()
             taskdeps[task] = alldeps

bitbake/lib/bb/tests/data.py

@@ -27,21 +27,21 @@ import bb.data
 class DataExpansions(unittest.TestCase):
     def setUp(self):
         self.d = bb.data.init()
-        self.d["foo"] = "value of foo"
-        self.d["bar"] = "value of bar"
-        self.d["value of foo"] = "value of 'value of foo'"
+        self.d["foo"] = "value_of_foo"
+        self.d["bar"] = "value_of_bar"
+        self.d["value_of_foo"] = "value_of_'value_of_foo'"
 
     def test_one_var(self):
         val = self.d.expand("${foo}")
-        self.assertEqual(str(val), "value of foo")
+        self.assertEqual(str(val), "value_of_foo")
 
     def test_indirect_one_var(self):
         val = self.d.expand("${${foo}}")
-        self.assertEqual(str(val), "value of 'value of foo'")
+        self.assertEqual(str(val), "value_of_'value_of_foo'")
 
     def test_indirect_and_another(self):
         val = self.d.expand("${${foo}} ${bar}")
-        self.assertEqual(str(val), "value of 'value of foo' value of bar")
+        self.assertEqual(str(val), "value_of_'value_of_foo' value_of_bar")
 
     def test_python_snippet(self):
         val = self.d.expand("${@5*12}")
@@ -49,11 +49,11 @@ class DataExpansions(unittest.TestCase):
 
     def test_expand_in_python_snippet(self):
         val = self.d.expand("${@'boo ' + '${foo}'}")
-        self.assertEqual(str(val), "boo value of foo")
+        self.assertEqual(str(val), "boo value_of_foo")
 
     def test_python_snippet_getvar(self):
         val = self.d.expand("${@d.getVar('foo', True) + ' ${bar}'}")
-        self.assertEqual(str(val), "value of foo value of bar")
+        self.assertEqual(str(val), "value_of_foo value_of_bar")
 
     def test_python_snippet_syntax_error(self):
         self.d.setVar("FOO", "${@foo = 5}")
@@ -70,7 +70,7 @@ class DataExpansions(unittest.TestCase):
 
     def test_value_containing_value(self):
         val = self.d.expand("${@d.getVar('foo', True) + ' ${bar}'}")
-        self.assertEqual(str(val), "value of foo value of bar")
+        self.assertEqual(str(val), "value_of_foo value_of_bar")
 
     def test_reference_undefined_var(self):
         val = self.d.expand("${undefinedvar} meh")
@@ -109,7 +109,7 @@ class DataExpansions(unittest.TestCase):
 
     def test_rename(self):
         self.d.renameVar("foo", "newfoo")
-        self.assertEqual(self.d.getVar("newfoo"), "value of foo")
+        self.assertEqual(self.d.getVar("newfoo"), "value_of_foo")
         self.assertEqual(self.d.getVar("foo"), None)
 
     def test_deletion(self):
@@ -118,17 +118,17 @@ class DataExpansions(unittest.TestCase):
 
     def test_keys(self):
         keys = self.d.keys()
-        self.assertEqual(keys, ['value of foo', 'foo', 'bar'])
+        self.assertEqual(keys, ['value_of_foo', 'foo', 'bar'])
 
 class TestNestedExpansions(unittest.TestCase):
     def setUp(self):
         self.d = bb.data.init()
         self.d["foo"] = "foo"
         self.d["bar"] = "bar"
-        self.d["value of foobar"] = "187"
+        self.d["value_of_foobar"] = "187"
 
     def test_refs(self):
-        val = self.d.expand("${value of ${foo}${bar}}")
+        val = self.d.expand("${value_of_${foo}${bar}}")
         self.assertEqual(str(val), "187")
 
     #def test_python_refs(self):
@@ -154,11 +154,11 @@ class TestNestedExpansions(unittest.TestCase):
     #    self.assertEqual(str(val), str(depth + 1))
 
     def test_mixed(self):
-        val = self.d.expand("${value of ${@('${foo}'+'bar')[0:3]}${${@'BAR'.lower()}}}")
+        val = self.d.expand("${value_of_${@('${foo}'+'bar')[0:3]}${${@'BAR'.lower()}}}")
         self.assertEqual(str(val), "187")
 
     def test_runtime(self):
-        val = self.d.expand("${${@'value of' + ' f'+'o'+'o'+'b'+'a'+'r'}}")
+        val = self.d.expand("${${@'value_of' + '_f'+'o'+'o'+'b'+'a'+'r'}}")
         self.assertEqual(str(val), "187")
 
 class TestMemoize(unittest.TestCase):

bitbake/lib/bb/ui/buildinfohelper.py

@@ -0,0 +1,738 @@
+#
+# BitBake ToasterUI Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import datetime
+import sys
+import bb
+import re
+import subprocess
+
+
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "toaster.toastermain.settings")
+
+import toaster.toastermain.settings as toaster_django_settings
+from toaster.orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage
+from toaster.orm.models import Variable, VariableHistory
+from toaster.orm.models import Target_Package, Build_Package, Build_File
+from toaster.orm.models import Task_Dependency, Build_Package_Dependency
+from toaster.orm.models import Target_Package_Dependency, Recipe_Dependency
+from bb.msg import BBLogFormatter as format
+
+class ORMWrapper(object):
+    """ This class creates the dictionaries needed to store information in the database
+        following the format defined by the Django models. It is also used to save this
+        information in the database.
+    """
+
+    def __init__(self):
+        pass
+
+
+    def create_build_object(self, build_info):
+
+        build = Build.objects.create(
+                                    machine=build_info['machine'],
+                                    image_fstypes=build_info['image_fstypes'],
+                                    distro=build_info['distro'],
+                                    distro_version=build_info['distro_version'],
+                                    started_on=build_info['started_on'],
+                                    completed_on=build_info['completed_on'],
+                                    cooker_log_path=build_info['cooker_log_path'],
+                                    build_name=build_info['build_name'],
+                                    bitbake_version=build_info['bitbake_version'])
+
+        return build
+
+    def create_target_objects(self, target_info):
+        targets = []
+        for tgt_name in target_info['targets']:
+            tgt_object = Target.objects.create( build = target_info['build'],
+                                    target = tgt_name,
+                                    is_image = False,
+                                    file_name = "",
+                                    file_size = 0);
+            targets.append(tgt_object)
+        return targets
+
+    def update_build_object(self, build, errors, warnings, taskfailures):
+
+        outcome = Build.SUCCEEDED
+        if errors or taskfailures:
+            outcome = Build.FAILED
+
+        build.completed_on = datetime.datetime.now()
+        build.errors_no = errors
+        build.warnings_no = warnings
+        build.outcome = outcome
+        build.save()
+
+
+    def get_update_task_object(self, task_information):
+        task_object, created = Task.objects.get_or_create(
+                                build=task_information['build'],
+                                recipe=task_information['recipe'],
+                                task_name=task_information['task_name'],
+                                )
+
+        for v in vars(task_object):
+            if v in task_information.keys():
+                vars(task_object)[v] = task_information[v]
+        # if we got covered by a setscene task, we're SSTATE
+        if task_object.outcome == Task.OUTCOME_COVERED and 1 == Task.objects.filter(task_executed=True, build = task_object.build, recipe = task_object.recipe, task_name=task_object.task_name+"_setscene").count():
+            task_object.outcome = Task.OUTCOME_SSTATE
+            outcome_task_setscene = Task.objects.get(task_executed=True, build = task_object.build,
+                                    recipe = task_object.recipe, task_name=task_object.task_name+"_setscene").outcome
+            if outcome_task_setscene == Task.OUTCOME_SUCCESS:
+                task_object.sstate_result = Task.SSTATE_RESTORED
+            elif outcome_task_setscene == Task.OUTCOME_FAILED:
+                task_object.sstate_result = Task.SSTATE_FAILED
+
+        # mark down duration if we have a start time
+        if 'start_time' in task_information.keys():
+            duration = datetime.datetime.now() - task_information['start_time']
+            task_object.elapsed_time = duration.total_seconds()
+
+        task_object.save()
+        return task_object
+
+
+    def get_update_recipe_object(self, recipe_information):
+
+        recipe_object, created = Recipe.objects.get_or_create(
+                                         layer_version=recipe_information['layer_version'],
+                                         file_path=recipe_information['file_path'])
+
+        for v in vars(recipe_object):
+            if v in recipe_information.keys():
+                vars(recipe_object)[v] = recipe_information[v]
+
+        recipe_object.save()
+
+        return recipe_object
+
+    def get_layer_version_object(self, layer_version_information):
+
+        layer_version_object = Layer_Version.objects.get_or_create(
+                                    layer = layer_version_information['layer'],
+                                    branch = layer_version_information['branch'],
+                                    commit = layer_version_information['commit'],
+                                    priority = layer_version_information['priority']
+                                    )
+
+        layer_version_object[0].save()
+
+        return layer_version_object[0]
+
+    def get_update_layer_object(self, layer_information):
+
+        layer_object = Layer.objects.get_or_create(
+                                name=layer_information['name'],
+                                local_path=layer_information['local_path'],
+                                layer_index_url=layer_information['layer_index_url'])
+        layer_object[0].save()
+
+        return layer_object[0]
+
+
+    def save_target_package_information(self, target_obj, packagedict, bldpkgs, recipes):
+        for p in packagedict:
+            packagedict[p]['object'] = Target_Package.objects.create( target = target_obj,
+                                        name = p,
+                                        size = packagedict[p]['size'])
+            if p in bldpkgs:
+                packagedict[p]['object'].version = bldpkgs[p]['version']
+                packagedict[p]['object'].recipe =  recipes[bldpkgs[p]['pn']]
+                packagedict[p]['object'].save()
+
+        for p in packagedict:
+            for (px,deptype) in packagedict[p]['depends']:
+                Target_Package_Dependency.objects.create( package = packagedict[p]['object'],
+                                        depends_on = packagedict[px]['object'],
+                                        dep_type = deptype);
+
+
+    def create_logmessage(self, log_information):
+        log_object = LogMessage.objects.create(
+                        build = log_information['build'],
+                        level = log_information['level'],
+                        message = log_information['message'])
+
+        for v in vars(log_object):
+            if v in log_information.keys():
+                vars(log_object)[v] = log_information[v]
+
+        return log_object.save()
+
+
+    def save_build_package_information(self, build_obj, package_info, recipes):
+        # create and save the object
+        bp_object = Build_Package.objects.create( build = build_obj,
+                                       recipe = recipes[package_info['PN']],
+                                       name = package_info['PKG'],
+                                       version = package_info['PKGV'],
+                                       revision = package_info['PKGR'],
+                                       summary = package_info['SUMMARY'],
+                                       description = package_info['DESCRIPTION'],
+                                       size = int(package_info['PKGSIZE']) * 1024,
+                                       section = package_info['SECTION'],
+                                       license = package_info['LICENSE'],
+                                       )
+        # save any attached file information
+        for path in package_info['FILES_INFO']:
+                fo = Build_File.objects.create( bpackage = bp_object,
+                                        path = path,
+                                        size = package_info['FILES_INFO'][path] )
+
+        # save soft dependency information
+        if 'RDEPENDS' in package_info and package_info['RDEPENDS']:
+            for p in bb.utils.explode_deps(package_info['RDEPENDS']):
+                Build_Package_Dependency.objects.get_or_create( package = bp_object,
+                    depends_on = p, dep_type = Build_Package_Dependency.TYPE_RDEPENDS)
+        if 'RPROVIDES' in package_info and package_info['RPROVIDES']:
+            for p in bb.utils.explode_deps(package_info['RPROVIDES']):
+                Build_Package_Dependency.objects.get_or_create( package = bp_object,
+                    depends_on = p, dep_type = Build_Package_Dependency.TYPE_RPROVIDES)
+        if 'RRECOMMENDS' in package_info and package_info['RRECOMMENDS']:
+            for p in bb.utils.explode_deps(package_info['RRECOMMENDS']):
+                Build_Package_Dependency.objects.get_or_create( package = bp_object,
+                    depends_on = p, dep_type = Build_Package_Dependency.TYPE_RRECOMMENDS)
+        if 'RSUGGESTS' in package_info and package_info['RSUGGESTS']:
+            for p in bb.utils.explode_deps(package_info['RSUGGESTS']):
+                Build_Package_Dependency.objects.get_or_create( package = bp_object,
+                    depends_on = p, dep_type = Build_Package_Dependency.TYPE_RSUGGESTS)
+        if 'RREPLACES' in package_info and package_info['RREPLACES']:
+            for p in bb.utils.explode_deps(package_info['RREPLACES']):
+                Build_Package_Dependency.objects.get_or_create( package = bp_object,
+                    depends_on = p, dep_type = Build_Package_Dependency.TYPE_RREPLACES)
+        if 'RCONFLICTS' in package_info and package_info['RCONFLICTS']:
+            for p in bb.utils.explode_deps(package_info['RCONFLICTS']):
+                Build_Package_Dependency.objects.get_or_create( package = bp_object,
+                    depends_on = p, dep_type = Build_Package_Dependency.TYPE_RCONFLICTS)
+
+        return bp_object
+
+    def save_build_variables(self, build_obj, vardump):
+        for k in vardump:
+            if not bool(vardump[k]['func']):
+                value = vardump[k]['v'];
+                if value is None:
+                    value = ''
+                desc = vardump[k]['doc'];
+                if desc is None:
+                    var_words = [word for word in k.split('_')]
+                    root_var = "_".join([word for word in var_words if word.isupper()])
+                    if root_var and root_var != k and root_var in vardump:
+                        desc = vardump[root_var]['doc']
+                if desc is None:
+                    desc = ''
+                variable_obj = Variable.objects.create( build = build_obj,
+                    variable_name = k,
+                    variable_value = value,
+                    description = desc)
+                for vh in vardump[k]['history']:
+                    VariableHistory.objects.create( variable = variable_obj,
+                            file_name = vh['file'],
+                            line_number = vh['line'],
+                            operation = vh['op'])
+
+class BuildInfoHelper(object):
+    """ This class gathers the build information from the server and sends it
+        towards the ORM wrapper for storing in the database
+        It is instantiated once per build
+        Keeps in memory all data that needs matching before writing it to the database
+    """
+
+    def __init__(self, server, has_build_history = False):
+        self._configure_django()
+        self.internal_state = {}
+        self.task_order = 0
+        self.server = server
+        self.orm_wrapper = ORMWrapper()
+        self.has_build_history = has_build_history
+        self.tmp_dir = self.server.runCommand(["getVariable", "TMPDIR"])[0]
+
+    def _configure_django(self):
+        # Add toaster to sys path for importing modules
+        sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'toaster'))
+
+    ###################
+    ## methods to convert event/external info into objects that the ORM layer uses
+
+    def _get_layer_dict(self, layer_path):
+
+        layer_info = {}
+        layer_name = layer_path.split('/')[-1]
+        layer_url = 'http://layers.openembedded.org/layerindex/layer/{layer}/'
+        layer_url_name = self._get_url_map_name(layer_name)
+
+        layer_info['name'] = layer_name
+        layer_info['local_path'] = layer_path
+        layer_info['layer_index_url'] = layer_url.format(layer=layer_url_name)
+
+        return layer_info
+
+    def _get_url_map_name(self, layer_name):
+        """ Some layers have a different name on openembedded.org site,
+            this method returns the correct name to use in the URL
+        """
+
+        url_name = layer_name
+        url_mapping = {'meta': 'openembedded-core'}
+
+        for key in url_mapping.keys():
+            if key == layer_name:
+                url_name = url_mapping[key]
+
+        return url_name
+
+    def _get_layer_information(self):
+
+        layer_info = {}
+
+        return layer_info
+
+    def _get_layer_version_information(self, layer_object):
+
+        layer_version_info = {}
+        layer_version_info['build'] = self.internal_state['build']
+        layer_version_info['layer'] = layer_object
+        layer_version_info['branch'] = self._get_git_branch(layer_object.local_path)
+        layer_version_info['commit'] = self._get_git_revision(layer_object.local_path)
+        layer_version_info['priority'] = 0
+
+        return layer_version_info
+
+
+    def _get_git_branch(self, layer_path):
+        branch = subprocess.Popen("git symbolic-ref HEAD 2>/dev/null ", cwd=layer_path, shell=True, stdout=subprocess.PIPE).communicate()[0]
+        branch = branch.replace('refs/heads/', '').rstrip()
+        return branch
+
+    def _get_git_revision(self, layer_path):
+        revision = subprocess.Popen("git rev-parse HEAD 2>/dev/null ", cwd=layer_path, shell=True, stdout=subprocess.PIPE).communicate()[0].rstrip()
+        return revision
+
+
+    def _get_build_information(self):
+        build_info = {}
+        # Generate an identifier for each new build
+
+        build_info['machine'] = self.server.runCommand(["getVariable", "MACHINE"])[0]
+        build_info['distro'] = self.server.runCommand(["getVariable", "DISTRO"])[0]
+        build_info['distro_version'] = self.server.runCommand(["getVariable", "DISTRO_VERSION"])[0]
+        build_info['started_on'] = datetime.datetime.now()
+        build_info['completed_on'] = datetime.datetime.now()
+        build_info['image_fstypes'] = self._remove_redundant(self.server.runCommand(["getVariable", "IMAGE_FSTYPES"])[0] or "")
+        build_info['cooker_log_path'] = self.server.runCommand(["getVariable", "BB_CONSOLELOG"])[0]
+        build_info['build_name'] = self.server.runCommand(["getVariable", "BUILDNAME"])[0]
+        build_info['bitbake_version'] = self.server.runCommand(["getVariable", "BB_VERSION"])[0]
+
+        return build_info
+
+    def _get_task_information(self, event, recipe):
+
+
+        task_information = {}
+        task_information['build'] = self.internal_state['build']
+        task_information['outcome'] = Task.OUTCOME_NA
+        task_information['recipe'] = recipe
+        task_information['task_name'] = event.taskname
+        try:
+            # some tasks don't come with a hash. and that's ok
+            task_information['sstate_checksum'] = event.taskhash
+        except AttributeError:
+            pass
+        return task_information
+
+    def _get_layer_version_for_path(self, path):
+        def _slkey(layer_version):
+            return len(layer_version.layer.local_path)
+
+        # Heuristics: we always match recipe to the deepest layer path that
+        # we can match to the recipe file path
+        for bl in sorted(self.internal_state['layer_versions'], reverse=True, key=_slkey):
+            if (path.startswith(bl.layer.local_path)):
+                return bl
+
+        #TODO: if we get here, we didn't read layers correctly
+        assert False
+        return None
+
+    def _get_recipe_information_from_build_event(self, event):
+
+        layer_version_obj = self._get_layer_version_for_path(re.split(':', event.taskfile)[-1])
+
+        recipe_info = {}
+        recipe_info['layer_version'] = layer_version_obj
+        recipe_info['file_path'] = re.split(':', event.taskfile)[-1]
+
+        return recipe_info
+
+    def _get_task_build_stats(self, task_object):
+        bs_path = self._get_path_information(task_object)
+        for bp in bs_path:  # TODO: split for each target
+            task_build_stats = self._get_build_stats_from_file(bp, task_object.task_name)
+
+        return task_build_stats
+
+    def _get_path_information(self, task_object):
+        build_stats_format = "{tmpdir}/buildstats/{target}-{machine}/{buildname}/{package}/"
+        build_stats_path = []
+
+        for t in self.internal_state['targets']:
+            target = t.target
+            machine = self.internal_state['build'].machine
+            buildname = self.internal_state['build'].build_name
+            pe, pv = task_object.recipe.version.split(":",1)
+            if len(pe) > 0:
+                package = task_object.recipe.name + "-" + pe + "_" + pv
+            else:
+                package = task_object.recipe.name + "-" + pv
+
+            build_stats_path.append(build_stats_format.format(tmpdir=self.tmp_dir, target=target,
+                                                     machine=machine, buildname=buildname,
+                                                     package=package))
+
+        return build_stats_path
+
+    def _get_build_stats_from_file(self, bs_path, task_name):
+
+        task_bs_filename = str(bs_path) + str(task_name)
+        task_bs = open(task_bs_filename, 'r')
+
+        cpu_usage = 0
+        disk_io = 0
+        startio = ''
+        endio = ''
+
+        for line in task_bs.readlines():
+            if line.startswith('CPU usage: '):
+                cpu_usage = line[11:]
+            elif line.startswith('EndTimeIO: '):
+                endio = line[11:]
+            elif line.startswith('StartTimeIO: '):
+                startio = line[13:]
+
+        task_bs.close()
+
+        if startio and endio:
+            disk_io = int(endio.strip('\n ')) - int(startio.strip('\n '))
+
+        if cpu_usage:
+            cpu_usage = float(cpu_usage.strip('% \n'))
+
+        task_build_stats = {'cpu_usage': cpu_usage, 'disk_io': disk_io}
+
+        return task_build_stats
+
+    def _remove_redundant(self, string):
+        ret = []
+        for i in string.split():
+            if i not in ret:
+                ret.append(i)
+        return " ".join(ret)
+
+
+    ################################
+    ## external available methods to store information
+
+    def store_layer_info(self):
+        layers = self.server.runCommand(["getVariable", "BBLAYERS"])[0].strip().split(" ")
+        self.internal_state['layers'] = []
+        for layer_path in { l for l in layers if len(l) }:
+            layer_information = self._get_layer_dict(layer_path)
+            self.internal_state['layers'].append(self.orm_wrapper.get_update_layer_object(layer_information))
+
+    def store_started_build(self, event):
+
+        build_information = self._get_build_information()
+
+        build_obj = self.orm_wrapper.create_build_object(build_information)
+        self.internal_state['build'] = build_obj
+
+        # create target information
+        target_information = {}
+        target_information['targets'] = event.getPkgs()
+        target_information['build'] = build_obj
+
+        self.internal_state['targets'] = self.orm_wrapper.create_target_objects(target_information)
+
+        # Load layer information for the build
+        self.internal_state['layer_versions'] = []
+        for layer_object in self.internal_state['layers']:
+            layer_version_information = self._get_layer_version_information(layer_object)
+            self.internal_state['layer_versions'].append(self.orm_wrapper.get_layer_version_object(layer_version_information))
+
+        del self.internal_state['layers']
+        # Save build configuration
+        self.orm_wrapper.save_build_variables(build_obj, self.server.runCommand(["getAllKeysWithFlags", ["doc", "func"]])[0])
+
+
+    def update_build_information(self, event, errors, warnings, taskfailures):
+        if 'build' in self.internal_state:
+            self.orm_wrapper.update_build_object(self.internal_state['build'], errors, warnings, taskfailures)
+
+    def store_started_task(self, event):
+        identifier = event.taskfile + event.taskname
+
+        recipe_information = self._get_recipe_information_from_build_event(event)
+        recipe = self.orm_wrapper.get_update_recipe_object(recipe_information)
+
+        task_information = self._get_task_information(event, recipe)
+        task_information['outcome'] = Task.OUTCOME_NA
+
+        if isinstance(event, bb.runqueue.runQueueTaskSkipped):
+            task_information['task_executed'] = False
+            if event.reason == "covered":
+                task_information['outcome'] = Task.OUTCOME_COVERED
+            if event.reason == "existing":
+                task_information['outcome'] = Task.OUTCOME_EXISTING
+        else:
+            task_information['task_executed'] = True
+            if 'noexec' in vars(event) and event.noexec == True:
+                task_information['script_type'] = Task.CODING_NOEXEC
+
+        self.task_order += 1
+        task_information['order'] = self.task_order
+        task_obj = self.orm_wrapper.get_update_task_object(task_information)
+
+        self.internal_state[identifier] = {'start_time': datetime.datetime.now()}
+
+    def update_and_store_task(self, event):
+        identifier = event.taskfile + event.taskname
+        recipe_information = self._get_recipe_information_from_build_event(event)
+        recipe = self.orm_wrapper.get_update_recipe_object(recipe_information)
+        task_information = self._get_task_information(event,recipe)
+        try:
+            task_information['start_time'] = self.internal_state[identifier]['start_time']
+        except:
+            pass
+
+        if 'logfile' in vars(event):
+            task_information['logfile'] = event.logfile
+
+        if '_message' in vars(event):
+            task_information['message'] = event._message
+
+        if 'taskflags' in vars(event):
+            # with TaskStarted, we get even more information
+            if 'python' in event.taskflags.keys() and event.taskflags['python'] == '1':
+                task_information['script_type'] = Task.CODING_PYTHON
+            else:
+                task_information['script_type'] = Task.CODING_SHELL
+
+        if isinstance(event, (bb.runqueue.runQueueTaskCompleted, bb.runqueue.sceneQueueTaskCompleted)):
+            task_information['outcome'] = Task.OUTCOME_SUCCESS
+            task_build_stats = self._get_task_build_stats(self.orm_wrapper.get_update_task_object(task_information))
+            task_information['cpu_usage'] = task_build_stats['cpu_usage']
+            task_information['disk_io'] = task_build_stats['disk_io']
+            del self.internal_state[identifier]
+
+        if isinstance(event, (bb.runqueue.runQueueTaskFailed, bb.runCommand.sceneQueueTaskFailed)):
+            task_information['outcome'] = Task.OUTCOME_FAILED
+            del self.internal_state[identifier]
+
+        self.orm_wrapper.get_update_task_object(task_information)
+
+
+    def read_target_package_dep_data(self, event):
+        # for all targets
+        for target in self.internal_state['targets']:
+            # verify that we have something to read
+            if not target.is_image or not self.has_build_history:
+                print "not collecting package info ", target.is_image, self.has_build_history
+                break
+
+            # TODO this is a temporary replication of the code in buildhistory.bbclass
+            # This MUST be changed to query the actual BUILD_DIR_IMAGE in the target context when
+            # the capability will be implemented in Bitbake
+
+            MACHINE_ARCH, error = self.server.runCommand(['getVariable', 'MACHINE_ARCH'])
+            TCLIBC, error = self.server.runCommand(['getVariable', 'TCLIBC'])
+            BUILDHISTORY_DIR, error = self.server.runCommand(['getVariable', 'BUILDHISTORY_DIR'])
+            BUILDHISTORY_DIR_IMAGE = "%s/images/%s/%s/%s" % (BUILDHISTORY_DIR, MACHINE_ARCH, TCLIBC, target.target)
+
+            self.internal_state['packages'] = {}
+
+            with open("%s/installed-package-sizes.txt" % BUILDHISTORY_DIR_IMAGE, "r") as fin:
+                for line in fin:
+                    line = line.rstrip(";")
+                    psize, px = line.split("\t")
+                    punit, pname = px.split(" ")
+                    self.internal_state['packages'][pname.strip()] = {'size':int(psize)*1024, 'depends' : []}
+
+            with open("%s/depends.dot" % BUILDHISTORY_DIR_IMAGE, "r") as fin:
+                p = re.compile(r' -> ')
+                dot = re.compile(r'.*style=dotted')
+                for line in fin:
+                    line = line.rstrip(';')
+                    linesplit = p.split(line)
+                    if len(linesplit) == 2:
+                        pname = linesplit[0].rstrip('"').strip('"')
+                        dependsname = linesplit[1].split(" ")[0].strip().strip(";").strip('"').rstrip('"')
+                        deptype = Target_Package_Dependency.TYPE_DEPENDS
+                        if dot.match(line):
+                            deptype = Target_Package_Dependency.TYPE_RECOMMENDS
+                        if not pname in self.internal_state['packages']:
+                            self.internal_state['packages'][pname] = {'size': 0, 'depends' : []}
+                        if not dependsname in self.internal_state['packages']:
+                            self.internal_state['packages'][dependsname] = {'size': 0, 'depends' : []}
+                        self.internal_state['packages'][pname]['depends'].append((dependsname, deptype))
+
+            self.orm_wrapper.save_target_package_information(target,
+                        self.internal_state['packages'],
+                        self.internal_state['bldpkgs'], self.internal_state['recipes'])
+
+
+    def store_dependency_information(self, event):
+        # save layer version priorities
+        if 'layer-priorities' in event._depgraph.keys():
+            for lv in event._depgraph['layer-priorities']:
+                (name, path, regexp, priority) = lv
+                layer_version_obj = self._get_layer_version_for_path(path[1:]) # paths start with a ^
+                assert layer_version_obj is not None
+                layer_version_obj.priority = priority
+                layer_version_obj.save()
+
+        # save build time package information
+        self.internal_state['bldpkgs'] = {}
+        for pkg  in event._depgraph['packages']:
+            self.internal_state['bldpkgs'][pkg] = event._depgraph['packages'][pkg]
+
+        # save recipe information
+        self.internal_state['recipes'] = {}
+        for pn in event._depgraph['pn']:
+
+            file_name = re.split(':', event._depgraph['pn'][pn]['filename'])[-1]
+            layer_version_obj = self._get_layer_version_for_path(re.split(':', file_name)[-1])
+
+            assert layer_version_obj is not None
+
+            recipe_info = {}
+            recipe_info['name'] = pn
+            recipe_info['version'] = event._depgraph['pn'][pn]['version']
+            recipe_info['layer_version'] = layer_version_obj
+            recipe_info['summary'] = event._depgraph['pn'][pn]['summary']
+            recipe_info['license'] = event._depgraph['pn'][pn]['license']
+            recipe_info['description'] = event._depgraph['pn'][pn]['description']
+            recipe_info['section'] = event._depgraph['pn'][pn]['section']
+            recipe_info['licensing_info'] = 'Not Available'
+            recipe_info['homepage'] = event._depgraph['pn'][pn]['homepage']
+            recipe_info['bugtracker'] = event._depgraph['pn'][pn]['bugtracker']
+            recipe_info['file_path'] = file_name
+            recipe = self.orm_wrapper.get_update_recipe_object(recipe_info)
+            if 'inherits' in event._depgraph['pn'][pn].keys():
+                recipe.is_image = True in map(lambda x: x.endswith('image.bbclass'), event._depgraph['pn'][pn]['inherits'])
+            else:
+                recipe.is_image = False
+            if recipe.is_image:
+                for t in self.internal_state['targets']:
+                    if pn == t.target:
+                        t.is_image = True
+                        t.save()
+            self.internal_state['recipes'][pn] = recipe
+
+        # save recipe dependency
+        # buildtime
+        for recipe in event._depgraph['depends']:
+            try:
+                target = self.internal_state['recipes'][recipe]
+                for dep in event._depgraph['depends'][recipe]:
+                    dependency = self.internal_state['recipes'][dep]
+                    Recipe_Dependency.objects.get_or_create( recipe = target,
+                            depends_on = dependency, dep_type = Recipe_Dependency.TYPE_DEPENDS)
+            except KeyError:    # we'll not get recipes for key w/ values listed in ASSUME_PROVIDED
+                pass
+
+        # runtime
+        for recipe in event._depgraph['rdepends-pn']:
+            try:
+                target = self.internal_state['recipes'][recipe]
+                for dep in event._depgraph['rdepends-pn'][recipe]:
+                    dependency = self.internal_state['recipes'][dep]
+                    Recipe_Dependency.objects.get_or_create( recipe = target,
+                            depends_on = dependency, dep_type = Recipe_Dependency.TYPE_RDEPENDS)
+
+            except KeyError:    # we'll not get recipes for key w/ values listed in ASSUME_PROVIDED
+                pass
+
+        # save all task information
+        def _save_a_task(taskdesc):
+            spec = re.split(r'\.', taskdesc);
+            pn = ".".join(spec[0:-1])
+            taskname = spec[-1]
+            e = event
+            e.taskname = pn
+            recipe = self.internal_state['recipes'][pn]
+            task_info = self._get_task_information(e, recipe)
+            task_info['task_name'] = taskname
+            task_obj = self.orm_wrapper.get_update_task_object(task_info)
+            return task_obj
+
+        for taskdesc in event._depgraph['tdepends']:
+            target = _save_a_task(taskdesc)
+            for taskdesc1 in event._depgraph['tdepends'][taskdesc]:
+                dep = _save_a_task(taskdesc1)
+                Task_Dependency.objects.get_or_create( task = target, depends_on = dep )
+
+    def store_build_package_information(self, event):
+        package_info = event.data
+        self.orm_wrapper.save_build_package_information(self.internal_state['build'],
+                            package_info,
+                            self.internal_state['recipes'],
+                            )
+
+    def _store_log_information(self, level, text):
+        log_information = {}
+        log_information['build'] = self.internal_state['build']
+        log_information['level'] = level
+        log_information['message'] = text
+        self.orm_wrapper.create_logmessage(log_information)
+
+    def store_log_info(self, text):
+        self._store_log_information(LogMessage.INFO, text)
+
+    def store_log_warn(self, text):
+        self._store_log_information(LogMessage.WARNING, text)
+
+    def store_log_error(self, text):
+        self._store_log_information(LogMessage.ERROR, text)
+
+    def store_log_event(self, event):
+        # look up license files info from insane.bbclass
+        m = re.match("([^:]*): md5 checksum matched for ([^;]*)", event.msg)
+        if m:
+            (pn, fn) = m.groups()
+            self.internal_state['recipes'][pn].licensing_info = fn
+            self.internal_state['recipes'][pn].save()
+
+        if event.levelno < format.WARNING:
+            return
+        if not 'build' in self.internal_state:
+            return
+        log_information = {}
+        log_information['build'] = self.internal_state['build']
+        if event.levelno >= format.ERROR:
+            log_information['level'] = LogMessage.ERROR
+        elif event.levelno == format.WARNING:
+            log_information['level'] = LogMessage.WARNING
+        log_information['message'] = event.msg
+        log_information['pathname'] = event.pathname
+        log_information['lineno'] = event.lineno
+        self.orm_wrapper.create_logmessage(log_information)
+

bitbake/lib/bb/ui/crumbs/builder.py

@@ -31,6 +31,7 @@ import re
 import logging
 import sys
 import signal
+import time
 from bb.ui.crumbs.imageconfigurationpage import ImageConfigurationPage
 from bb.ui.crumbs.recipeselectionpage import RecipeSelectionPage
 from bb.ui.crumbs.packageselectionpage import PackageSelectionPage
@@ -197,7 +198,7 @@ class Configuration:
         handler.set_var_in_file("BBLAYERS", self.layers, "bblayers.conf")
         # local.conf
         if not defaults:
-            handler.set_var_in_file("MACHINE", self.curr_mach, "local.conf")
+            handler.early_assign_var_in_file("MACHINE", self.curr_mach, "local.conf")
         handler.set_var_in_file("DISTRO", self.curr_distro, "local.conf")
         handler.set_var_in_file("DL_DIR", self.dldir, "local.conf")
         handler.set_var_in_file("SSTATE_DIR", self.sstatedir, "local.conf")
@@ -217,7 +218,7 @@ class Configuration:
         handler.set_var_in_file("SDKMACHINE", self.curr_sdk_machine, "local.conf")
         handler.set_var_in_file("CONF_VERSION", self.conf_version, "local.conf")
         handler.set_var_in_file("LCONF_VERSION", self.lconf_version, "bblayers.conf")
-        handler.set_var_in_file("EXTRA_SETTING", self.extra_setting, "local.conf")
+        handler.set_extra_config(self.extra_setting)
         handler.set_var_in_file("TOOLCHAIN_BUILD", self.toolchain_build, "local.conf")
         handler.set_var_in_file("IMAGE_FSTYPES", self.image_fstypes, "local.conf")
         if not defaults:
@@ -1466,3 +1467,10 @@ class Builder(gtk.Window):
 
     def get_topdir(self):
         return self.handler.get_topdir()
+
+    def wait(self, delay):
+        time_start = time.time()
+        time_end = time_start + delay
+        while time_end > time.time():
+            while gtk.events_pending():
+                gtk.main_iteration()

bitbake/lib/bb/ui/crumbs/hig/advancedsettingsdialog.py

@@ -234,7 +234,10 @@ class AdvancedSettingsDialog (CrumbsDialog, SettingsUIHelper):
             article = ""
             if image_type.startswith(("a", "e", "i", "o", "u")):
                 article = "n"
-            self.image_types_checkbuttons[image_type].set_tooltip_text("Build a%s %s image" % (article, image_type))
+            if image_type == "live":
+                self.image_types_checkbuttons[image_type].set_tooltip_text("Build iso and hddimg images")
+            else:
+                self.image_types_checkbuttons[image_type].set_tooltip_text("Build a%s %s image" % (article, image_type))
             table.attach(self.image_types_checkbuttons[image_type], j - 1, j + 3, i, i + 1)
             if image_type in self.configuration.image_fstypes.split():
                 self.image_types_checkbuttons[image_type].set_active(True)

bitbake/lib/bb/ui/crumbs/hig/saveimagedialog.py

@@ -70,9 +70,11 @@ class SaveImageDialog (CrumbsDialog):
         sub_label.set_alignment(0, 0)
         sub_label.set_markup("The description should be less than 150 characters long.")
         self.description_entry = gtk.TextView()
-        self.description_entry.get_buffer().set_text(self.description_field)
+        description_buffer = self.description_entry.get_buffer()
+        description_buffer.set_text(self.description_field)
+        description_buffer.connect("insert-text", self.limit_description_length)
         self.description_entry.set_wrap_mode(gtk.WRAP_WORD)
-        self.description_entry.set_size_request(350,150)
+        self.description_entry.set_size_request(350,50)
         sub_vbox.pack_start(label, expand=False, fill=False)
         sub_vbox.pack_start(sub_label, expand=False, fill=False)
         sub_vbox.pack_start(self.description_entry, expand=False, fill=False)
@@ -108,6 +110,13 @@ class SaveImageDialog (CrumbsDialog):
 
         self.show_all()
 
+    def limit_description_length(self, textbuffer, iter, text, length):
+        buffer_bounds = textbuffer.get_bounds()
+        entire_text = textbuffer.get_text(*buffer_bounds)
+        entire_text += text
+        if len(entire_text)>150 or text=="\n":
+            textbuffer.emit_stop_by_name("insert-text")
+
     def name_entry_changed(self, entry):
         text = entry.get_text()
         if text == '':

bitbake/lib/bb/ui/crumbs/hobeventhandler.py

@@ -21,6 +21,7 @@
 
 import gobject
 import logging
+import ast
 from bb.ui.crumbs.runningbuild import RunningBuild
 
 class HobHandler(gobject.GObject):
@@ -315,7 +316,7 @@ class HobHandler(gobject.GObject):
 
     def set_machine(self, machine):
         if machine:
-            self.set_var_in_file("MACHINE", machine, "local.conf")
+            self.early_assign_var_in_file("MACHINE", machine, "local.conf")
 
     def set_sdk_machine(self, sdk_machine):
         self.set_var_in_file("SDKMACHINE", sdk_machine, "local.conf")
@@ -357,7 +358,20 @@ class HobHandler(gobject.GObject):
     def set_incompatible_license(self, incompat_license):
         self.set_var_in_file("INCOMPATIBLE_LICENSE", incompat_license, "local.conf")
 
+    def set_extra_setting(self, extra_setting):
+        self.set_var_in_file("EXTRA_SETTING", extra_setting, "local.conf")
+
     def set_extra_config(self, extra_setting):
+        old_extra_setting = ast.literal_eval(self.runCommand(["getVariable", "EXTRA_SETTING"]) or "{}")
+        if extra_setting:
+            self.set_var_in_file("EXTRA_SETTING", extra_setting, "local.conf")
+        else:
+            self.remove_var_from_file("EXTRA_SETTING")
+
+        #remove not needed settings from conf
+        for key in old_extra_setting:
+            if key not in extra_setting:
+                self.remove_var_from_file(key)
         for key in extra_setting.keys():
             value = extra_setting[key]
             self.set_var_in_file(key, value, "local.conf")
@@ -440,12 +454,12 @@ class HobHandler(gobject.GObject):
         self.building = False
 
     def cancel_parse(self):
-        self.runCommand(["stateStop"])
+        self.runCommand(["stateForceShutdown"])
 
     def cancel_build(self, force=False):
         if force:
             # Force the cooker to stop as quickly as possible
-            self.runCommand(["stateStop"])
+            self.runCommand(["stateForceShutdown"])
         else:
             # Wait for tasks to complete before shutting down, this helps
             # leave the workdir in a usable state
@@ -472,6 +486,14 @@ class HobHandler(gobject.GObject):
         self.server.runCommand(["setVarFile", var, val, default_file, "set"])
         self.runCommand(["disableDataTracking"])
 
+    def early_assign_var_in_file(self, var, val, default_file=None):
+        self.runCommand(["enableDataTracking"])
+        self.server.runCommand(["setVarFile", var, val, default_file, "earlyAssign"])
+        self.runCommand(["disableDataTracking"])
+
+    def remove_var_from_file(self, var):
+        self.server.runCommand(["removeVarFile", var])
+
     def append_var_in_file(self, var, val, default_file=None):
         self.server.runCommand(["setVarFile", var, val, default_file, "append"])
 

bitbake/lib/bb/ui/crumbs/hoblistmodel.py

@@ -690,7 +690,7 @@ class RecipeListModel(gtk.ListStore):
             inherits = event_model["pn"][item]["inherits"]
             summary = event_model["pn"][item]["summary"]
             version = event_model["pn"][item]["version"]
-            revision = event_model["pn"][item]["revision"]
+            revision = event_model["pn"][item]["prevision"]
             homepage = event_model["pn"][item]["homepage"]
             bugtracker = event_model["pn"][item]["bugtracker"]
             filename = event_model["pn"][item]["filename"]

bitbake/lib/bb/ui/crumbs/imageconfigurationpage.py

@@ -300,7 +300,12 @@ class ImageConfigurationPage (HobPage):
     def view_warnings_button_clicked_cb(self, button):
         self.builder.show_warning_dialog()
 
+    def machine_combo_changed_idle_cb(self):
+        self.builder.window.set_cursor(None)
+
     def machine_combo_changed_cb(self, machine_combo):
+        self.builder.window.set_cursor(gtk.gdk.Cursor(gtk.gdk.WATCH))
+        self.builder.wait(0.1) #wait for combo and cursor to update
         self.stopping = False
         self.builder.parsing_warnings = []
         combo_item = machine_combo.get_active_text()
@@ -324,6 +329,8 @@ class ImageConfigurationPage (HobPage):
         # Do reparse recipes
         self.builder.populate_recipe_package_info_async()
 
+        glib.idle_add(self.machine_combo_changed_idle_cb)
+
     def update_machine_combo(self):
         self.disable_warnings_bar()
         all_machines = [self.__dummy_machine__] + self.builder.parameters.all_machines
@@ -387,6 +394,10 @@ class ImageConfigurationPage (HobPage):
                 selected_image = self.__dummy_image__
                 self.update_image_combo(self.builder.recipe_model, None)
             dialog.destroy()
+        else:
+            if self.custom_image_selected:
+                self.custom_image_selected = None
+                self.update_image_combo(self.builder.recipe_model, selected_image)
 
         if not selected_image or (selected_image == self.__dummy_image__):
             self.builder.window_sensitive(True)
@@ -523,7 +534,10 @@ class ImageConfigurationPage (HobPage):
         if not response:
             return
         if settings_changed:
+            self.builder.window.set_cursor(gtk.gdk.Cursor(gtk.gdk.WATCH))
+            self.builder.wait(0.1) #wait for adv_settings_dialog to terminate
             self.builder.reparse_post_adv_settings()
+            self.builder.window.set_cursor(None)
 
     def just_bake_button_clicked_cb(self, button):
         self.builder.parsing_warnings = []

bitbake/lib/bb/ui/crumbs/imagedetailspage.py

@@ -635,6 +635,14 @@ class ImageDetailsPage (HobPage):
         images_dir = topdir + "/recipes/images/"
         self.builder.ensure_dir(images_dir)
 
+        self.name_field_template = self.builder.image_configuration_page.custom_image_selected
+        if self.name_field_template:
+            image_path = self.builder.recipe_model.pn_path[self.name_field_template]
+            image_iter = self.builder.recipe_model.get_iter(image_path)
+            self.description_field_template = self.builder.recipe_model.get_value(image_iter, self.builder.recipe_model.COL_DESC)
+        else:
+            self.name_field_template = ""
+
         dialog = SaveImageDialog(images_dir, self.name_field_template, self.description_field_template,
                  "Save image recipe", self.builder, gtk.DIALOG_MODAL | gtk.DIALOG_DESTROY_WITH_PARENT)
         response = dialog.run()

bitbake/lib/bb/ui/depexp.py

@@ -314,7 +314,7 @@ def main(server, eventHandler, params):
                 break
             if shutdown == 1:
                 print("\nSecond Keyboard Interrupt, stopping...\n")
-                _, error = server.runCommand(["stateStop"])
+                _, error = server.runCommand(["stateForceShutdown"])
                 if error:
                     print('Unable to cleanly stop: %s' % error)
             if shutdown == 0:

bitbake/lib/bb/ui/goggle.py

@@ -117,5 +117,5 @@ def main (server, eventHandler, params):
     except KeyboardInterrupt:
         pass
     finally:
-        server.runCommand(["stateStop"])
+        server.runCommand(["stateForceShutdown"])
 

bitbake/lib/bb/ui/hob.py

@@ -46,7 +46,7 @@ from bb.ui.crumbs.hoblistmodel import RecipeListModel, PackageListModel
 from bb.ui.crumbs.hobeventhandler import HobHandler
 from bb.ui.crumbs.builder import Builder
 
-extraCaches = ['bb.cache_extra:HobRecipeInfo']
+featureSet = [bb.cooker.CookerFeatures.HOB_EXTRA_CACHES]
 
 def event_handle_idle_func(eventHandler, hobHandler):
     # Consume as many messages as we can in the time available to us

bitbake/lib/bb/ui/knotty.py

@@ -306,8 +306,10 @@ def main(server, eventHandler, params, tf = TerminalFilter):
 
     while True:
         try:
-            termfilter.updateFooter()
-            event = eventHandler.waitEvent(0.25)
+            event = eventHandler.waitEvent(0)
+            if event is None:
+                termfilter.updateFooter()
+                event = eventHandler.waitEvent(0.25)
             if event is None:
                 if main.shutdown > 1:
                     break
@@ -403,8 +405,9 @@ def main(server, eventHandler, params, tf = TerminalFilter):
 
             if isinstance(event, bb.command.CommandFailed):
                 return_value = event.exitcode
-                errors = errors + 1
-                logger.error("Command execution failed: %s", event.error)
+                if event.error:
+                    errors = errors + 1
+                    logger.error("Command execution failed: %s", event.error)
                 main.shutdown = 2
                 continue
             if isinstance(event, bb.command.CommandExit):
@@ -469,8 +472,12 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                              event.taskid, event.taskstring, event.exitcode)
                 continue
 
+            if isinstance(event, bb.event.DepTreeGenerated):
+                continue
+
             # ignore
             if isinstance(event, (bb.event.BuildBase,
+                                  bb.event.MetadataEvent,
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
                                   bb.event.RecipeParsed,
@@ -497,7 +504,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                 main.shutdown = 2
             if not params.observe_only and main.shutdown == 1:
                 print("\nSecond Keyboard Interrupt, stopping...\n")
-                _, error = server.runCommand(["stateStop"])
+                _, error = server.runCommand(["stateForceShutdown"])
                 if error:
                     logger.error("Unable to cleanly stop: %s" % error)
             if not params.observe_only and main.shutdown == 0:
@@ -518,7 +525,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     if warnings:
         summary += pluralise("\nSummary: There was %s WARNING message shown.",
                              "\nSummary: There were %s WARNING messages shown.", warnings)
-    if return_value:
+    if return_value and errors:
         summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
                              "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
     if summary:

bitbake/lib/bb/ui/ncurses.py

@@ -350,7 +350,7 @@ class NCursesUI:
                     exitflag = True
                 if shutdown == 1:
                     mw.appendText("Second Keyboard Interrupt, stopping...\n")
-                    _, error = server.runCommand(["stateStop"])
+                    _, error = server.runCommand(["stateForceShutdown"])
                     if error:
                         print("Unable to cleanly stop: %s" % error)
                 if shutdown == 0:

bitbake/lib/bb/ui/toasterui.py

@@ -0,0 +1,272 @@
+#
+# BitBake ToasterUI Implementation
+# based on (No)TTY UI Implementation by Richard Purdie
+#
+# Handling output to TTYs or files (no TTY)
+#
+# Copyright (C) 2006-2012 Richard Purdie
+# Copyright (C) 2013      Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from __future__ import division
+try:
+    import bb
+except RuntimeError as exc:
+    sys.exit(str(exc))
+
+from bb.ui import uihelper
+from bb.ui.buildinfohelper import BuildInfoHelper
+
+import bb.msg
+import copy
+import fcntl
+import logging
+import os
+import progressbar
+import signal
+import struct
+import sys
+import time
+import xmlrpclib
+
+featureSet = [bb.cooker.CookerFeatures.HOB_EXTRA_CACHES, bb.cooker.CookerFeatures.SEND_DEPENDS_TREE, bb.cooker.CookerFeatures.BASEDATASTORE_TRACKING]
+
+logger = logging.getLogger("BitBake")
+interactive = sys.stdout.isatty()
+
+
+
+def _log_settings_from_server(server):
+    # Get values of variables which control our output
+    includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
+    if error:
+        logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
+        raise BaseException(error)
+    loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
+    if error:
+        logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
+        raise BaseException(error)
+    return includelogs, loglines
+
+def main(server, eventHandler, params ):
+
+    includelogs, loglines = _log_settings_from_server(server)
+
+    # verify and warn
+    build_history_enabled = True
+    inheritlist, error = server.runCommand(["getVariable", "INHERIT"])
+    if not "buildhistory" in inheritlist.split(" "):
+        logger.warn("buildhistory is not enabled. Please enable INHERIT += \"buildhistory\" to see image details.")
+        build_history_enabled = False
+
+    helper = uihelper.BBUIHelper()
+
+    console = logging.StreamHandler(sys.stdout)
+    format_str = "%(levelname)s: %(message)s"
+    format = bb.msg.BBLogFormatter(format_str)
+    bb.msg.addDefaultlogFilter(console)
+    console.setFormatter(format)
+    logger.addHandler(console)
+
+    if not params.observe_only:
+        logger.error("ToasterUI can only work in observer mode")
+        return
+
+
+    main.shutdown = 0
+    interrupted = False
+    return_value = 0
+    errors = 0
+    warnings = 0
+    taskfailures = []
+
+    buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+    buildinfohelper.store_layer_info()
+
+
+    while True:
+        try:
+            event = eventHandler.waitEvent(0.25)
+
+            if event is None:
+                if main.shutdown > 0:
+                    break
+                continue
+
+            helper.eventHandler(event)
+
+            if isinstance(event, bb.event.BuildStarted):
+                buildinfohelper.store_started_build(event)
+
+            if isinstance(event, (bb.build.TaskStarted, bb.build.TaskSucceeded, bb.build.TaskFailedSilent)):
+                buildinfohelper.update_and_store_task(event)
+                continue
+
+            if isinstance(event, bb.event.LogExecTTY):
+                logger.warn(event.msg)
+                continue
+
+            if isinstance(event, logging.LogRecord):
+                buildinfohelper.store_log_event(event)
+                if event.levelno >= format.ERROR:
+                    errors = errors + 1
+                    return_value = 1
+                elif event.levelno == format.WARNING:
+                    warnings = warnings + 1
+                # For "normal" logging conditions, don't show note logs from tasks
+                # but do show them if the user has changed the default log level to
+                # include verbose/debug messages
+                if event.taskpid != 0 and event.levelno <= format.NOTE:
+                    continue
+
+                logger.handle(event)
+                continue
+
+            if isinstance(event, bb.build.TaskFailed):
+                buildinfohelper.update_and_store_task(event)
+                return_value = 1
+                logfile = event.logfile
+                if logfile and os.path.exists(logfile):
+                    bb.error("Logfile of failure stored in: %s" % logfile)
+                continue
+
+            # these events are unprocessed now, but may be used in the future to log
+            # timing and error informations from the parsing phase in Toaster
+            if isinstance(event, bb.event.ParseStarted):
+                continue
+            if isinstance(event, bb.event.ParseProgress):
+                continue
+            if isinstance(event, bb.event.ParseCompleted):
+                continue
+            if isinstance(event, bb.event.CacheLoadStarted):
+                continue
+            if isinstance(event, bb.event.CacheLoadProgress):
+                continue
+            if isinstance(event, bb.event.CacheLoadCompleted):
+                continue
+            if isinstance(event, bb.event.MultipleProviders):
+                continue
+            if isinstance(event, bb.event.NoProvider):
+                return_value = 1
+                errors = errors + 1
+                if event._runtime:
+                    r = "R"
+                else:
+                    r = ""
+
+                if event._dependees:
+                    text = "Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)" % (r, event._item, ", ".join(event._dependees), r)
+                else:
+                    text = "Nothing %sPROVIDES '%s'" % (r, event._item)
+
+                logger.error(text)
+                if event._reasons:
+                    for reason in event._reasons:
+                        logger.error("%s", reason)
+                        text += reason
+                buildinfohelper.store_log_error(text)
+                continue
+
+            if isinstance(event, bb.event.ConfigParsed):
+                continue
+            if isinstance(event, bb.event.RecipeParsed):
+                continue
+
+            # end of saved events
+
+            if isinstance(event, (bb.runqueue.sceneQueueTaskStarted, bb.runqueue.runQueueTaskStarted, bb.runqueue.runQueueTaskSkipped)):
+                buildinfohelper.store_started_task(event)
+                continue
+
+            if isinstance(event, bb.runqueue.runQueueTaskCompleted):
+                buildinfohelper.update_and_store_task(event)
+                continue
+
+            if isinstance(event, bb.runqueue.runQueueTaskFailed):
+                buildinfohelper.update_and_store_task(event)
+                taskfailures.append(event.taskstring)
+                logger.error("Task %s (%s) failed with exit code '%s'",
+                             event.taskid, event.taskstring, event.exitcode)
+                continue
+
+            if isinstance(event, (bb.runqueue.sceneQueueTaskCompleted, bb.runqueue.sceneQueueTaskFailed)):
+                buildinfohelper.update_and_store_task(event)
+                continue
+
+
+            if isinstance(event, (bb.event.TreeDataPreparationStarted, bb.event.TreeDataPreparationCompleted)):
+                continue
+
+            if isinstance(event, (bb.event.BuildCompleted)):
+                buildinfohelper.read_target_package_dep_data(event)
+                buildinfohelper.update_build_information(event, errors, warnings, taskfailures)
+                continue
+
+            if isinstance(event, (bb.command.CommandCompleted,
+                                  bb.command.CommandFailed,
+                                  bb.command.CommandExit)):
+
+                buildinfohelper.update_build_information(event, errors, warnings, taskfailures)
+
+                # we start a new build info
+                errors = 0
+                warnings = 0
+                taskfailures = []
+                buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+                buildinfohelper.store_layer_info()
+                continue
+
+            if isinstance(event, bb.event.MetadataEvent):
+                if event.type == "SinglePackageInfo":
+                    buildinfohelper.store_build_package_information(event)
+                continue
+
+            # ignore
+            if isinstance(event, (bb.event.BuildBase,
+                                  bb.event.StampUpdate,
+                                  bb.event.RecipePreFinalise,
+                                  bb.runqueue.runQueueEvent,
+                                  bb.runqueue.runQueueExitWait,
+                                  bb.event.OperationProgress,
+                                  bb.command.CommandFailed,
+                                  bb.command.CommandExit,
+                                  bb.command.CommandCompleted,
+                                  bb.cooker.CookerExit)):
+                continue
+
+            if isinstance(event, bb.event.DepTreeGenerated):
+                buildinfohelper.store_dependency_information(event)
+                continue
+
+            logger.error("Unknown event: %s", event)
+
+        except EnvironmentError as ioerror:
+            # ignore interrupted io
+            if ioerror.args[0] == 4:
+                pass
+        except KeyboardInterrupt:
+            main.shutdown = 1
+            pass
+        except Exception as e:
+            logger.error(e)
+            import traceback
+            traceback.print_exc()
+            pass
+
+    if interrupted:
+        if return_value == 0:
+            return_value = 1
+
+    return return_value

bitbake/lib/bb/ui/uievent.py

@@ -28,7 +28,7 @@ import socket, threading, pickle
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 
 class BBUIEventQueue:
-    def __init__(self, BBServer, clientinfo=("localhost, 0")):
+    def __init__(self, BBServer, clientinfo=("localhost, 0"), featureset=[]):
 
         self.eventQueue = []
         self.eventQueueLock = threading.Lock()
@@ -44,7 +44,10 @@ class BBUIEventQueue:
         server.register_function( self.send_event, "event.sendpickle" )
         server.socket.settimeout(1)
 
-        self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
+        self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port, featureset)
+
+        if (self.EventHandle == None):
+            bb.fatal("Could not register UI event handler")
 
         self.server = server
 

bitbake/lib/bb/utils.py

@@ -532,6 +532,8 @@ def approved_variables():
     Determine and return the list of whitelisted variables which are approved
     to remain in the envrionment.
     """
+    if 'BB_PRESERVE_ENV' in os.environ:
+        return os.environ.keys()
     approved = []
     if 'BB_ENV_WHITELIST' in os.environ:
         approved = os.environ['BB_ENV_WHITELIST'].split()
@@ -722,7 +724,7 @@ def copyfile(src, dest, newmtime = None, sstat = None):
         if not sstat:
             sstat = os.lstat(src)
     except Exception as e:
-        print("copyfile: Stating source file failed...", e)
+        logger.warn("copyfile: stat of %s failed (%s)" % (src, e))
         return False
 
     destexists = 1
@@ -749,7 +751,7 @@ def copyfile(src, dest, newmtime = None, sstat = None):
             #os.lchown(dest,sstat[stat.ST_UID],sstat[stat.ST_GID])
             return os.lstat(dest)
         except Exception as e:
-            print("copyfile: failed to properly create symlink:", dest, "->", target, e)
+            logger.warn("copyfile: failed to create symlink %s to %s (%s)" % (dest, target, e))
             return False
 
     if stat.S_ISREG(sstat[stat.ST_MODE]):
@@ -764,7 +766,7 @@ def copyfile(src, dest, newmtime = None, sstat = None):
             shutil.copyfile(src, dest + "#new")
             os.rename(dest + "#new", dest)
         except Exception as e:
-            print('copyfile: copy', src, '->', dest, 'failed.', e)
+            logger.warn("copyfile: copy %s to %s failed (%s)" % (src, dest, e))
             return False
         finally:
             if srcchown:
@@ -775,13 +777,13 @@ def copyfile(src, dest, newmtime = None, sstat = None):
         #we don't yet handle special, so we need to fall back to /bin/mv
         a = getstatusoutput("/bin/cp -f " + "'" + src + "' '" + dest + "'")
         if a[0] != 0:
-            print("copyfile: Failed to copy special file:" + src + "' to '" + dest + "'", a)
+            logger.warn("copyfile: failed to copy special file %s to %s (%s)" % (src, dest, a))
             return False # failure
     try:
         os.lchown(dest, sstat[stat.ST_UID], sstat[stat.ST_GID])
         os.chmod(dest, stat.S_IMODE(sstat[stat.ST_MODE])) # Sticky is reset on chown
     except Exception as e:
-        print("copyfile: Failed to chown/chmod/unlink", dest, e)
+        logger.warn("copyfile: failed to chown/chmod %s (%s)" % (dest, e))
         return False
 
     if newmtime:

bitbake/lib/prserv/db.py

@@ -43,6 +43,10 @@ class PRTable(object):
                     continue
                 raise exc
 
+    def sync(self):
+        self.conn.commit()
+        self._execute("BEGIN EXCLUSIVE TRANSACTION")
+
     def _getValueHist(self, version, pkgarch, checksum):
         data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
                            (version, pkgarch, checksum))

bitbake/lib/prserv/serv.py

@@ -74,7 +74,7 @@ class PRServer(SimpleXMLRPCServer):
             except:
                 self.handle_error(request, client_address)
                 self.shutdown_request(request)
-
+                self.table.sync()
 
     def process_request(self, request, client_address):
         self.requestqueue.put((request, client_address))
@@ -120,6 +120,7 @@ class PRServer(SimpleXMLRPCServer):
         while not self.quit:
             self.handle_request()
 
+        self.table.sync()
         logger.info("PRServer: stopping...")
         self.server_close()
         return

bitbake/lib/toaster/bldviewer/api.py

@@ -0,0 +1,37 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from django.conf.urls import patterns, include, url
+
+
+urlpatterns = patterns('bldviewer.views',
+        url(r'^builds$', 'model_explorer',  {'model_name':'build'}, name='builds'),
+        url(r'^targets$', 'model_explorer',  {'model_name':'target'}, name='targets'),
+        url(r'^tasks$', 'model_explorer', {'model_name':'task'}, name='task'),
+        url(r'^task_dependencies$', 'model_explorer',  {'model_name':'task_dependency'}, name='task_dependencies'),
+        url(r'^packages$', 'model_explorer',  {'model_name':'build_package'}, name='build_packages'),
+        url(r'^package_dependencies$', 'model_explorer',  {'model_name':'build_package_dependency'}, name='build_package_dependencies'),
+        url(r'^target_packages$', 'model_explorer',  {'model_name':'target_package'}, name='target_packages'),
+        url(r'^package_files$', 'model_explorer',  {'model_name':'build_file'}, name='build_files'),
+        url(r'^layers$', 'model_explorer', {'model_name':'layer'}, name='layer'),
+        url(r'^layerversions$', 'model_explorer', {'model_name':'layerversion'}, name='layerversion'),
+        url(r'^recipes$', 'model_explorer', {'model_name':'recipe'}, name='recipe'),
+        url(r'^recipe_dependencies$', 'model_explorer',  {'model_name':'recipe_dependency'}, name='recipe_dependencies'),
+        url(r'^variables$', 'model_explorer',  {'model_name':'variable'}, name='variables'),
+        url(r'^logmessages$', 'model_explorer',  {'model_name':'logmessage'}, name='logmessages'),
+)

bitbake/lib/toaster/bldviewer/templates/base.html

@@ -0,0 +1,30 @@
+<!DOCTYPE html>
+{% load static %}
+<html>
+	<head>
+		<title>Toaster Simple Explorer</title>
+<script src="{% static 'js/jquery-2.0.3.js' %}">
+</script>
+<script src="{% static 'js/bootstrap.js' %}">
+</script>
+<link href="{% static 'css/bootstrap.css' %}" rel="stylesheet" type="text/css">
+	</head>
+
+<body style="height: 100%">
+<div style="width:100%; height: 100%; position:absolute">
+<div style="width: 100%; height: 3em" class="nav">
+    <ul class="nav nav-tabs">
+        <li><a href="{% url all-builds %}">All Builds</a></li>
+        <li><a href="{% url all-layers %}">All Layers</a></li>
+    </ul>
+</div>
+
+<div style="overflow-y:scroll; width: 100%; position: absolute; top: 3em; bottom:70px ">
+{% block pagecontent %}
+{% endblock %}
+</div>
+<div class="navbar" style="position: absolute; bottom: 0; width:100%"><br/>About Toaster | Yocto Project </div>
+</div>
+</body>
+</html>
+

bitbake/lib/toaster/bldviewer/templates/basebuildpage.html

@@ -0,0 +1,17 @@
+{% extends "basetable.html" %}
+
+{% block pagename %}
+<ul class="nav nav-tabs" style="display: inline-block">
+  <li><a>Build {{build.target_set.all|join:"&nbsp;"}} at {{build.started_on}} : </a></li>
+  <li><a href="{% url task build.id %}"> Tasks </a></li>
+  <li><a href="{% url bpackage build.id %}"> Build Packages </a></li>
+    {% for t in build.target_set.all %}
+        {% if t.is_image %}
+  <li><a href="{% url tpackage build.id t.pk %}"> Packages for {{t.target}} </a> </li>
+        {% endif %}
+    {% endfor %}
+  <li><a href="{% url configuration build.id %}"> Configuration </a> </li>
+</ul>
+     <h1>Toaster - Build {% block pagetitle %} {% endblock %}</h1>
+{% endblock %}
+

bitbake/lib/toaster/bldviewer/templates/basetable.html

@@ -0,0 +1,46 @@
+{% extends "base.html" %}
+
+{% block pagecontent %}
+<script>
+function showhideTableColumn(i, sh) {
+    if (sh)
+        $('td:nth-child('+i+'),th:nth-child('+i+')').show();
+    else
+        $('td:nth-child('+i+'),th:nth-child('+i+')').hide();
+}
+
+
+function filterTableRows(test) {
+    if (test.length > 0) {
+        var r = test.split(/[ ,]+/).map(function (e) { return new RegExp(e, 'i') });
+        $('tr.data').map( function (i, el) {
+            (! r.map(function (j) { return j.test($(el).html())}).reduce(function (c, p) { return c && p;} )) ? $(el).hide() : $(el).show();
+        });
+    } else
+    {
+        $('tr.data').show();
+    }
+}
+</script>
+<div style="margin-bottom: 0.5em">
+
+    {% block pagename %}
+    {% endblock %}
+    <div align="left" style="display:inline-block; width: 40%; margin-left: 2em"> Search: <input type="search" id="filterstring" style="width: 80%" onkeyup="filterTableRows($('#filterstring').val())" autocomplete="off">
+    </div>
+    {% if hideshowcols %}
+    <div align="right" style="display: inline-block; width: 40%">Show/Hide columns:
+    {% for i in hideshowcols %}
+        <span>{{i.name}} <input type="checkbox" id="ct{{i.name}}" onchange="showhideTableColumn({{i.order}}, $('#ct{{i.name}}').is(':checked'))" checked autocomplete="off"></span> |
+    {% endfor %}
+    </div>
+    {% endif %}
+</div>
+<div>
+    <table class="table table-striped table-condensed" style="width:95%">
+{% block pagetable %}
+{% endblock %}
+    </table>
+</div>
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templates/bfile.html

@@ -0,0 +1,24 @@
+{% extends "basebuildpage.html" %}
+
+{% block pagetitle %}Files for package {{files.0.bpackage.name}} {% endblock %}
+{% block pagetable %}
+    {% if not files %}
+        <p>No files were recorded for this package!</p>
+    {% else %}
+
+            <tr>
+            <th>Name</th>
+            <th>Size (Bytes)</th>
+            </tr>
+
+            {% for file in files %}
+
+            <tr class="data">
+            <td>{{file.path}}</td>
+            <td>{{file.size}}</td>
+
+            {% endfor %}
+
+    {% endif %}
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templates/bpackage.html

@@ -0,0 +1,44 @@
+{% extends "basebuildpage.html" %}
+
+{% block pagetitle %}Packages{% endblock %}
+{% block pagetable %}
+    {% if not packages %}
+        <p>No packages were recorded for this target!</p>
+    {% else %}
+
+            <tr>
+            <th>Name</th>
+            <th>Version</th>
+            <th>Recipe</th>
+            <th>Summary</th>
+            <th>Section</th>
+            <th>Description</th>
+            <th>Size on host disk (Bytes)</th>
+            <th>License</th>
+            <th>Dependencies List (all)</th>
+            </tr>
+
+            {% for package in packages %}
+
+            <tr class="data">
+                <td><a name="#{{package.name}}" href="{% url bfile build.pk package.pk %}">{{package.name}} ({{package.filelist_bpackage.count}} files)</a></td>
+                <td>{{package.version}}-{{package.revision}}</td>
+                <td><a href="{% url layer_versions_recipes package.recipe.layer_version_id %}#{{package.recipe.name}}">{{package.recipe.name}}</a>{{package.package_name}}</a></td>
+
+            <td>{{package.summary}}</td>
+            <td>{{package.section}}</td>
+            <td>{{package.description}}</td>
+            <td>{{package.size}}</td>
+            <td>{{package.license}}</td>
+            <td>
+        <div style="height: 3em; overflow:auto">
+            {% for bpd in package.bpackage_dependencies_package.all %}
+                {{bpd.dep_type}}: {{bpd.depends_on}} <br/>
+            {% endfor %}
+        </div>
+            </td>
+            {% endfor %}
+
+    {% endif %}
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templates/build.html

@@ -0,0 +1,43 @@
+{% extends "basetable.html" %}
+
+{% block pagename %}
+    <h1>Toaster - Builds</h1>
+{% endblock %}
+
+{% block pagetable %}
+
+    {% load projecttags %}
+        <tr>
+            <th>Outcome</th>
+            <th>Started On</th>
+            <th>Completed On</th>
+            <th>Target</th>
+            <th>Machine</th>
+            <th>Time</th>
+            <th>Errors</th>
+            <th>Warnings</th>
+            <th>Output</th>
+            <th>Log</th>
+            <th>Bitbake Version</th>
+            <th>Build Name</th>
+        </tr>
+        {% for build in builds %}
+        <tr class="data">
+            <td><a href="{% url configuration build.id %}">{{build.get_outcome_display}}</a></td>
+            <td>{{build.started_on}}</td>
+            <td>{{build.completed_on}}</td>
+            <td>{% for t in build.target_set.all %}<a href="{% url tpackage build.id t.id %}">{{t.target}}</a>{% if t.is_image %} (Img){% endif %}<br/>{% endfor %}</td>
+            <td>{{build.machine}}</td>
+            <td>{% time_difference build.started_on build.completed_on %}</td>
+            <td>{{build.errors_no}}:{% if  build.errors_no %}{% for error in logs %}{% if error.build == build %}{% if error.level == 2 %}<p>{{error.message}}</p>{% endif %}{% endif %}{% endfor %}{% else %}None{% endif %}</td>
+            <td>{{build.warnings_no}}:{% if  build.warnings_no %}{% for warning in logs %}{% if warning.build == build %}{% if warning.level == 1 %}<p>{{warning.message}}</p>{% endif %}{% endif %}{% endfor %}{% else %}None{% endif %}</td>
+            <td>{% if build.outcome == 0 %}{% for t in build.target_set.all %}{% if t.is_image %}{{build.image_fstypes}}{% endif %}{% endfor %}{% endif %}</td>
+            <td>{{build.cooker_log_path}}</td>
+            <td>{{build.bitbake_version}}</td>
+            <td>{{build.build_name}}</td>
+        </tr>
+
+        {% endfor %}
+{% endblock %}
+
+

bitbake/lib/toaster/bldviewer/templates/configuration.html

@@ -0,0 +1,22 @@
+{% extends "basebuildpage.html" %}
+
+{% block pagetitle %}Configuration{% endblock %}
+{% block pagetable %}
+
+            <tr>
+            <th>Name</th>
+            <th>Description</th>
+            <th>Definition history</th>
+            <th>Value</th>
+            </tr>
+
+            {% for variable in configuration %}
+
+            <tr class="data">
+                <td>{{variable.variable_name}}</td>
+                <td>{% if variable.description %}{{variable.description}}{% endif %}</td>
+                <td>{% for vh in variable.variablehistory_set.all %}{{vh.operation}} in {{vh.file_name}}:{{vh.line_number}}<br/>{%endfor%}</td>
+                <td>{{variable.variable_value}}</td>
+            {% endfor %}
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templates/layer.html

@@ -0,0 +1,34 @@
+{% extends "basetable.html" %}
+
+{% block pagename %}
+    <h1>Toaster - Layers</h1>
+{% endblock %}
+
+{% block pagetable %}
+    {% load projecttags %}
+
+        <tr>
+            <th>Name</th>
+            <th>Local Path</th>
+            <th>Layer Index URL</th>
+            <th>Known Versions</th>
+        </tr>
+
+        {% for layer in layers %}
+
+        <tr class="data">
+            <td>{{layer.name}}</td>
+            <td>{{layer.local_path}}</td>
+            <td><a href='{{layer.layer_index_url}}'>{{layer.layer_index_url}}</a></td>
+            <td><table>
+            {% for lv in layer.versions %}
+                <tr><td>
+        <a href="{% url layer_versions_recipes lv.id %}">({{lv.priority}}){{lv.branch}}:{{lv.commit}} ({{lv.count}} recipes)</a>
+                </td></tr>
+            {% endfor %}
+            </table></td>
+        </tr>
+
+        {% endfor %}
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templates/package.html

@@ -0,0 +1,36 @@
+{% extends "basebuildpage.html" %}
+
+{% block pagetable %}
+    {% if not packages %}
+        <p>No packages were recorded for this target!</p>
+    {% else %}
+
+            <tr>
+            <th>Name</th>
+            <th>Version</th>
+            <th>Size (Bytes)</th>
+            <th>Recipe</th>
+            <th>Depends on</th>
+            </tr>
+
+            {% for package in packages %}
+
+            <tr class="data">
+                <td><a name="#{{package.name}}">{{package.name}}</a></td>
+                <td>{{package.version}}</td>
+                <td>{{package.size}}</td>
+                <td>{%if package.recipe %}<a name="{{package.recipe.name}}.{{package.package_name}}">
+                <a href="{% url layer_versions_recipes package.recipe.layer_version_id %}#{{package.recipe.name}}">{{package.recipe.name}}</a>{{package.package_name}}</a>{%endif%}</td>
+                <td>
+            <div style="height: 4em; overflow:auto">
+                    {% for d in package.tpackage_dependencies_package.all %}
+                    <a href="#{{d.name}}">{{d.depends_on.name}}</a><br/>
+                    {% endfor %}
+            </div>
+                </td>
+
+            {% endfor %}
+
+    {% endif %}
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templates/recipe.html

@@ -0,0 +1,53 @@
+{% extends "basetable.html" %}
+
+{% block pagename %}
+<ul class="nav nav-tabs" style="display: inline-block">
+  <li><a>Layer  {{layer_version.layer.name}}&nbsp;:&nbsp;{{layer_version.branch}}&nbsp;:&nbsp;{{layer_version.commit}}&nbsp;:&nbsp;{{layer_version.priority}}</a></li>
+</ul>
+    <h1>Toaster - Recipes for a Layer</h1>
+{% endblock %}
+
+{% block pagetable %}
+    {% load projecttags %}
+
+        <tr>
+        </tr>
+            <th>Name</th>
+            <th>Version</th>
+            <th>Summary</th>
+            <th>Description</th>
+            <th>Section</th>
+            <th>License</th>
+            <th>License file</th>
+            <th>Homepage</th>
+            <th>Bugtracker</th>
+            <th>Author</th>
+            <th>File_path</th>
+            <th style="width: 30em">Recipe Dependency</th>
+
+
+        {% for recipe in recipes %}
+
+        <tr class="data">
+            <td><a name="{{recipe.name}}">{{recipe.name}}</a></td>
+            <td>{{recipe.version}}</td>
+            <td>{{recipe.summary}}</td>
+            <td>{{recipe.description}}</td>
+            <td>{{recipe.section}}</td>
+            <td>{{recipe.license}}</td>
+            <td>{{recipe.licensing_info}}</td>
+            <td>{{recipe.homepage}}</td>
+            <td>{{recipe.bugtracker}}</td>
+            <td>{{recipe.file_path}}</td>
+            <td>
+        <div style="height: 5em; overflow:auto">
+            {% for rr in recipe.r_dependencies_recipe.all %}
+                <a href="#{{rr.depends_on.name}}">{{rr.depends_on.name}}</a><br/>
+            {% endfor %}
+        </div>
+            </td>
+        </tr>
+
+        {% endfor %}
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templates/task.html

@@ -0,0 +1,63 @@
+{% extends "basebuildpage.html" %}
+
+{% block pagetitle %}Tasks{% endblock %}
+{% block pagetable %}
+    {% if not tasks %}
+        <p>No tasks were executed in this build!</p>
+    {% else %}
+
+            <tr>
+            <th>Order</th>
+            <th>Task</th>
+            <th>Recipe Version</th>
+            <th>Task Type</th>
+            <th>Checksum</th>
+            <th>Outcome</th>
+            <th>Message</th>
+            <th>Logfile</th>
+            <th>Time</th>
+            <th>CPU usage</th>
+            <th>Disk I/O</th>
+            <th>Script type</th>
+            <th>File path</th>
+            <th>Depends</th>
+            </tr>
+
+            {% for task in tasks %}
+
+            <tr class="data">
+                <td>{{task.order}}</td>
+                <td><a name="{{task.recipe.name}}.{{task.task_name}}">
+                <a href="{% url layer_versions_recipes task.recipe.layer_version_id %}#{{task.recipe.name}}">{{task.recipe.name}}</a>.{{task.task_name}}</a></td>
+                <td>{{task.recipe.version}}</td>
+
+                {% if task.task_executed %}
+                <td>Executed</td>
+                {% else %}
+                <td>Prebuilt</td>
+                {% endif %}
+
+                <td>{{task.sstate_checksum}}</td>
+                <td>{{task.get_outcome_display}}{% if task.provider %}</br>(by <a href="#{{task.provider.recipe.name}}.{{task.provider.task_name}}">{{task.provider.recipe.name}}.{{task.provider.task_name}}</a>){% endif %}</td>
+                <td><p>{{task.message}}</td>
+                <td><a target="_fileview" href="file:///{{task.logfile}}">{{task.logfile}}</a></td>
+                <td>{{task.elapsed_time}}</td>
+                <td>{{task.cpu_usage}}</td>
+                <td>{{task.disk_io}}</td>
+                <td>{{task.get_script_type_display}}</td>
+                <td><a target="_fileview" href="file:///{{task.recipe.file_path}}">{{task.recipe.file_path}}</a></td>
+                <td>
+            <div style="height: 3em; overflow:auto">
+                {% for tt in task.task_dependencies_task.all %}
+                    <a href="#{{tt.depends_on.recipe.name}}.{{tt.depends_on.task_name}}">
+                    {{tt.depends_on.recipe.name}}.{{tt.depends_on.task_name}}</a><br/>
+                {% endfor %}
+            </div>
+                </td>
+                </tr>
+
+            {% endfor %}
+
+    {% endif %}
+
+{% endblock %}

bitbake/lib/toaster/bldviewer/templatetags/projecttags.py

@@ -0,0 +1,26 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from datetime import datetime
+from django import template
+
+register = template.Library()
+
+@register.simple_tag
+def time_difference(start_time, end_time):
+    return end_time - start_time

bitbake/lib/toaster/bldviewer/urls.py

@@ -0,0 +1,32 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from django.conf.urls import patterns, include, url
+from django.views.generic.simple import redirect_to
+
+urlpatterns = patterns('bldviewer.views',
+        url(r'^builds/$', 'build', name='all-builds'),
+        url(r'^build/(?P<build_id>\d+)/task/$', 'task', name='task'),
+        url(r'^build/(?P<build_id>\d+)/packages/$', 'bpackage', name='bpackage'),
+        url(r'^build/(?P<build_id>\d+)/package/(?P<package_id>\d+)/files/$', 'bfile', name='bfile'),
+        url(r'^build/(?P<build_id>\d+)/target/(?P<target_id>\d+)/packages/$', 'tpackage', name='tpackage'),
+        url(r'^build/(?P<build_id>\d+)/configuration/$', 'configuration', name='configuration'),
+        url(r'^layers/$', 'layer', name='all-layers'),
+        url(r'^layerversions/(?P<layerversion_id>\d+)/recipes/.*$', 'layer_versions_recipes', name='layer_versions_recipes'),
+        url(r'^$', redirect_to, {'url': 'builds/'}),
+)

bitbake/lib/toaster/bldviewer/views.py

@@ -0,0 +1,260 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import operator
+
+from django.db.models import Q
+from django.shortcuts import render
+from orm.models import Build, Target, Task, Layer, Layer_Version, Recipe, Target_Package, LogMessage, Variable
+from orm.models import Task_Dependency, Recipe_Dependency, Build_Package, Build_File, Build_Package_Dependency
+from django.views.decorators.cache import cache_control
+
+@cache_control(no_store=True)
+def build(request):
+    template = 'build.html'
+    build_info = Build.objects.all()
+
+    logs = LogMessage.objects.all()
+
+    context = {'builds': build_info, 'logs': logs ,
+        'hideshowcols' : [
+                {'name': 'Output', 'order':10},
+                {'name': 'Log', 'order':11},
+            ]}
+
+    return render(request, template, context)
+
+
+def _find_task_revdep(task):
+    tp = []
+    for p in Task_Dependency.objects.filter(depends_on=task):
+        tp.append(p.task);
+    return tp
+
+def _find_task_provider(task):
+    task_revdeps = _find_task_revdep(task)
+    for tr in task_revdeps:
+        if tr.outcome != Task.OUTCOME_COVERED:
+            return tr
+    for tr in task_revdeps:
+        trc = _find_task_provider(tr)
+        if trc is not None:
+            return trc
+    return None
+
+def task(request, build_id):
+    template = 'task.html'
+
+    tasks = Task.objects.filter(build=build_id)
+
+    for t in tasks:
+        if t.outcome == Task.OUTCOME_COVERED:
+            t.provider = _find_task_provider(t)
+
+    context = {'build': Build.objects.filter(pk=build_id)[0], 'tasks': tasks}
+
+    return render(request, template, context)
+
+def configuration(request, build_id):
+    template = 'configuration.html'
+    variables = Variable.objects.filter(build=build_id)
+    context = {'build': Build.objects.filter(pk=build_id)[0], 'configuration' : variables}
+    return render(request, template, context)
+
+def bpackage(request, build_id):
+    template = 'bpackage.html'
+    packages = Build_Package.objects.filter(build = build_id)
+    context = {'build': Build.objects.filter(pk=build_id)[0], 'packages' : packages}
+    return render(request, template, context)
+
+def bfile(request, build_id, package_id):
+    template = 'bfile.html'
+    files = Build_File.objects.filter(bpackage = package_id)
+    context = {'build': Build.objects.filter(pk=build_id)[0], 'files' : files}
+    return render(request, template, context)
+
+def tpackage(request, build_id, target_id):
+    template = 'package.html'
+
+    packages = Target_Package.objects.filter(target=target_id)
+
+    context = {'build' : Build.objects.filter(pk=build_id)[0],'packages': packages}
+
+    return render(request, template, context)
+
+def layer(request):
+    template = 'layer.html'
+    layer_info = Layer.objects.all()
+
+    for li in layer_info:
+        li.versions = Layer_Version.objects.filter(layer = li)
+        for liv in li.versions:
+            liv.count = Recipe.objects.filter(layer_version__id = liv.id).count()
+
+    context = {'layers': layer_info}
+
+    return render(request, template, context)
+
+
+def layer_versions_recipes(request, layerversion_id):
+    template = 'recipe.html'
+    recipes = Recipe.objects.filter(layer_version__id = layerversion_id)
+
+    context = {'recipes': recipes,
+            'layer_version' : Layer_Version.objects.filter( id = layerversion_id )[0]
+    }
+
+    return render(request, template, context)
+
+#### API
+
+import json
+from django.core import serializers
+from django.http import HttpResponse, HttpResponseBadRequest
+
+
+def model_explorer(request, model_name):
+
+    DESCENDING = 'desc'
+    response_data = {}
+    model_mapping = {
+        'build': Build,
+        'target': Target,
+        'target_package': Target_Package,
+        'task': Task,
+        'task_dependency': Task_Dependency,
+        'package': Build_Package,
+        'layer': Layer,
+        'layerversion': Layer_Version,
+        'recipe': Recipe,
+        'recipe_dependency': Recipe_Dependency,
+        'build_package': Build_Package,
+        'build_package_dependency': Build_Package_Dependency,
+        'build_file': Build_File,
+        'variable': Variable,
+        'logmessage': LogMessage,
+        }
+
+    if model_name not in model_mapping.keys():
+        return HttpResponseBadRequest()
+
+    model = model_mapping[model_name]
+
+    try:
+        limit = int(request.GET.get('limit', 0))
+    except ValueError:
+        limit = 0
+
+    try:
+        offset = int(request.GET.get('offset', 0))
+    except ValueError:
+        offset = 0
+
+    ordering_string, invalid = _validate_input(request.GET.get('orderby', ''),
+                                               model)
+    if invalid:
+        return HttpResponseBadRequest()
+
+    filter_string, invalid = _validate_input(request.GET.get('filter', ''),
+                                             model)
+    if invalid:
+        return HttpResponseBadRequest()
+
+    search_term = request.GET.get('search', '')
+
+    if filter_string:
+        filter_terms = _get_filtering_terms(filter_string)
+        try:
+            queryset = model.objects.filter(**filter_terms)
+        except ValueError:
+            queryset = []
+    else:
+        queryset = model.objects.all()
+
+    if search_term:
+        queryset = _get_search_results(search_term, queryset, model)
+
+    if ordering_string and queryset:
+        column, order = ordering_string.split(':')
+        if order.lower() == DESCENDING:
+            queryset = queryset.order_by('-' + column)
+        else:
+            queryset = queryset.order_by(column)
+
+    if offset and limit:
+        queryset = queryset[offset:(offset+limit)]
+    elif offset:
+        queryset = queryset[offset:]
+    elif limit:
+        queryset = queryset[:limit]
+
+    if queryset:
+        response_data['count'] = queryset.count()
+    else:
+        response_data['count'] = 0
+
+    response_data['list'] = serializers.serialize('json', queryset)
+
+    return HttpResponse(json.dumps(response_data),
+                        content_type='application/json')
+
+def _get_filtering_terms(filter_string):
+
+    search_terms = filter_string.split(":")
+    keys = search_terms[0].split(',')
+    values = search_terms[1].split(',')
+
+    return dict(zip(keys, values))
+
+def _validate_input(input, model):
+
+    invalid = 0
+
+    if input:
+        input_list = input.split(":")
+
+        # Check we have only one colon
+        if len(input_list) != 2:
+            invalid = 1
+            return None, invalid
+
+        # Check we have an equal number of terms both sides of the colon
+        if len(input_list[0].split(',')) != len(input_list[1].split(',')):
+            invalid = 1
+            return None, invalid
+
+        # Check we are looking for a valid field
+        valid_fields = model._meta.get_all_field_names()
+        for field in input_list[0].split(','):
+            if field not in valid_fields:
+                invalid = 1
+                return None, invalid
+
+    return input, invalid
+
+def _get_search_results(search_term, queryset, model):
+    search_objects = []
+    for st in search_term.split(" "):
+        q_map = map(lambda x: Q(**{x+'__icontains': st}),
+                model.search_allowed_fields)
+
+        search_objects.append(reduce(operator.or_, q_map))
+    search_object = reduce(operator.and_, search_objects)
+    queryset = queryset.filter(search_object)
+
+    return queryset

bitbake/lib/toaster/manage.py

@@ -0,0 +1,10 @@
+#!/usr/bin/env python
+import os
+import sys
+
+if __name__ == "__main__":
+    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "toastermain.settings")
+
+    from django.core.management import execute_from_command_line
+
+    execute_from_command_line(sys.argv)

bitbake/lib/toaster/orm/models.py

@@ -0,0 +1,265 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from django.db import models
+from django.utils.encoding import python_2_unicode_compatible
+
+
+class Build(models.Model):
+    SUCCEEDED = 0
+    FAILED = 1
+    IN_PROGRESS = 2
+
+    BUILD_OUTCOME = (
+        (SUCCEEDED, 'Succeeded'),
+        (FAILED, 'Failed'),
+        (IN_PROGRESS, 'In Progress'),
+    )
+
+    search_allowed_fields = ['machine',
+                             'cooker_log_path']
+
+    machine = models.CharField(max_length=100)
+    image_fstypes = models.CharField(max_length=100)
+    distro = models.CharField(max_length=100)
+    distro_version = models.CharField(max_length=100)
+    started_on = models.DateTimeField()
+    completed_on = models.DateTimeField()
+    outcome = models.IntegerField(choices=BUILD_OUTCOME, default=IN_PROGRESS)
+    errors_no = models.IntegerField(default=0)
+    warnings_no = models.IntegerField(default=0)
+    cooker_log_path = models.CharField(max_length=500)
+    build_name = models.CharField(max_length=100)
+    bitbake_version = models.CharField(max_length=50)
+
+@python_2_unicode_compatible
+class Target(models.Model):
+    search_allowed_fields = ['target', 'image_fstypes', 'file_name']
+    build = models.ForeignKey(Build)
+    target = models.CharField(max_length=100)
+    is_image = models.BooleanField(default = False)
+    file_name = models.CharField(max_length=100)
+    file_size = models.IntegerField()
+
+    def __str__(self):
+        return self.target
+
+
+class Task(models.Model):
+
+    SSTATE_NA = 0
+    SSTATE_MISS = 1
+    SSTATE_FAILED = 2
+    SSTATE_RESTORED = 3
+
+    SSTATE_RESULT = (
+        (SSTATE_NA, 'Not Applicable'), # For rest of tasks, but they still need checking.
+        (SSTATE_MISS, 'Missing'), # it is a miss
+        (SSTATE_FAILED, 'Failed'), # there was a pkg, but the script failed
+        (SSTATE_RESTORED, 'Restored'), # succesfully restored
+    )
+
+    CODING_NA = 0
+    CODING_NOEXEC = 1
+    CODING_PYTHON = 2
+    CODING_SHELL = 3
+
+    TASK_CODING = (
+        (CODING_NA, 'N/A'),
+        (CODING_NOEXEC, 'NoExec'),
+        (CODING_PYTHON, 'Python'),
+        (CODING_SHELL, 'Shell'),
+    )
+
+    OUTCOME_SUCCESS = 0
+    OUTCOME_COVERED = 1
+    OUTCOME_SSTATE = 2
+    OUTCOME_EXISTING = 3
+    OUTCOME_FAILED = 4
+    OUTCOME_NA = 5
+
+    TASK_OUTCOME = (
+        (OUTCOME_SUCCESS, 'Succeeded'),
+        (OUTCOME_COVERED, 'Covered'),
+        (OUTCOME_SSTATE, 'Sstate'),
+        (OUTCOME_EXISTING, 'Existing'),
+        (OUTCOME_FAILED, 'Failed'),
+        (OUTCOME_NA, 'Not Available'),
+    )
+
+    build = models.ForeignKey(Build, related_name='task_build')
+    order = models.IntegerField(null=True)
+    task_executed = models.BooleanField(default=False) # True means Executed, False means Prebuilt
+    outcome = models.IntegerField(choices=TASK_OUTCOME, default=OUTCOME_NA)
+    sstate_checksum = models.CharField(max_length=100, blank=True)
+    path_to_sstate_obj = models.FilePathField(max_length=500, blank=True)
+    recipe = models.ForeignKey('Recipe', related_name='build_recipe')
+    task_name = models.CharField(max_length=100)
+    source_url = models.FilePathField(max_length=255, blank=True)
+    work_directory = models.FilePathField(max_length=255, blank=True)
+    script_type = models.IntegerField(choices=TASK_CODING, default=CODING_NA)
+    line_number = models.IntegerField(default=0)
+    disk_io = models.IntegerField(null=True)
+    cpu_usage = models.DecimalField(max_digits=6, decimal_places=2, null=True)
+    elapsed_time = models.CharField(max_length=50, default=0)
+    sstate_result = models.IntegerField(choices=SSTATE_RESULT, default=SSTATE_NA)
+    message = models.CharField(max_length=240)
+    logfile = models.FilePathField(max_length=255, blank=True)
+
+    class Meta:
+        ordering = ('order', 'recipe' ,)
+
+
+class Task_Dependency(models.Model):
+    task = models.ForeignKey(Task, related_name='task_dependencies_task')
+    depends_on = models.ForeignKey(Task, related_name='task_dependencies_depends')
+
+
+class Build_Package(models.Model):
+    build = models.ForeignKey('Build')
+    recipe = models.ForeignKey('Recipe', null=True)
+    name = models.CharField(max_length=100)
+    version = models.CharField(max_length=100, blank=True)
+    revision = models.CharField(max_length=32, blank=True)
+    summary = models.CharField(max_length=200, blank=True)
+    description = models.CharField(max_length=200, blank=True)
+    size = models.IntegerField(default=0)
+    section = models.CharField(max_length=80, blank=True)
+    license = models.CharField(max_length=80, blank=True)
+
+class Build_Package_Dependency(models.Model):
+    TYPE_RDEPENDS = 0
+    TYPE_RPROVIDES = 1
+    TYPE_RRECOMMENDS = 2
+    TYPE_RSUGGESTS = 3
+    TYPE_RREPLACES = 4
+    TYPE_RCONFLICTS = 5
+    DEPENDS_TYPE = (
+        (TYPE_RDEPENDS, "rdepends"),
+        (TYPE_RPROVIDES, "rprovides"),
+        (TYPE_RRECOMMENDS, "rrecommends"),
+        (TYPE_RSUGGESTS, "rsuggests"),
+        (TYPE_RREPLACES, "rreplaces"),
+        (TYPE_RCONFLICTS, "rconflicts"),
+    )
+    package = models.ForeignKey(Build_Package, related_name='bpackage_dependencies_package')
+    depends_on = models.CharField(max_length=100)   # soft dependency
+    dep_type = models.IntegerField(choices=DEPENDS_TYPE)
+
+
+class Target_Package(models.Model):
+    target = models.ForeignKey('Target')
+    recipe = models.ForeignKey('Recipe', null=True)
+    name = models.CharField(max_length=100)
+    version = models.CharField(max_length=100, blank=True)
+    size = models.IntegerField()
+
+
+class Target_Package_Dependency(models.Model):
+    TYPE_DEPENDS = 0
+    TYPE_RDEPENDS = 1
+    TYPE_RECOMMENDS = 2
+
+    DEPENDS_TYPE = (
+        (TYPE_DEPENDS, "depends"),
+        (TYPE_RDEPENDS, "rdepends"),
+        (TYPE_RECOMMENDS, "recommends"),
+    )
+    package = models.ForeignKey(Target_Package, related_name='tpackage_dependencies_package')
+    depends_on = models.ForeignKey(Target_Package, related_name='tpackage_dependencies_depends')
+    dep_type = models.IntegerField(choices=DEPENDS_TYPE)
+
+
+class Build_File(models.Model):
+    bpackage = models.ForeignKey(Build_Package, related_name='filelist_bpackage')
+    path = models.FilePathField(max_length=255, blank=True)
+    size = models.IntegerField()
+
+class Target_File(models.Model):
+    tpackage = models.ForeignKey(Target_Package, related_name='filelist_tpackage')
+    path = models.FilePathField(max_length=255, blank=True)
+    size = models.IntegerField()
+
+
+class Recipe(models.Model):
+    name = models.CharField(max_length=100, blank=True)
+    version = models.CharField(max_length=100, blank=True)
+    layer_version = models.ForeignKey('Layer_Version', related_name='recipe_layer_version')
+    summary = models.CharField(max_length=100, blank=True)
+    description = models.CharField(max_length=100, blank=True)
+    section = models.CharField(max_length=100, blank=True)
+    license = models.CharField(max_length=200, blank=True)
+    licensing_info = models.TextField(blank=True)
+    homepage = models.URLField(blank=True)
+    bugtracker = models.URLField(blank=True)
+    file_path = models.FilePathField(max_length=255)
+
+
+class Recipe_Dependency(models.Model):
+    TYPE_DEPENDS = 0
+    TYPE_RDEPENDS = 1
+
+    DEPENDS_TYPE = (
+        (TYPE_DEPENDS, "depends"),
+        (TYPE_RDEPENDS, "rdepends"),
+    )
+    recipe = models.ForeignKey(Recipe, related_name='r_dependencies_recipe')
+    depends_on = models.ForeignKey(Recipe, related_name='r_dependencies_depends')
+    dep_type = models.IntegerField(choices=DEPENDS_TYPE)
+
+class Layer(models.Model):
+    name = models.CharField(max_length=100)
+    local_path = models.FilePathField(max_length=255)
+    layer_index_url = models.URLField()
+
+
+class Layer_Version(models.Model):
+    layer = models.ForeignKey(Layer, related_name='layer_version_layer')
+    branch = models.CharField(max_length=50)
+    commit = models.CharField(max_length=100)
+    priority = models.IntegerField()
+
+
+class Variable(models.Model):
+    build = models.ForeignKey(Build, related_name='variable_build')
+    variable_name = models.CharField(max_length=100)
+    variable_value = models.TextField(blank=True)
+    changed = models.BooleanField(default=False)
+    human_readable_name = models.CharField(max_length=200)
+    description = models.TextField(blank=True)
+
+class VariableHistory(models.Model):
+    variable = models.ForeignKey(Variable)
+    file_name = models.FilePathField(max_length=255)
+    line_number = models.IntegerField(null=True)
+    operation = models.CharField(max_length=16)
+
+class LogMessage(models.Model):
+    INFO = 0
+    WARNING = 1
+    ERROR = 2
+
+    LOG_LEVEL = ( (INFO, "info"),
+            (WARNING, "warn"),
+            (ERROR, "error") )
+
+    build = models.ForeignKey(Build)
+    level = models.IntegerField(choices=LOG_LEVEL, default=INFO)
+    message=models.CharField(max_length=240)
+    pathname = models.FilePathField(max_length=255, blank=True)
+    lineno = models.IntegerField(null=True)

bitbake/lib/toaster/toastergui/templates/index.html

@@ -0,0 +1,13 @@
+<html>
+<head>
+	<title>GUI Page</title>
+</head>
+<body>
+{% load staticfiles %}
+
+<img src="{% static "/static/images/yocto.jpg" %}" alt="Yocto"/>
+
+This is your basic index page!
+</body>
+
+</html>

bitbake/lib/toaster/toastergui/urls.py

@@ -0,0 +1,27 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+
+from django.conf import settings
+from django.conf.urls import patterns, include, url
+
+
+urlpatterns = patterns('toastergui.views',
+        url(r'^$', 'guihome', name='guihome'),
+)

bitbake/lib/toaster/toastergui/views.py

@@ -0,0 +1,26 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from django.shortcuts import render
+from orm.models import Build, Task
+
+
+def guihome(request):
+    template = 'index.html'
+
+    return render(request, template)

bitbake/lib/toaster/toastermain/settings.py

@@ -0,0 +1,191 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# Django settings for Toaster project.
+
+DEBUG = True
+TEMPLATE_DEBUG = DEBUG
+
+ADMINS = (
+    # ('Your Name', 'your_email@example.com'),
+)
+
+MANAGERS = ADMINS
+
+DATABASES = {
+    'default': {
+        'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
+        'NAME': 'toaster.sqlite',                      # Or path to database file if using sqlite3.
+        'USER': '',
+        'PASSWORD': '',
+        'HOST': '127.0.0.1',                      # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
+        'PORT': '3306',                      # Set to empty string for default.
+    }
+}
+
+# Hosts/domain names that are valid for this site; required if DEBUG is False
+# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
+ALLOWED_HOSTS = []
+
+# Local time zone for this installation. Choices can be found here:
+# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
+# although not all choices may be available on all operating systems.
+# In a Windows environment this must be set to your system time zone.
+
+# Always use local computer's time zone
+import time
+TIME_ZONE = time.tzname[0]
+
+# Language code for this installation. All choices can be found here:
+# http://www.i18nguy.com/unicode/language-identifiers.html
+LANGUAGE_CODE = 'en-us'
+
+SITE_ID = 1
+
+# If you set this to False, Django will make some optimizations so as not
+# to load the internationalization machinery.
+USE_I18N = True
+
+# If you set this to False, Django will not format dates, numbers and
+# calendars according to the current locale.
+USE_L10N = True
+
+# If you set this to False, Django will not use timezone-aware datetimes.
+USE_TZ = True
+
+# Absolute filesystem path to the directory that will hold user-uploaded files.
+# Example: "/var/www/example.com/media/"
+MEDIA_ROOT = ''
+
+# URL that handles the media served from MEDIA_ROOT. Make sure to use a
+# trailing slash.
+# Examples: "http://example.com/media/", "http://media.example.com/"
+MEDIA_URL = ''
+
+# Absolute path to the directory static files should be collected to.
+# Don't put anything in this directory yourself; store your static files
+# in apps' "static/" subdirectories and in STATICFILES_DIRS.
+# Example: "/var/www/example.com/static/"
+STATIC_ROOT = ''
+
+# URL prefix for static files.
+# Example: "http://example.com/static/", "http://static.example.com/"
+STATIC_URL = '/static/'
+
+# Additional locations of static files
+STATICFILES_DIRS = (
+    # Put strings here, like "/home/html/static" or "C:/www/django/static".
+    # Always use forward slashes, even on Windows.
+    # Don't forget to use absolute paths, not relative paths.
+)
+
+# List of finder classes that know how to find static files in
+# various locations.
+STATICFILES_FINDERS = (
+    'django.contrib.staticfiles.finders.FileSystemFinder',
+    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
+#    'django.contrib.staticfiles.finders.DefaultStorageFinder',
+)
+
+# Make this unique, and don't share it with anybody.
+SECRET_KEY = 'NOT_SUITABLE_FOR_HOSTED_DEPLOYMENT'
+
+# List of callables that know how to import templates from various sources.
+TEMPLATE_LOADERS = (
+    'django.template.loaders.filesystem.Loader',
+    'django.template.loaders.app_directories.Loader',
+#     'django.template.loaders.eggs.Loader',
+)
+
+MIDDLEWARE_CLASSES = (
+    'django.middleware.common.CommonMiddleware',
+    'django.contrib.sessions.middleware.SessionMiddleware',
+    'django.middleware.csrf.CsrfViewMiddleware',
+    'django.contrib.auth.middleware.AuthenticationMiddleware',
+    'django.contrib.messages.middleware.MessageMiddleware',
+    # Uncomment the next line for simple clickjacking protection:
+    # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
+)
+
+ROOT_URLCONF = 'toastermain.urls'
+
+# Python dotted path to the WSGI application used by Django's runserver.
+WSGI_APPLICATION = 'toastermain.wsgi.application'
+
+TEMPLATE_DIRS = (
+    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
+    # Always use forward slashes, even on Windows.
+    # Don't forget to use absolute paths, not relative paths.
+)
+
+INSTALLED_APPS = (
+    #'django.contrib.auth',
+    #'django.contrib.contenttypes',
+    #'django.contrib.sessions',
+    #'django.contrib.sites',
+    #'django.contrib.messages',
+    'django.contrib.staticfiles',
+    # Uncomment the next line to enable the admin:
+    # 'django.contrib.admin',
+    # Uncomment the next line to enable admin documentation:
+    # 'django.contrib.admindocs',
+    'orm',
+    'toastermain',
+    'bldviewer',
+    'toastergui',
+)
+
+# A sample logging configuration. The only tangible logging
+# performed by this configuration is to send an email to
+# the site admins on every HTTP 500 error when DEBUG=False.
+# See http://docs.djangoproject.com/en/dev/topics/logging for
+# more details on how to customize your logging configuration.
+LOGGING = {
+    'version': 1,
+    'disable_existing_loggers': False,
+    'filters': {
+        'require_debug_false': {
+            '()': 'django.utils.log.RequireDebugFalse'
+        }
+    },
+    'handlers': {
+        'mail_admins': {
+            'level': 'ERROR',
+            'filters': ['require_debug_false'],
+            'class': 'django.utils.log.AdminEmailHandler'
+        }
+    },
+    'loggers': {
+        'django.request': {
+            'handlers': ['mail_admins'],
+            'level': 'ERROR',
+            'propagate': True,
+        },
+    }
+}
+
+# If we're using sqlite, we need to tweak the performance a bit
+from django.db.backends.signals import connection_created
+def activate_synchronous_off(sender, connection, **kwargs):
+    if connection.vendor == 'sqlite':
+        cursor = connection.cursor()
+        cursor.execute('PRAGMA synchronous = 0;')
+connection_created.connect(activate_synchronous_off)
+#
+
+

bitbake/lib/toaster/toastermain/urls.py

@@ -0,0 +1,41 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2013        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from django.conf.urls import patterns, include, url
+from django.views.generic.simple import redirect_to
+from django.views.decorators.cache import never_cache
+
+
+# Uncomment the next two lines to enable the admin:
+# from django.contrib import admin
+# admin.autodiscover()
+
+urlpatterns = patterns('',
+    url(r'^simple/', include('bldviewer.urls')),
+    url(r'^api/1.0/', include('bldviewer.api')),
+    url(r'^gui/', include('toastergui.urls')),
+    url(r'^$', never_cache(redirect_to), {'url': '/simple/'}),
+    # Examples:
+    # url(r'^toaster/', include('toaster.foo.urls')),
+
+    # Uncomment the admin/doc line below to enable admin documentation:
+    # url(r'^admin/doc/', include('django.contrib.admindocs.urls')),
+
+    # Uncomment the next line to enable the admin:
+    # url(r'^admin/', include(admin.site.urls)),
+)

bitbake/lib/toaster/toastermain/wsgi.py

@@ -0,0 +1,32 @@
+"""
+WSGI config for Toaster project.
+
+This module contains the WSGI application used by Django's development server
+and any production WSGI deployments. It should expose a module-level variable
+named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
+this application via the ``WSGI_APPLICATION`` setting.
+
+Usually you will have the standard Django WSGI application here, but it also
+might make sense to replace the whole Django WSGI application with a custom one
+that later delegates to the Django one. For example, you could introduce WSGI
+middleware here, or combine a Django application with an application of another
+framework.
+
+"""
+import os
+
+# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
+# if running multiple sites in the same mod_wsgi process. To fix this, use
+# mod_wsgi daemon mode with each site in its own daemon process, or use
+# os.environ["DJANGO_SETTINGS_MODULE"] = "Toaster.settings"
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "toastermain.settings")
+
+# This application object is used by any WSGI server configured to use this
+# file. This includes Django's development server, if the WSGI_APPLICATION
+# setting points here.
+from django.core.wsgi import get_wsgi_application
+application = get_wsgi_application()
+
+# Apply WSGI middleware here.
+# from helloworld.wsgi import HelloWorldApplication
+# application = HelloWorldApplication(application)

documentation/Makefile

@@ -200,7 +200,8 @@ TARFILES = mega-manual.html mega-style.css figures/yocto-environment.png figures
 	figures/yocto-environment-ref.png figures/user-configuration.png figures/source-input.png \
 	figures/package-feeds.png figures/layer-input.png figures/images.png figures/sdk.png \
 	figures/source-fetching.png figures/patching.png figures/configuration-compile-autoreconf.png \
-	figures/analysis-for-package-splitting.png
+	figures/analysis-for-package-splitting.png figures/image-generation.png \
+	figures/sdk-generation.png
 	endif
 
 MANUALS = $(DOC)/$(DOC).html
@@ -219,7 +220,8 @@ TARFILES = ref-manual.html ref-style.css figures/poky-title.png \
 	figures/user-configuration.png figures/yocto-environment-ref.png \
 	figures/images.png figures/sdk.png figures/source-fetching.png \
 	figures/patching.png figures/configuration-compile-autoreconf.png \
-	figures/analysis-for-package-splitting.png
+	figures/analysis-for-package-splitting.png figures/image-generation.png \
+	figures/sdk-generation.png
 MANUALS = $(DOC)/$(DOC).html $(DOC)/$(DOC).pdf $(DOC)/eclipse
 FIGURES = figures
 STYLESHEET = $(DOC)/*.css
@@ -381,8 +383,8 @@ publish:
             echo " "; \
             echo "******** Publishing "$(DOC)".html"; \
             echo " "; \
-            scp -r $(MANUALS) $(STYLESHEET) www.yoctoproject.org:/srv/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
-            cd $(DOC); scp -r $(FIGURES) www.yoctoproject.org:/srv/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
+            scp -r $(MANUALS) $(STYLESHEET) docs.yp:/var/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
+            cd $(DOC); scp -r $(FIGURES) docs.yp:/var/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
 	else \
           echo " "; \
           echo $(DOC)".html missing.  Generate the file first then try again."; \

documentation/adt-manual/adt-manual.xml

@@ -63,7 +63,7 @@
             </revision>
             <revision>
                 <revnumber>1.5</revnumber>
-                <date>Sometime in 2013</date>
+                <date>October 2013</date>
                 <revremark>Released with the Yocto Project 1.5 Release.</revremark>
             </revision>
        </revhistory>

documentation/adt-manual/adt-prepare.xml

@@ -91,7 +91,9 @@
             <para>
                 If you use BitBake to generate the ADT Installer tarball, you must
                 <filename>source</filename> the environment setup script
-                (<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>)
+                (<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>
+                or
+                <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink>)
                 located in the Source Directory before running the
                 BitBake command that creates the tarball.
             </para>
@@ -266,7 +268,7 @@
                             target, go into the <filename>x86_64</filename>
                             folder and download the following installer:
                             <literallayout class='monospaced'>
-     poky-eglibc-x86_64-core-image-sato-i586.sh
+     poky-eglibc-x86_64-core-image-sato-i586-&DISTRO;.sh
                             </literallayout></para></listitem>
                         <listitem><para>Build your own toolchain installer.
                             For cases where you cannot use an installer
@@ -285,7 +287,7 @@
                     The example assumes the toolchain installer is located
                     in <filename>~/Downloads/</filename>.
                     <literallayout class='monospaced'>
-     $ ~/Downloads/poky-eglibc-x86_64-core-image-sato-i586.sh
+     $ ~/Downloads/poky-eglibc-x86_64-core-image-sato-i586-&DISTRO;.sh
                     </literallayout>
                     <note>
                         If you do not have write permissions for the directory
@@ -374,10 +376,11 @@
     </para>
 
     <para>
-        Be sure to run the environment setup script that matches the architecture for
-        which you are developing.
-        Environment setup scripts begin with the string "<filename>environment-setup</filename>"
-        and include as part of their name the architecture.
+        Be sure to run the environment setup script that matches the
+        architecture for which you are developing.
+        Environment setup scripts begin with the string
+        "<filename>environment-setup</filename>" and include as part of their
+        name the architecture.
         For example, the toolchain environment setup script for a 64-bit
         IA-based architecture installed in the default installation directory
         would be the following:
@@ -540,7 +543,7 @@
 
     <para>
         A final method of building the toolchain installer exists that has
-        significant advantages over the previous two methods.
+        significant advantages over the previous method.
         This method results in a toolchain installer that contains the sysroot
         that matches your target root filesystem.
         To build this installer, use the
@@ -549,8 +552,12 @@
 
     <para>
          Remember, before using any <filename>bitbake</filename> command, you
-         must source the <filename>&OE_INIT_PATH;</filename> script located in
-         the Source Directory and you must make sure your
+         must source the build environment setup script
+         (i.e.
+         <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>
+         or
+         <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink>)
+         located in the Source Directory and you must make sure your
          <filename>conf/local.conf</filename> variables are correct.
          In particular, you need to be sure the
          <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>

documentation/bsp-guide/bsp-guide.xml

@@ -75,7 +75,7 @@
             </revision>
             <revision>
                 <revnumber>1.5</revnumber>
-                <date>Sometime in 2013</date>
+                <date>October 2013</date>
                 <revremark>Released with the Yocto Project 1.5 Release.</revremark>
             </revision>
         </revhistory>

documentation/bsp-guide/bsp.xml

@@ -183,17 +183,15 @@
      meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config/
      meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config/crownbay/
      meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config/crownbay/xorg.conf
-     meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config/crownbay-noemgd/
-     meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config/crownbay-noemgd/xorg.conf
      meta-crownbay/recipes-kernel/
      meta-crownbay/recipes-kernel/linux/
-     meta-crownbay/recipes-kernel/linux/linux-yocto_3.2.bbappend
      meta-crownbay/recipes-kernel/linux/linux-yocto_3.4.bbappend
      meta-crownbay/recipes-kernel/linux/linux-yocto_3.8.bbappend
+     meta-crownbay/recipes-kernel/linux/linux-yocto_3.10.bbappend
      meta-crownbay/recipes-kernel/linux/linux-yocto-dev.bbappend
-     meta-crownbay/recipes-kernel/linux/linux-yocto-rt_3.2.bbappend
      meta-crownbay/recipes-kernel/linux/linux-yocto-rt_3.4.bbappend
      meta-crownbay/recipes-kernel/linux/linux-yocto-rt_3.8.bbappend
+     meta-crownbay/recipes-kernel/linux/linux-yocto-rt_3.10.bbappend
                 </literallayout>
             </para>
 
@@ -376,15 +374,6 @@
             <para>
                 Each BSP Layer requires at least one machine file.
                 However, you can supply more than one file.
-                For example, in the Crown Bay BSP shown earlier in this section, the
-                <filename>conf/machine</filename> directory contains two configuration files:
-                <filename>crownbay.conf</filename> and <filename>crownbay-noemgd.conf</filename>.
-                The <filename>crownbay.conf</filename> file is used for the Crown Bay BSP
-                that supports the <trademark class='registered'>Intel</trademark> Embedded
-                Media and Graphics Driver (<trademark class='registered'>Intel</trademark>
-                EMGD), while the <filename>crownbay-noemgd</filename> file is used for the
-                Crown Bay BSP that supports Video Electronics Standards Association (VESA)
-                graphics only.
             </para>
 
             <para>
@@ -428,15 +417,13 @@
                 This optional directory contains miscellaneous recipe files for the BSP.
                 Most notably would be the formfactor files.
                 For example, in the Crown Bay BSP there is the
-                <filename>formfactor_0.0.bbappend</filename> file, which is an append file used
-                to augment the recipe that starts the build.
-                Furthermore, there are machine-specific settings used during the build that are
-                defined by the <filename>machconfig</filename> files.
-                In the Crown Bay example, two <filename>machconfig</filename> files exist:
-                one that supports the
-                <trademark class='registered'>Intel</trademark> Embedded
-                Media and Graphics Driver (<trademark class='registered'>Intel</trademark>
-                EMGD) and one that does not:
+                <filename>formfactor_0.0.bbappend</filename> file, which is an
+                append file used to augment the recipe that starts the build.
+                Furthermore, there are machine-specific settings used during the
+                build that are defined by the <filename>machconfig</filename>.
+                In the Crown Bay example, two <filename>machconfig</filename> files
+                exist: one that supports the Intel® Embedded Media and Graphics
+                Driver (Intel® EMGD) and one that does not:
                 <literallayout class='monospaced'>
      meta-crownbay/recipes-bsp/formfactor/formfactor/crownbay/machconfig
      meta-crownbay/recipes-bsp/formfactor/formfactor/crownbay-noemgd/machconfig
@@ -467,15 +454,12 @@
                 This optional directory contains recipes for the BSP if it has
                 special requirements for graphics support.
                 All files that are needed for the BSP to support a display are kept here.
-                For example, the Crown Bay BSP contains two versions of the
-                <filename>xorg.conf</filename> file.
-                The version in <filename>crownbay</filename> builds a BSP that supports the
-                <trademark class='registered'>Intel</trademark> Embedded Media Graphics Driver (EMGD),
-                while the version in <filename>crownbay-noemgd</filename> builds
-                a BSP that supports Video Electronics Standards Association (VESA) graphics only:
+                For example, the Crown Bay BSP's <filename>xorg.conf</filename> file
+                detects the graphics support needed (i.e. the Intel® Embedded Media
+                Graphics Driver (EMGD) or the Video Electronics Standards Association
+                (VESA) graphics):
                 <literallayout class='monospaced'>
      meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config_0.1.bbappend
-     meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config/crownbay/xorg.conf
      meta-crownbay/recipes-graphics/xorg-xserver/xserver-xf86-config/crownbay-noemgd/xorg.conf
                 </literallayout>
             </para>
@@ -502,28 +486,28 @@
                 the <filename>meta-&lt;bsp_name&gt;/recipes-kernel/linux</filename> directory).
             </para>
             <para>
-                Suppose you are using the <filename>linux-yocto_3.8.bb</filename> recipe to build
+                Suppose you are using the <filename>linux-yocto_3.10.bb</filename> recipe to build
                 the kernel.
                 In other words, you have selected the kernel in your
                 <filename>&lt;bsp_name&gt;.conf</filename> file by adding these types
                 of statements:
                 <literallayout class='monospaced'>
      PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
-     PREFERRED_VERSION_linux-yocto ?= "3.8%"
+     PREFERRED_VERSION_linux-yocto ?= "3.10%"
                 </literallayout>
                 <note>
                     When the preferred provider is assumed by default, the
                     <filename>PREFERRED_PROVIDER</filename> statement does not appear in the
                     <filename>&lt;bsp_name&gt;.conf</filename> file.
                 </note>
-                You would use the <filename>linux-yocto_3.8.bbappend</filename> file to append
+                You would use the <filename>linux-yocto_3.10.bbappend</filename> file to append
                 specific BSP settings to the kernel, thus configuring the kernel for your particular BSP.
             </para>
             <para>
                 As an example, look at the existing Crown Bay BSP.
                 The append file used is:
                 <literallayout class='monospaced'>
-     meta-crownbay/recipes-kernel/linux/linux-yocto_3.8.bbappend
+     meta-crownbay/recipes-kernel/linux/linux-yocto_3.10.bbappend
                 </literallayout>
                 The following listing shows the file.
                 Be aware that the actual commit ID strings in this example listing might be different
@@ -532,46 +516,18 @@
                 <literallayout class='monospaced'>
      FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
 
-     COMPATIBLE_MACHINE_crownbay = "crownbay"
-     KMACHINE_crownbay = "crownbay"
-     KBRANCH_crownbay = "standard/crownbay"
-     KERNEL_FEATURES_crownbay_append = " features/drm-emgd/drm-emgd-1.16 cfg/vesafb"
-
      COMPATIBLE_MACHINE_crownbay-noemgd = "crownbay-noemgd"
      KMACHINE_crownbay-noemgd = "crownbay"
      KBRANCH_crownbay-noemgd = "standard/crownbay"
-     KERNEL_FEATURES_crownbay-noemgd_append = " cfg/vesafb"
+     KERNEL_FEATURES_append_crownbay-noemgd = " cfg/vesafb"
 
-     LINUX_VERSION = "3.8.4"
+     LINUX_VERSION = "3.10.11"
 
-     SRCREV_meta_crownbay = "2a6d36e75ca0a121570a389d7bab76ec240cbfda"
-     SRCREV_machine_crownbay = "47aed0c17c1c55988198ad39f86ae88894c8e0a4"
-     SRCREV_emgd_crownbay = "c780732f175ff0ec866fac2130175876b519b576"
-
-     SRCREV_meta_crownbay-noemgd = "2a6d36e75ca0a121570a389d7bab76ec240cbfda"
-     SRCREV_machine_crownbay-noemgd = "47aed0c17c1c55988198ad39f86ae88894c8e0a4"
-
-     SRC_URI_crownbay = "git://git.yoctoproject.org/linux-yocto-3.8.git;protocol=git;nocheckout=1;branch=${KBRANCH},${KMETA},emgd-1.16;name=machine,meta,emgd"
+     SRCREV_meta_crownbay-noemgd = "285f93bf942e8f6fa678ffc6cc53696ed5400718"
+     SRCREV_machine_crownbay-noemgd = "702040ac7c7ec66a29b4d147665ccdd0ff015577"
                 </literallayout>
-                This append file contains statements used to support the Crown Bay BSP for both
-                <trademark class='registered'>Intel</trademark> EMGD and the VESA graphics.
-                The build process, in this case, recognizes and uses only the statements that
-                apply to the defined machine name - <filename>crownbay</filename> in this case.
-                So, the applicable statements in the <filename>linux-yocto_3.8.bbappend</filename>
-                file are follows:
-                <literallayout class='monospaced'>
-     FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
-
-     COMPATIBLE_MACHINE_crownbay = "crownbay"
-     KMACHINE_crownbay = "crownbay"
-     KBRANCH_crownbay = "standard/crownbay"
-     KERNEL_FEATURES_crownbay_append = " features/drm-emgd/drm-emgd-1.16 cfg/vesafb"
-
-     SRCREV_meta_crownbay = "2a6d36e75ca0a121570a389d7bab76ec240cbfda"
-     SRCREV_machine_crownbay = "47aed0c17c1c55988198ad39f86ae88894c8e0a4"
-     SRCREV_emgd_crownbay = "c780732f175ff0ec866fac2130175876b519b576"
-                </literallayout>
-                The append file defines <filename>crownbay</filename> as the
+                This append file contains statements used to support the Crown Bay BSP.
+                The file defines <filename>crownbay</filename> as the
                 <ulink url='&YOCTO_DOCS_REF_URL;#var-COMPATIBLE_MACHINE'><filename>COMPATIBLE_MACHINE</filename></ulink>
                 and uses the
                 <ulink url='&YOCTO_DOCS_REF_URL;#var-KMACHINE'><filename>KMACHINE</filename></ulink> variable to
@@ -588,12 +544,6 @@
                 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink> Git
                 repository and the <filename>meta</filename> Git repository branches to identify the
                 exact kernel needed to build the Crown Bay BSP.
-                <note>
-                    For <filename>crownbay</filename>, a specific commit is also needed to point
-                    to the branch that supports EMGD graphics.
-                    At a minimum, every BSP points to the
-                    <filename>machine</filename> and <filename>meta</filename> commits.
-                </note>
             </para>
 
             <para>
@@ -1281,31 +1231,35 @@
                     Following is the complete example:
                     <literallayout class='monospaced'>
      $ yocto-bsp create myarm qemu
+     Checking basic git connectivity...
+     Done.
+
      Which qemu architecture would you like to use? [default: i386]
-	     1) i386    (32-bit)
+     	     1) i386    (32-bit)
 	     2) x86_64  (64-bit)
 	     3) ARM     (32-bit)
 	     4) PowerPC (32-bit)
 	     5) MIPS    (32-bit)
      3
-     Would you like to use the default (3.8) kernel? (y/n) [default: y]
+     Would you like to use the default (3.10) kernel? (y/n) [default: y] y
      Do you need a new machine branch for this BSP (the alternative is to re-use an existing branch)? [y/n] [default: y]
-     Getting branches from remote repo git://git.yoctoproject.org/linux-yocto-3.8.git...
+     Getting branches from remote repo git://git.yoctoproject.org/linux-yocto-3.10.git...
      Please choose a machine branch to base your new BSP branch on: [default: standard/base]
-             1) standard/arm-versatile-926ejs
-             2) standard/base
-             3) standard/beagleboard
-             4) standard/ck
-             5) standard/crownbay
-             6) standard/edf
-             7) standard/emenlow
-             8) standard/fri2
-             9) standard/fsl-mpc8315e-rdb
-             10) standard/mti-malta32
-             11) standard/mti-malta64
-             12) standard/qemuppc
-             13) standard/routerstationpro
-             14) standard/sys940x
+	     1) standard/arm-versatile-926ejs
+	     2) standard/base
+	     3) standard/beagleboard
+	     4) standard/ck
+	     5) standard/crownbay
+	     6) standard/edf
+	     7) standard/emenlow
+	     8) standard/fri2
+	     9) standard/fsl-mpc8315e-rdb
+	     10) standard/minnow
+	     11) standard/mti-malta32
+	     12) standard/mti-malta64
+	     13) standard/qemuppc
+	     14) standard/routerstationpro
+	     15) standard/sys940x
      1
      Would you like SMP support? (y/n) [default: y]
      Does your BSP have a touchscreen? (y/n) [default: n]
@@ -1320,7 +1274,7 @@
                             In the example, we use the ARM architecture.
                             </para></listitem>
                         <listitem><para>The script then prompts you for the kernel.
-                            The default 3.8 kernel is acceptable.
+                            The default 3.10 kernel is acceptable.
                             So, the example accepts the default.
                             If you enter 'n', the script prompts you to further enter the kernel
                             you do want to use (e.g. 3.2, 3.2_preempt-rt, and so forth.).</para></listitem>

documentation/dev-manual/dev-manual-intro.xml

@@ -24,17 +24,6 @@
             set up your host development system and build an image, which you
             find in the
             <ulink url='&YOCTO_DOCS_QS_URL;'>Yocto Project Quick Start</ulink>.
-            <note>
-                By default, using the Yocto Project creates a Poky distribution.
-                However, you can create your own distribution by providing key
-                <link linkend='metadata'>Metadata</link>.
-                A good example is Angstrom, which has had a distribution
-                based on the Yocto Project since its inception.
-                Other examples include  commercial distributions like
-                Wind River Linux, Mentor Embedded Linux, and ENEA Linux.
-                See the "<link linkend='creating-your-own-distribution'>Creating Your Own Distribution</link>"
-                section for more information.
-            </note>
         </para>
 
         <para>
@@ -43,13 +32,25 @@
             reconfigure the kernel, and develop an application using the
             popular <trademark class='trade'>Eclipse</trademark> IDE.
         </para>
+
+        <note>
+            By default, using the Yocto Project creates a Poky distribution.
+            However, you can create your own distribution by providing key
+            <link linkend='metadata'>Metadata</link>.
+            A good example is Angstrom, which has had a distribution
+            based on the Yocto Project since its inception.
+            Other examples include  commercial distributions like
+            Wind River Linux, Mentor Embedded Linux, and ENEA Linux.
+            See the "<link linkend='creating-your-own-distribution'>Creating Your Own Distribution</link>"
+            section for more information.
+        </note>
     </section>
 
     <section id='what-this-manual-provides'>
         <title>What This Manual Provides</title>
 
         <para>
-            The following list describes what you can get from this guide:
+            The following list describes what you can get from this manual:
             <itemizedlist>
                 <listitem><para>Information that lets you get set
                     up to develop using the Yocto Project.</para></listitem>
@@ -75,17 +76,17 @@
         <para>
             This manual will not give you the following:
             <itemizedlist>
-                <listitem><para>Step-by-step instructions if those instructions exist in other Yocto
-                    Project documentation.
+                <listitem><para><emphasis>Step-by-step instructions when those instructions exist in other Yocto
+                    Project documentation:</emphasis>
                     For example, the Yocto Project Application Developer's Guide contains detailed
                     instructions on how to run the
-                    <ulink url='&YOCTO_DOCS_ADT_URL;#installing-the-adt'>Installing the ADT and Toolchains</ulink>,
+                    <ulink url='&YOCTO_DOCS_ADT_URL;#installing-the-adt'>ADT Installer</ulink>,
                     which is used to set up a cross-development environment.</para></listitem>
-                <listitem><para>Reference material.
+                <listitem><para><emphasis>Reference material:</emphasis>
                     This type of material resides in an appropriate reference manual.
                     For example, system variables are documented in the
                     <ulink url='&YOCTO_DOCS_REF_URL;'>Yocto Project Reference Manual</ulink>.</para></listitem>
-                <listitem><para>Detailed public information that is not specific to the Yocto Project.
+                <listitem><para><emphasis>Detailed public information that is not specific to the Yocto Project:</emphasis>
                     For example, exhaustive information on how to use Git is covered better through the
                     Internet than in this manual.</para></listitem>
             </itemizedlist>
@@ -109,7 +110,8 @@
                     with the Yocto Project and quickly begin building an image.</para></listitem>
                 <listitem><para><emphasis>
                     <ulink url='&YOCTO_DOCS_REF_URL;'>Yocto Project Reference Manual</ulink>:</emphasis> This manual is a reference
-                    guide to the OpenEmbedded build system known as "Poky."
+                    guide to the OpenEmbedded build system, which is based on BitBake.
+                    The build system is sometimes referred to as "Poky".
                     </para></listitem>
                 <listitem><para><emphasis>
                     <ulink url='&YOCTO_DOCS_ADT_URL;'>Yocto Project Application Developer's Guide</ulink>:</emphasis>

documentation/dev-manual/dev-manual-newbie.xml

@@ -254,9 +254,9 @@
                 <listitem><para><ulink url='http://sitaramc.github.com/gitolite/master-toc.html'>The <filename>gitolite</filename> master index</ulink>:
                     All topics for <filename>gitolite</filename>.
                     </para></listitem>
-                <listitem><para><ulink url='http://hjemli.net/git/cgit/tree/README'><filename>cgit</filename> index</ulink>:
-                    A <filename>README</filename> file on how to create a
-                    fast web interface for Git.</para></listitem>
+                <listitem><para><ulink url='https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools'>Interfaces, frontends, and tools</ulink>:
+                    Documentation on how to create interfaces and frontends
+                    for Git.</para></listitem>
             </itemizedlist>
         </note>
     </section>
@@ -275,9 +275,8 @@
         </para>
 
         <para>
-            See "<ulink url='http://autobuilder.yoctoproject.org:8010/'>Welcome to the buildbot for the Yocto Project</ulink>"
-            for the Yocto Project's reference implementation that uses
-            buildbot.
+            See "<ulink url='http://autobuilder.yoctoproject.org'>Yocto Project Autobuilder</ulink>"
+            for more information and links to buildbot.
             The Yocto Project team has found this implementation
             works well in this role.
             A public example of this is the Yocto Project
@@ -394,7 +393,7 @@
                     For some guidance on mailing lists to use, see the list in the
                     "<link linkend='how-to-submit-a-change'>How to Submit a Change</link>"
                     section.
-                    For a description of the available mailing lists, see
+                    For a description of the available mailing lists, see the
                     "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing Lists</ulink>"
                     section in the Yocto Project Reference Manual.
                     </para></listitem>
@@ -509,7 +508,9 @@
                 This term refers to the area used by the OpenEmbedded build system for builds.
                 The area is created when you <filename>source</filename> the setup
                 environment script that is found in the Source Directory
-                (i.e. <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>).
+                (i.e. <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>
+                or
+                <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink>).
                 The <ulink url='&YOCTO_DOCS_REF_URL;#var-TOPDIR'><filename>TOPDIR</filename></ulink>
                 variable points to the Build Directory.</para>
 
@@ -613,7 +614,7 @@
                 "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>"
                 section in the Yocto Project Board Support Packages (BSP)
                 Developer's Guide.</para></listitem>
-            <listitem><para id='meta-toochain'><emphasis>Meta-Toolchain:</emphasis>
+            <listitem><para id='meta-toolchain'><emphasis>Meta-Toolchain:</emphasis>
                 A term sometimes used for
                 <link linkend='cross-development-toolchain'>Cross-Development Toolchain</link>.
                 </para></listitem>
@@ -687,7 +688,7 @@
                 For example, downloading and unpacking <filename>&YOCTO_POKY_TARBALL;</filename>
                 results in a Source Directory whose top-level folder is named
                 <filename>&YOCTO_POKY;</filename>.
-                If you create a local copy of the Git repository, then you can name the repository
+                If you create a local copy of the Git repository, you can name the repository
                 anything you like.
                 Throughout much of the documentation, <filename>poky</filename> is used as the name of
                 the top-level folder of the local copy of the poky Git repository.
@@ -863,8 +864,8 @@
             It is important to understand that Git tracks content change and not files.
             Git uses "branches" to organize different development efforts.
             For example, the <filename>poky</filename> repository has
-            <filename>bernard</filename>,
-            <filename>edison</filename>, <filename>denzil</filename>, <filename>danny</filename>
+            <filename>denzil</filename>, <filename>danny</filename>,
+            <filename>dylan</filename>, <filename>dora</filename>,
             and <filename>master</filename> branches among others.
             You can see all the branches by going to
             <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and
@@ -1231,6 +1232,10 @@
                 occurred.</para></listitem>
             <listitem><para>Be sure to indicate the Severity of the bug.
                 Severity communicates how the bug impacted your work.</para></listitem>
+            <listitem><para>Select the appropriate "Documentation change" item
+                for the bug.
+                Fixing a bug may or may not affect the Yocto Project
+                documentation.</para></listitem>
             <listitem><para>Provide a brief summary of the issue.
                 Try to limit your summary to just a line or two and be sure to capture the
                 essence of the issue.</para></listitem>

documentation/dev-manual/dev-manual-start.xml

@@ -32,7 +32,7 @@
 
     <para>
         You can use the OpenEmbedded build system, which uses
-        BitBake to develop complete Linux
+        BitBake, to develop complete Linux
         images and associated user-space applications for architectures based
         on ARM, MIPS, PowerPC, x86 and x86-64.
         <note>
@@ -122,11 +122,11 @@
                         <literallayout class='monospaced'>
      $ git clone git://git.yoctoproject.org/poky
      Cloning into 'poky'...
-     remote: Counting objects: 183981, done.
-     remote: Compressing objects: 100% (47428/47428), done.
-     remote: Total 183981 (delta 132271), reused 183703 (delta 132044)
-     Receiving objects: 100% (183981/183981), 89.71 MiB | 2.93 MiB/s, done.
-     Resolving deltas: 100% (132271/132271), done.
+     remote: Counting objects: 203728, done.
+     remote: Compressing objects: 100% (52371/52371), done.
+     remote: Total 203728 (delta 147444), reused 202891 (delta 146614)
+     Receiving objects: 100% (203728/203728), 95.54 MiB | 308 KiB/s, done.
+     Resolving deltas: 100% (147444/147444), done.
                         </literallayout></para>
                         <para>For another example of how to set up your own local Git repositories, see this
                         <ulink url='&YOCTO_WIKI_URL;/wiki/Transcript:_from_git_checkout_to_meta-intel_BSP'>
@@ -145,27 +145,28 @@
                 For simplicity, it is recommended that you create these structures outside of the
                 Source Directory (usually <filename>poky</filename>).</para>
                 <para>As an example, the following transcript shows how to create the bare clone
-                of the <filename>linux-yocto-3.8</filename> kernel and then create a copy of
+                of the <filename>linux-yocto-3.10</filename> kernel and then create a copy of
                 that clone.
                 <note>When you have a local Yocto Project kernel Git repository, you can
                 reference that repository rather than the upstream Git repository as
                 part of the <filename>clone</filename> command.
                 Doing so can speed up the process.</note></para>
                 <para>In the following example, the bare clone is named
-                <filename>linux-yocto-3.8.git</filename>, while the
-                copy is named <filename>my-linux-yocto-3.8-work</filename>:
+                <filename>linux-yocto-3.10.git</filename>, while the
+                copy is named <filename>my-linux-yocto-3.10-work</filename>:
                 <literallayout class='monospaced'>
-     $ git clone --bare git://git.yoctoproject.org/linux-yocto-3.8 linux-yocto-3.8.git
-     Cloning into bare repository 'linux-yocto-3.8.git'...
-     remote: Counting objects: 2847090, done.
-     remote: Compressing objects: 100% (454675/454675), done.
-     remote: Total 2847090 (delta 2386170), reused 2825793 (delta 2364886)
-     Receiving objects: 100% (2847090/2847090), 603.19 MiB | 3.54 MiB/s, done.
-     Resolving deltas: 100% (2386170/2386170), done.                </literallayout></para>
+     $ git clone --bare git://git.yoctoproject.org/linux-yocto-3.10 linux-yocto-3.10.git
+     Cloning into bare repository 'linux-yocto-3.10.git'...
+     remote: Counting objects: 3364487, done.
+     remote: Compressing objects: 100% (507178/507178), done.
+     remote: Total 3364487 (delta 2827715), reused 3364481 (delta 2827709)
+     Receiving objects: 100% (3364487/3364487), 722.95 MiB | 423 KiB/s, done.
+     Resolving deltas: 100% (2827715/2827715), done.
+                </literallayout></para>
                 <para>Now create a clone of the bare clone just created:
                 <literallayout class='monospaced'>
-     $ git clone linux-yocto-3.8.git my-linux-yocto-3.8-work
-     Cloning into 'my-linux-yocto-3.8-work'...
+     $ git clone linux-yocto-3.10.git my-linux-yocto-3.10-work
+     Cloning into 'my-linux-yocto-3.10-work'...
      done.
                 </literallayout></para></listitem>
             <listitem id='meta-yocto-kernel-extras-repo'><para><emphasis>
@@ -189,11 +190,12 @@
      $ cd ~/poky
      $ git clone git://git.yoctoproject.org/meta-yocto-kernel-extras meta-yocto-kernel-extras
      Cloning into 'meta-yocto-kernel-extras'...
-     remote: Counting objects: 690, done.
-     remote: Compressing objects: 100% (431/431), done.
-     remote: Total 690 (delta 238), reused 690 (delta 238)
-     Receiving objects: 100% (690/690), 532.60 KiB, done.
-     Resolving deltas: 100% (238/238), done.                </literallayout></para></listitem>
+     remote: Counting objects: 727, done.
+     remote: Compressing objects: 100% (452/452), done.
+     remote: Total 727 (delta 260), reused 719 (delta 252)
+     Receiving objects: 100% (727/727), 536.36 KiB | 102 KiB/s, done.
+     Resolving deltas: 100% (260/260), done.
+               </literallayout></para></listitem>
            <listitem><para id='supported-board-support-packages-(bsps)'><emphasis>Supported Board
                 Support Packages (BSPs):</emphasis>
                 The Yocto Project provides a layer called <filename>meta-intel</filename> and
@@ -226,8 +228,8 @@
                         BSP tarball from the same "Downloads" page of the
                         <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>
                         to get the Yocto Project release.
-                        Once on the "Download" page, look for "BSP" under the
-                        "Type" heading.</para>
+                        Once on the "Download" page, look to the right of the
+                        page and scroll down to find the BSP tarballs.</para>
                         <para>Once you have the tarball, just extract it into a directory of your choice.
                         Again, this method just produces a snapshot of the BSP layer in the form
                         of a hierarchical directory structure.</para></listitem>
@@ -246,16 +248,17 @@
      $ cd ~/poky
      $ git clone git://git.yoctoproject.org/meta-intel.git
      Cloning into 'meta-intel'...
-     remote: Counting objects: 6264, done.
-     remote: Compressing objects: 100% (2135/2135), done.
-     remote: Total 6264 (delta 3321), reused 6235 (delta 3293)
-     Receiving objects: 100% (6264/6264), 2.17 MiB | 2.63 MiB/s, done.
-     Resolving deltas: 100% (3321/3321), done.
+     remote: Counting objects: 7366, done.
+     remote: Compressing objects: 100% (2491/2491), done.
+     remote: Total 7366 (delta 3997), reused 7299 (delta 3930)
+     Receiving objects: 100% (7366/7366), 2.31 MiB | 95 KiB/s, done.
+     Resolving deltas: 100% (3997/3997), done.
                         </literallayout></para>
                         <para>The same
-                        <ulink url='&YOCTO_WIKI_URL;/wiki/Transcript:_from_git_checkout_to_meta-intel_BSP'>
-                        wiki page</ulink> referenced earlier covers how to
-                        set up the <filename>meta-intel</filename> Git repository.</para></listitem>
+                        <ulink url='&YOCTO_WIKI_URL;/wiki/Transcript:_from_git_checkout_to_meta-intel_BSP'>wiki page</ulink>
+                        referenced earlier covers how to
+                        set up the <filename>meta-intel</filename> Git repository.
+                        </para></listitem>
                 </itemizedlist></para></listitem>
             <listitem><para><emphasis>Eclipse Yocto Plug-in:</emphasis>  If you are developing
                 applications using the Eclipse Integrated Development Environment (IDE),

documentation/dev-manual/dev-manual.xml

@@ -53,7 +53,7 @@
             </revision>
             <revision>
                 <revnumber>1.5</revnumber>
-                <date>Sometime in 2013</date>
+                <date>October 2013</date>
                 <revremark>Released with the Yocto Project 1.5 Release.</revremark>
             </revision>
         </revhistory>

documentation/kernel-dev/kernel-dev-advanced.xml

@@ -296,6 +296,20 @@
      $ git commit --allow-empty -m "Create orphan meta branch"
             </literallayout>
         </para>
+
+        <para>
+            If you modify the Metadata in the linux-yocto 
+            <filename>meta</filename> branch, you must not forget to update 
+            the 
+            <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
+            statements in the kernel's recipe.
+            In particular, you need to update the 
+            <filename>SRCREV_meta</filename> variable to match the commit in 
+            the <filename>KMETA</filename> branch you wish to use.
+            Changing the data in these branches and not updating the 
+            <filename>SRCREV</filename> statements to match will cause the 
+            build to fetch an older commit.
+        </para>
     </section>
 </section>
 

documentation/kernel-dev/kernel-dev.xml

@@ -38,7 +38,7 @@
             </revision>
             <revision>
                 <revnumber>1.5</revnumber>
-                <date>Sometime in 2013</date>
+                <date>October 2013</date>
                 <revremark>Released with the Yocto Project 1.5 Release.</revremark>
             </revision>
         </revhistory>