buildtools/uninative-tarball: Fix deployment overlap issues

We still have problems where deploying SDKMACHINE=i686 can cause removal
of SDKMACHINE=x86_64 artefacts.

The reason is that x86_64 is a BUILD_ARCH as well as an SDK_ARCH and
the manifest namespaces overlap. To fix this, set PACKAGE_ARCH and
the stamp-extra-into to include SDK_OS. SDK_OS may not be entirely correct
but it is what sstate.bbclass uses for nativesdk and fixing that is
a separate issue.

This is confirmed to resolve artefact problems on the AB which have been
delaying a new uninative release.

(From OE-Core rev: 1dbc6ec4ca061570d2482c9abebcf720298db9b7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml

@@ -44,6 +44,21 @@
      VARIABLE = " "
                 </literallayout>
             </para>
+
+            <para>
+                You can use single quotes instead of double quotes
+                when setting a variable's value.
+                Doing so allows you to use values that contain the double
+                character:
+                <literallayout class='monospaced'>
+     VARIABLE = 'I have a " in my value'
+                </literallayout>
+                <note>
+                    Unlike in Bourne shells, single quotes work identically
+                    to double quotes in all other ways.
+                    They do not suppress variable expansions.
+                </note>
+            </para>
         </section>
 
         <section id='variable-expansion'>
@@ -1228,7 +1243,7 @@
                 </literallayout>
                 For methods you can use with anonymous Python functions,
                 see the
-                "<link linkend='accessing-datastore-variables-using-python'>Accessing Datastore Variables Using Python</link>"
+                "<link linkend='functions-you-can-call-from-within-python'>Functions You Can Call From Within Python</link>"
                 section.
                 For a different method to run Python code during parsing,
                 see the
@@ -2082,102 +2097,130 @@
         </section>
     </section>
 
-    <section id='accessing-datastore-variables-using-python'>
-        <title>Accessing Datastore Variables Using Python</title>
+    <section id='functions-you-can-call-from-within-python'>
+        <title>Functions You Can Call From Within Python</title>
 
         <para>
-            It is often necessary to access variables in the
-            BitBake datastore using Python functions.
-            The Bitbake datastore has an API that allows you this
-            access.
-            Here is a list of available operations:
+            BitBake provides many functions you can call from
+            within Python functions.
+            This section lists the most commonly used functions,
+            and mentions where to find others.
         </para>
 
-        <para>
-            <informaltable frame='none'>
-                <tgroup cols='2' align='left' colsep='1' rowsep='1'>
-                    <colspec colname='c1' colwidth='1*'/>
-                    <colspec colname='c2' colwidth='1*'/>
-                    <thead>
-                        <row>
-                            <entry align="left"><emphasis>Operation</emphasis></entry>
-                            <entry align="left"><emphasis>Description</emphasis></entry>
-                        </row>
-                    </thead>
-                    <tbody>
-                        <row>
-                            <entry align="left"><filename>d.getVar("X", expand)</filename></entry>
-                            <entry align="left">Returns the value of variable "X".
-                            Using "expand=True" expands the value.</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.setVar("X", "value")</filename></entry>
-                            <entry align="left">Sets the variable "X" to "value".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.appendVar("X", "value")</filename></entry>
-                            <entry align="left">Adds "value" to the end of the variable "X".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.prependVar("X", "value")</filename></entry>
-                            <entry align="left">Adds "value" to the start of the variable "X".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.delVar("X")</filename></entry>
-                            <entry align="left">Deletes the variable "X" from the datastore.</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.renameVar("X", "Y")</filename></entry>
-                            <entry align="left">Renames the variable "X" to "Y".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.getVarFlag("X", flag, expand)</filename></entry>
-                            <entry align="left">Returns the value of variable "X".
+        <section id='functions-for-accessing-datastore-variables'>
+            <title>Functions for Accessing Datastore Variables</title>
+
+            <para>
+                It is often necessary to access variables in the
+                BitBake datastore using Python functions.
+                The Bitbake datastore has an API that allows you this
+                access.
+                Here is a list of available operations:
+            </para>
+
+            <para>
+                <informaltable frame='none'>
+                    <tgroup cols='2' align='left' colsep='1' rowsep='1'>
+                        <colspec colname='c1' colwidth='1*'/>
+                        <colspec colname='c2' colwidth='1*'/>
+                        <thead>
+                            <row>
+                                <entry align="left"><emphasis>Operation</emphasis></entry>
+                                <entry align="left"><emphasis>Description</emphasis></entry>
+                            </row>
+                        </thead>
+                        <tbody>
+                            <row>
+                                <entry align="left"><filename>d.getVar("X", expand)</filename></entry>
+                                <entry align="left">Returns the value of variable "X".
                                 Using "expand=True" expands the value.</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.setVarFlag("X", flag, "value")</filename></entry>
-                            <entry align="left">Sets the named flag for variable "X" to "value".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.appendVarFlag("X", flag, "value")</filename></entry>
-                            <entry align="left">Appends "value" to the named flag on the
-                            variable "X".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.prependVarFlag("X", flag, "value")</filename></entry>
-                            <entry align="left">Prepends "value" to the named flag on
-                               the variable "X".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.delVarFlag("X", flag)</filename></entry>
-                            <entry align="left">Deletes the named flag on the variable
-                                "X" from the datastore.</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.setVarFlags("X", flagsdict)</filename></entry>
-                            <entry align="left">Sets the flags specified in
-                                the <filename>flagsdict()</filename> parameter.
-                                <filename>setVarFlags</filename> does not clear previous flags.
-                                Think of this operation as <filename>addVarFlags</filename>.</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.getVarFlags("X")</filename></entry>
-                            <entry align="left">Returns a <filename>flagsdict</filename> of the flags for
-                                the variable "X".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.delVarFlags("X")</filename></entry>
-                            <entry align="left">Deletes all the flags for the variable "X".</entry>
-                        </row>
-                        <row>
-                            <entry align="left"><filename>d.expand(expression)</filename></entry>
-                            <entry align="left">Expands variable references in the specified string expression.</entry>
-                        </row>
-                    </tbody>
-                </tgroup>
-            </informaltable>
-        </para>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.setVar("X", "value")</filename></entry>
+                                <entry align="left">Sets the variable "X" to "value".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.appendVar("X", "value")</filename></entry>
+                                <entry align="left">Adds "value" to the end of the variable "X".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.prependVar("X", "value")</filename></entry>
+                                <entry align="left">Adds "value" to the start of the variable "X".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.delVar("X")</filename></entry>
+                                <entry align="left">Deletes the variable "X" from the datastore.</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.renameVar("X", "Y")</filename></entry>
+                                <entry align="left">Renames the variable "X" to "Y".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.getVarFlag("X", flag, expand)</filename></entry>
+                                <entry align="left">Returns the value of variable "X".
+                                    Using "expand=True" expands the value.</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.setVarFlag("X", flag, "value")</filename></entry>
+                                <entry align="left">Sets the named flag for variable "X" to "value".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.appendVarFlag("X", flag, "value")</filename></entry>
+                                <entry align="left">Appends "value" to the named flag on the
+                                variable "X".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.prependVarFlag("X", flag, "value")</filename></entry>
+                                <entry align="left">Prepends "value" to the named flag on
+                                   the variable "X".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.delVarFlag("X", flag)</filename></entry>
+                                <entry align="left">Deletes the named flag on the variable
+                                    "X" from the datastore.</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.setVarFlags("X", flagsdict)</filename></entry>
+                                <entry align="left">Sets the flags specified in
+                                    the <filename>flagsdict()</filename> parameter.
+                                    <filename>setVarFlags</filename> does not clear previous flags.
+                                    Think of this operation as <filename>addVarFlags</filename>.</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.getVarFlags("X")</filename></entry>
+                                <entry align="left">Returns a <filename>flagsdict</filename> of the flags for
+                                    the variable "X".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.delVarFlags("X")</filename></entry>
+                                <entry align="left">Deletes all the flags for the variable "X".</entry>
+                            </row>
+                            <row>
+                                <entry align="left"><filename>d.expand(expression)</filename></entry>
+                                <entry align="left">Expands variable references in the specified string expression.</entry>
+                            </row>
+                        </tbody>
+                    </tgroup>
+                </informaltable>
+            </para>
+        </section>
+
+        <section id='other-functions'>
+            <title>Other Functions</title>
+
+            <para>
+                You can find many other functions that can be called
+                from Python by looking at the source code of the
+                <filename>bb</filename> module, which is in
+                <filename>bitbake/lib/bb</filename>.
+                For example,
+                <filename>bitbake/lib/bb/utils.py</filename> includes
+                the commonly used functions
+                <filename>bb.utils.contains()</filename> and
+                <filename>bb.utils.mkdirhier()</filename>, which come
+                with docstrings.
+            </para>
+        </section>
     </section>
 
     <section id='task-checksums-and-setscene'>

bitbake/lib/bb/build.py

@@ -193,12 +193,6 @@ def exec_func(func, d, dirs = None, pythonexception=False):
     except:
         oldcwd = None
 
-    body = d.getVar(func, False)
-    if not body:
-        if body is None:
-            logger.warning("Function %s doesn't exist", func)
-        return
-
     flags = d.getVarFlags(func)
     cleandirs = flags.get('cleandirs')
     if cleandirs:
@@ -217,6 +211,13 @@ def exec_func(func, d, dirs = None, pythonexception=False):
         adir = dirs[-1]
     else:
         adir = None
+
+    body = d.getVar(func, False)
+    if not body:
+        if body is None:
+            logger.warning("Function %s doesn't exist", func)
+        return
+
     ispython = flags.get('python')
 
     lockflag = flags.get('lockfiles')
@@ -574,6 +575,9 @@ def _exec_task(fn, task, d, quieterr):
             logger.error(str(exc))
             event.fire(TaskFailed(task, logfn, localdata, errprinted), localdata)
         return 1
+    except bb.BBHandledException:
+        event.fire(TaskFailed(task, logfn, localdata, True), localdata)
+        return 1
     finally:
         sys.stdout.flush()
         sys.stderr.flush()

bitbake/lib/bb/cooker.py

@@ -658,7 +658,7 @@ class BBCooker:
         if task is None:
             task = self.configuration.cmd
 
-        fulltargetlist = self.checkPackages(pkgs_to_build)
+        fulltargetlist = self.checkPackages(pkgs_to_build, task)
         taskdata = {}
         localdata = {}
 
@@ -1147,6 +1147,8 @@ class BBCooker:
             collection_list = collections.split()
             min_prio = 0
             for c in collection_list:
+                bb.debug(1,'Processing %s in collection list' % (c))
+
                 # Get collection priority if defined explicitly
                 priority = self.data.getVar("BBFILE_PRIORITY_%s" % c, True)
                 if priority:
@@ -1165,10 +1167,10 @@ class BBCooker:
                 deps = self.data.getVar("LAYERDEPENDS_%s" % c, True)
                 if deps:
                     try:
-                        deplist = bb.utils.explode_dep_versions2(deps)
+                        depDict = bb.utils.explode_dep_versions2(deps)
                     except bb.utils.VersionStringException as vse:
                         bb.fatal('Error parsing LAYERDEPENDS_%s: %s' % (c, str(vse)))
-                    for dep, oplist in list(deplist.items()):
+                    for dep, oplist in list(depDict.items()):
                         if dep in collection_list:
                             for opstr in oplist:
                                 layerver = self.data.getVar("LAYERVERSION_%s" % dep, True)
@@ -1187,10 +1189,39 @@ class BBCooker:
                         else:
                             parselog.error("Layer '%s' depends on layer '%s', but this layer is not enabled in your configuration", c, dep)
                             errors = True
-                    collection_depends[c] = deplist.keys()
+                    collection_depends[c] = list(depDict.keys())
                 else:
                     collection_depends[c] = []
 
+                # Check recommends and store information for priority calculation
+                recs = self.data.getVar("LAYERRECOMMENDS_%s" % c, True)
+                if recs:
+                    try:
+                        recDict = bb.utils.explode_dep_versions2(recs)
+                    except bb.utils.VersionStringException as vse:
+                        bb.fatal('Error parsing LAYERRECOMMENDS_%s: %s' % (c, str(vse)))
+                    for rec, oplist in list(recDict.items()):
+                        if rec in collection_list:
+                            if oplist:
+                                opstr = oplist[0]
+                                layerver = self.data.getVar("LAYERVERSION_%s" % rec, True)
+                                if layerver:
+                                    (op, recver) = opstr.split()
+                                    try:
+                                        res = bb.utils.vercmp_string_op(layerver, recver, op)
+                                    except bb.utils.VersionStringException as vse:
+                                        bb.fatal('Error parsing LAYERRECOMMENDS_%s: %s' % (c, str(vse)))
+                                    if not res:
+                                        parselog.debug(3,"Layer '%s' recommends version %s of layer '%s', but version %s is currently enabled in your configuration. Check that you are using the correct matching versions/branches of these two layers.", c, opstr, rec, layerver)
+                                        continue
+                                else:
+                                    parselog.debug(3,"Layer '%s' recommends version %s of layer '%s', which exists in your configuration but does not specify a version. Check that you are using the correct matching versions/branches of these two layers.", c, opstr, rec)
+                                    continue
+                            parselog.debug(3,"Layer '%s' recommends layer '%s', so we are adding it", c, rec)
+                            collection_depends[c].append(rec)
+                        else:
+                            parselog.debug(3,"Layer '%s' recommends layer '%s', but this layer is not enabled in your configuration", c, rec)
+
             # Recursively work out collection priorities based on dependencies
             def calc_layer_priority(collection):
                 if not collection_priorities[collection]:
@@ -1421,7 +1452,8 @@ class BBCooker:
         if not task.startswith("do_"):
             task = "do_%s" % task
 
-        packages = ["%s:%s" % (target, task) for target in targets]
+        packages = [target if ':' in target else '%s:%s' % (target, task) for target in targets]
+
         bb.event.fire(bb.event.BuildInit(packages), self.expanded_data)
 
         taskdata, runlist = self.buildTaskData(targets, task, self.configuration.abort)
@@ -1586,7 +1618,7 @@ class BBCooker:
 
         return True
 
-    def checkPackages(self, pkgs_to_build):
+    def checkPackages(self, pkgs_to_build, task=None):
 
         # Return a copy, don't modify the original
         pkgs_to_build = pkgs_to_build[:]
@@ -1602,7 +1634,7 @@ class BBCooker:
         if 'world' in pkgs_to_build:
             pkgs_to_build.remove('world')
             for mc in self.multiconfigs:
-                bb.providers.buildWorldTargetList(self.recipecaches[mc])
+                bb.providers.buildWorldTargetList(self.recipecaches[mc], task)
                 for t in self.recipecaches[mc].world_target:
                     if mc:
                         t = "multiconfig:" + mc + ":" + t

bitbake/lib/bb/cookerdata.py

@@ -270,9 +270,17 @@ class CookerDataBuilder(object):
 
             bb.event.fire(bb.event.ConfigParsed(), self.data)
 
-            if self.data.getVar("BB_INVALIDCONF", False) is True:
+            reparse_cnt = 0
+            while self.data.getVar("BB_INVALIDCONF", False) is True:
+                if reparse_cnt > 20:
+                    logger.error("Configuration has been re-parsed over 20 times, "
+                                 "breaking out of the loop...")
+                    raise Exception("Too deep config re-parse loop. Check locations where "
+                                    "BB_INVALIDCONF is being set (ConfigParsed event handlers)")
                 self.data.setVar("BB_INVALIDCONF", False)
                 self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
+                reparse_cnt += 1
+                bb.event.fire(bb.event.ConfigParsed(), self.data)
 
             bb.parse.init_parser(self.data)
             self.data_hash = self.data.get_hash()
@@ -284,7 +292,7 @@ class CookerDataBuilder(object):
                 bb.event.fire(bb.event.ConfigParsed(), mcdata)
                 self.mcdata[config] = mcdata
 
-        except SyntaxError:
+        except (SyntaxError, bb.BBHandledException):
             raise bb.BBHandledException
         except bb.data_smart.ExpansionError as e:
             logger.error(str(e))

bitbake/lib/bb/event.py

@@ -119,21 +119,28 @@ def print_ui_queue():
     logger = logging.getLogger("BitBake")
     if not _uiready:
         from bb.msg import BBLogFormatter
-        console = logging.StreamHandler(sys.stdout)
-        console.setFormatter(BBLogFormatter("%(levelname)s: %(message)s"))
-        logger.handlers = [console]
+        stdout = logging.StreamHandler(sys.stdout)
+        stderr = logging.StreamHandler(sys.stderr)
+        formatter = BBLogFormatter("%(levelname)s: %(message)s")
+        stdout.setFormatter(formatter)
+        stderr.setFormatter(formatter)
 
         # First check to see if we have any proper messages
         msgprint = False
         for event in ui_queue:
             if isinstance(event, logging.LogRecord):
                 if event.levelno > logging.DEBUG:
+                    if event.levelno >= logging.WARNING:
+                        logger.handlers = [stderr]
+                    else:
+                        logger.handlers = [stdout]
                     logger.handle(event)
                     msgprint = True
         if msgprint:
             return
 
         # Nope, so just print all of the messages we have (including debug messages)
+        logger.handlers = [stdout]
         for event in ui_queue:
             if isinstance(event, logging.LogRecord):
                 logger.handle(event)

bitbake/lib/bb/fetch2/__init__.py

@@ -761,6 +761,7 @@ def get_srcrev(d, method_name='sortable_revision'):
     if not format:
         raise FetchError("The SRCREV_FORMAT variable must be set when multiple SCMs are used.")
 
+    name_to_rev = {}
     seenautoinc = False
     for scm in scms:
         ud = urldata[scm]
@@ -769,7 +770,16 @@ def get_srcrev(d, method_name='sortable_revision'):
             seenautoinc = seenautoinc or autoinc
             if len(rev) > 10:
                 rev = rev[:10]
-            format = format.replace(name, rev)
+            name_to_rev[name] = rev
+    # Replace names by revisions in the SRCREV_FORMAT string. The approach used
+    # here can handle names being prefixes of other names and names appearing
+    # as substrings in revisions (in which case the name should not be
+    # expanded). The '|' regular expression operator tries matches from left to
+    # right, so we need to sort the names with the longest ones first.
+    names_descending_len = sorted(name_to_rev, key=len, reverse=True)
+    name_to_rev_re = "|".join(re.escape(name) for name in names_descending_len)
+    format = re.sub(name_to_rev_re, lambda match: name_to_rev[match.group(0)], format)
+
     if seenautoinc:
        format = "AUTOINC+" + format
 
@@ -1410,7 +1420,13 @@ class FetchMethod(object):
 
         # If 'subdir' param exists, create a dir and use it as destination for unpack cmd
         if 'subdir' in urldata.parm:
-            unpackdir = '%s/%s' % (rootdir, urldata.parm.get('subdir'))
+            subdir = urldata.parm.get('subdir')
+            if os.path.isabs(subdir):
+                if not os.path.realpath(subdir).startswith(os.path.realpath(rootdir)):
+                    raise UnpackError("subdir argument isn't a subdirectory of unpack root %s" % rootdir, urldata.url)
+                unpackdir = subdir
+            else:
+                unpackdir = os.path.join(rootdir, subdir)
             bb.utils.mkdirhier(unpackdir)
         else:
             unpackdir = rootdir

bitbake/lib/bb/fetch2/npm.py

@@ -165,7 +165,9 @@ class Npm(FetchMethod):
             pdata = json.loads('\n'.join(datalines))
         return pdata
 
-    def _getdependencies(self, pkg, data, version, d, ud, optional=False):
+    def _getdependencies(self, pkg, data, version, d, ud, optional=False, fetchedlist=None):
+        if fetchedlist is None:
+            fetchedlist = []
         pkgfullname = pkg
         if version != '*' and not '/' in version:
             pkgfullname += "@'%s'" % version
@@ -187,7 +189,9 @@ class Npm(FetchMethod):
         outputurl = pdata['dist']['tarball']
         data[pkg] = {}
         data[pkg]['tgz'] = os.path.basename(outputurl)
-        self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False)
+        if not outputurl in fetchedlist:
+            self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False)
+            fetchedlist.append(outputurl)
 
         dependencies = pdata.get('dependencies', {})
         optionalDependencies = pdata.get('optionalDependencies', {})
@@ -200,9 +204,9 @@ class Npm(FetchMethod):
             else:
                 depsfound[dep] = dependencies[dep]
         for dep, version in optdepsfound.items():
-            self._getdependencies(dep, data[pkg]['deps'], version, d, ud, optional=True)
+            self._getdependencies(dep, data[pkg]['deps'], version, d, ud, optional=True, fetchedlist=fetchedlist)
         for dep, version in depsfound.items():
-            self._getdependencies(dep, data[pkg]['deps'], version, d, ud)
+            self._getdependencies(dep, data[pkg]['deps'], version, d, ud, fetchedlist=fetchedlist)
 
     def _getshrinkeddependencies(self, pkg, data, version, d, ud, lockdown, manifest, toplevel=True):
         logger.debug(2, "NPM shrinkwrap file is %s" % data)

bitbake/lib/bb/providers.py

@@ -402,7 +402,7 @@ def getRuntimeProviders(dataCache, rdepend):
     return rproviders
 
 
-def buildWorldTargetList(dataCache):
+def buildWorldTargetList(dataCache, task=None):
     """
     Build package list for "bitbake world"
     """
@@ -413,6 +413,9 @@ def buildWorldTargetList(dataCache):
     for f in dataCache.possible_world:
         terminal = True
         pn = dataCache.pkg_fn[f]
+        if task and task not in dataCache.task_deps[f]['tasks']:
+            logger.debug(2, "World build skipping %s as task %s doesn't exist", f, task)
+            terminal = False
 
         for p in dataCache.pn_provides[pn]:
             if p.startswith('virtual/'):

bitbake/lib/bb/runqueue.py

@@ -1555,6 +1555,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
                 logger.debug(1, 'Considering %s: %s' % (tid, str(self.rqdata.runtaskentries[tid].revdeps)))
 
                 if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
+                    if tid in self.rq.scenequeue_notcovered:
+                        continue
                     found = True
                     self.rq.scenequeue_covered.add(tid)
 
@@ -2229,11 +2231,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
         #        revdeps = self.sq_revdeps[tid]
         #        bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
 
-        # Convert scenequeue_covered task numbers into full taskgraph ids
-        oldcovered = self.scenequeue_covered
-        self.rq.scenequeue_covered = set()
-        for task in oldcovered:
-            self.rq.scenequeue_covered.add(task)
+        self.rq.scenequeue_covered = self.scenequeue_covered
+        self.rq.scenequeue_notcovered = self.scenequeue_notcovered
 
         logger.debug(1, 'We can skip tasks %s', sorted(self.rq.scenequeue_covered))
 

bitbake/lib/bb/tests/fetch.py

@@ -508,6 +508,15 @@ class FetcherLocalTest(FetcherTest):
         tree = self.fetchUnpack(['file://dir/subdir/e;subdir=bar'])
         self.assertEqual(tree, ['bar/dir/subdir/e'])
 
+    def test_local_absolutedir(self):
+        # Unpacking to an absolute path that is a subdirectory of the root
+        # should work
+        tree = self.fetchUnpack(['file://a;subdir=%s' % os.path.join(self.unpackdir, 'bar')])
+
+        # Unpacking to an absolute path outside of the root should fail
+        with self.assertRaises(bb.fetch2.UnpackError):
+            self.fetchUnpack(['file://a;subdir=/bin/sh'])
+
 class FetcherNetworkTest(FetcherTest):
 
     if os.environ.get("BB_SKIP_NETTESTS") == "yes":

bitbake/lib/bb/ui/buildinfohelper.py

@@ -43,6 +43,7 @@ from orm.models import Package, Package_File, Target_Installed_Package, Target_F
 from orm.models import Task_Dependency, Package_Dependency
 from orm.models import Recipe_Dependency, Provides
 from orm.models import Project, CustomImagePackage, CustomImageRecipe
+from orm.models import signal_runbuilds
 
 from bldcontrol.models import BuildEnvironment, BuildRequest
 
@@ -234,6 +235,7 @@ class ORMWrapper(object):
         build.completed_on = timezone.now()
         build.outcome = outcome
         build.save()
+        signal_runbuilds()
 
     def update_target_set_license_manifest(self, target, license_manifest_path):
         target.license_manifest_path = license_manifest_path
@@ -449,7 +451,11 @@ class ORMWrapper(object):
             # note that this is different
             buildrequest = BuildRequest.objects.get(pk = br_id)
             for brl in buildrequest.brlayer_set.all():
-                localdirname = os.path.join(bc.getGitCloneDirectory(brl.giturl, brl.commit), brl.dirpath)
+                if brl.local_source_dir:
+                    localdirname = os.path.join(brl.local_source_dir,
+                                                brl.dirpath)
+                else:
+                    localdirname = os.path.join(bc.getGitCloneDirectory(brl.giturl, brl.commit), brl.dirpath)
                 # we get a relative path, unless running in HEAD mode where the path is absolute
                 if not localdirname.startswith("/"):
                     localdirname = os.path.join(bc.be.sourcedir, localdirname)
@@ -979,8 +985,6 @@ class BuildInfoHelper(object):
     def _get_layer_version_for_path(self, path):
         self._ensure_build()
 
-        assert path.startswith("/")
-
         def _slkey_interactive(layer_version):
             assert isinstance(layer_version, Layer_Version)
             return len(layer_version.local_path)
@@ -1354,6 +1358,7 @@ class BuildInfoHelper(object):
         self._ensure_build()
         self.internal_state['build'].outcome = Build.CANCELLED
         self.internal_state['build'].save()
+        signal_runbuilds()
 
     def store_dependency_information(self, event):
         assert '_depgraph' in vars(event)
@@ -1521,9 +1526,7 @@ class BuildInfoHelper(object):
             return
 
         br_id, be_id = self.brbe.split(":")
-        be = BuildEnvironment.objects.get(pk = be_id)
-        be.lock = BuildEnvironment.LOCK_LOCK
-        be.save()
+
         br = BuildRequest.objects.get(pk = br_id)
 
         # if we're 'done' because we got cancelled update the build outcome
@@ -1541,6 +1544,11 @@ class BuildInfoHelper(object):
             br.state = BuildRequest.REQ_FAILED
         br.save()
 
+        be = BuildEnvironment.objects.get(pk = be_id)
+        be.lock = BuildEnvironment.LOCK_FREE
+        be.save()
+        signal_runbuilds()
+
     def store_log_error(self, text):
         mockevent = MockEvent()
         mockevent.levelno = formatter.ERROR

bitbake/lib/bb/ui/knotty.py

@@ -170,6 +170,7 @@ class TerminalFilter(object):
         self.interactive = sys.stdout.isatty()
         self.footer_present = False
         self.lastpids = []
+        self.lasttime = None
         self.quiet = quiet
 
         if not self.interactive:
@@ -226,6 +227,10 @@ class TerminalFilter(object):
         activetasks = self.helper.running_tasks
         failedtasks = self.helper.failed_tasks
         runningpids = self.helper.running_pids
+        currenttime = time.time()
+        if not self.lasttime or (currenttime - self.lasttime > 5):
+            self.helper.needUpdate = True
+            self.lasttime = currenttime
         if self.footer_present and not self.helper.needUpdate:
             return
         self.helper.needUpdate = False
@@ -250,7 +255,11 @@ class TerminalFilter(object):
                     activetasks[t]["progressbar"] = pbar
                 tasks.append((pbar, progress, rate, start_time))
             else:
-                tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+                start_time = activetasks[t].get("starttime", None)
+                if start_time:
+                    tasks.append("%s - %ds (pid %s)" % (activetasks[t]["title"], currenttime - start_time, t))
+                else:
+                    tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
 
         if self.main.shutdown:
             content = "Waiting for %s running tasks to finish:" % len(activetasks)
@@ -271,7 +280,7 @@ class TerminalFilter(object):
             progress = self.helper.tasknumber_current - 1
             if progress < 0:
                 progress = 0
-            self.main_progress.update(progress)
+            content = self.main_progress.update(progress)
             print('')
         lines = 1 + int(len(content) / (self.columns + 1))
         if not self.quiet:
@@ -285,11 +294,10 @@ class TerminalFilter(object):
                     pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
                     if progress > -1:
                         pbar.setextra(rate)
-                        output = pbar.update(progress)
+                        content = pbar.update(progress)
                     else:
-                        output = pbar.update(1)
-                    if not output or (len(output) <= pbar.term_width):
-                        print('')
+                        content = pbar.update(1)
+                    print('')
                 else:
                     content = "%s: %s" % (tasknum, task)
                     print(content)

bitbake/lib/bb/ui/uihelper.py

@@ -35,25 +35,25 @@ class BBUIHelper:
             self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time() }
             self.running_pids.append(event.pid)
             self.needUpdate = True
-        if isinstance(event, bb.build.TaskSucceeded):
+        elif isinstance(event, bb.build.TaskSucceeded):
             del self.running_tasks[event.pid]
             self.running_pids.remove(event.pid)
             self.needUpdate = True
-        if isinstance(event, bb.build.TaskFailedSilent):
+        elif isinstance(event, bb.build.TaskFailedSilent):
             del self.running_tasks[event.pid]
             self.running_pids.remove(event.pid)
             # Don't add to the failed tasks list since this is e.g. a setscene task failure
             self.needUpdate = True
-        if isinstance(event, bb.build.TaskFailed):
+        elif isinstance(event, bb.build.TaskFailed):
             del self.running_tasks[event.pid]
             self.running_pids.remove(event.pid)
             self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
             self.needUpdate = True
-        if isinstance(event, bb.runqueue.runQueueTaskStarted) or isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+        elif isinstance(event, bb.runqueue.runQueueTaskStarted) or isinstance(event, bb.runqueue.sceneQueueTaskStarted):
             self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
             self.tasknumber_total = event.stats.total
             self.needUpdate = True
-        if isinstance(event, bb.build.TaskProgress):
+        elif isinstance(event, bb.build.TaskProgress):
             if event.pid > 0:
                 self.running_tasks[event.pid]['progress'] = event.progress
                 self.running_tasks[event.pid]['rate'] = event.rate
@@ -62,4 +62,3 @@ class BBUIHelper:
     def getTasks(self):
         self.needUpdate = False
         return (self.running_tasks, self.failed_tasks)
-

bitbake/lib/toaster/bldcontrol/localhostbecontroller.py

@@ -156,7 +156,7 @@ class LocalhostBEController(BuildEnvironmentController):
                 try:
                     localremotes = self._shellcmd("git remote -v",
                                                   localdirname)
-                    if not giturl in localremotes:
+                    if not giturl in localremotes and commit != 'HEAD':
                         raise BuildSetupException("Existing git repository at %s, but with different remotes ('%s', expected '%s'). Toaster will not continue out of fear of damaging something." % (localdirname, ", ".join(localremotes.split("\n")), giturl))
                 except ShellCmdException:
                     # our localdirname might not be a git repository

bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py

@@ -1,31 +1,27 @@
-from django.core.management.base import NoArgsCommand, CommandError
+from django.core.management.base import NoArgsCommand
 from django.db import transaction
 from django.db.models import Q
 
 from bldcontrol.bbcontroller import getBuildEnvironmentController
-from bldcontrol.bbcontroller import ShellCmdException, BuildSetupException
 from bldcontrol.models import BuildRequest, BuildEnvironment
 from bldcontrol.models import BRError, BRVariable
 
-from orm.models import Build, ToasterSetting, LogMessage, Target
+from orm.models import Build, LogMessage, Target
 
-import os
 import logging
-import time
-import sys
 import traceback
+import signal
 
 logger = logging.getLogger("toaster")
 
 class Command(NoArgsCommand):
-    args    = ""
-    help    = "Schedules and executes build requests as possible."
-    "Does not return (interrupt with Ctrl-C)"
-
+    args = ""
+    help = "Schedules and executes build requests as possible. "\
+           "Does not return (interrupt with Ctrl-C)"
 
     @transaction.atomic
     def _selectBuildEnvironment(self):
-        bec = getBuildEnvironmentController(lock = BuildEnvironment.LOCK_FREE)
+        bec = getBuildEnvironmentController(lock=BuildEnvironment.LOCK_FREE)
         bec.be.lock = BuildEnvironment.LOCK_LOCK
         bec.be.save()
         return bec
@@ -75,17 +71,15 @@ class Command(NoArgsCommand):
             else:
                 errmsg = str(e)
 
-            BRError.objects.create(req = br,
-                    errtype = str(type(e)),
-                    errmsg = errmsg,
-                    traceback = traceback.format_exc())
+            BRError.objects.create(req=br, errtype=str(type(e)), errmsg=errmsg,
+                                   traceback=traceback.format_exc())
             br.state = BuildRequest.REQ_FAILED
             br.save()
             bec.be.lock = BuildEnvironment.LOCK_FREE
             bec.be.save()
 
     def archive(self):
-        for br in BuildRequest.objects.filter(state = BuildRequest.REQ_ARCHIVE):
+        for br in BuildRequest.objects.filter(state=BuildRequest.REQ_ARCHIVE):
             if br.build == None:
                 br.state = BuildRequest.REQ_FAILED
             else:
@@ -102,14 +96,13 @@ class Command(NoArgsCommand):
                                        BuildRequest.REQ_COMPLETED,
                                        BuildRequest.REQ_CANCELLING]) &
             Q(lock=BuildEnvironment.LOCK_LOCK) &
-            Q(updated__lt=timezone.now() - timedelta(seconds = 30))
+            Q(updated__lt=timezone.now() - timedelta(seconds=30))
         ).update(lock=BuildEnvironment.LOCK_FREE)
 
 
         # update all Builds that were in progress and failed to start
-        for br in BuildRequest.objects.filter(
-            state=BuildRequest.REQ_FAILED,
-            build__outcome=Build.IN_PROGRESS):
+        for br in BuildRequest.objects.filter(state=BuildRequest.REQ_FAILED,
+                                              build__outcome=Build.IN_PROGRESS):
             # transpose the launch errors in ToasterExceptions
             br.build.outcome = Build.FAILED
             for brerror in br.brerror_set.all():
@@ -126,7 +119,7 @@ class Command(NoArgsCommand):
 
 
         # update all BuildRequests without a build created
-        for br in BuildRequest.objects.filter(build = None):
+        for br in BuildRequest.objects.filter(build=None):
             br.build = Build.objects.create(project=br.project,
                                             completed_on=br.updated,
                                             started_on=br.created)
@@ -151,10 +144,10 @@ class Command(NoArgsCommand):
 
         # Make sure the LOCK is removed for builds which have been fully
         # cancelled
-        for br in BuildRequest.objects.filter(
-            Q(build__outcome=Build.CANCELLED) &
-            Q(state=BuildRequest.REQ_CANCELLING) &
-            ~Q(environment=None)):
+        for br in BuildRequest.objects.filter(\
+                      Q(build__outcome=Build.CANCELLED) &
+                      Q(state=BuildRequest.REQ_CANCELLING) &
+                      ~Q(environment=None)):
             br.environment.lock = BuildEnvironment.LOCK_FREE
             br.environment.save()
 
@@ -175,6 +168,10 @@ class Command(NoArgsCommand):
             logger.warn("runbuilds: schedule exception %s" % str(e))
 
     def handle_noargs(self, **options):
+        self.runbuild()
+
+        signal.signal(signal.SIGUSR1, lambda sig, frame: None)
+
         while True:
+            signal.pause()
             self.runbuild()
-            time.sleep(1)

bitbake/lib/toaster/bldcontrol/migrations/0007_brlayers_optional_gitinfo.py

@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bldcontrol', '0006_brlayer_local_source_dir'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='brlayer',
+            name='commit',
+            field=models.CharField(max_length=254, null=True),
+        ),
+        migrations.AlterField(
+            model_name='brlayer',
+            name='dirpath',
+            field=models.CharField(max_length=254, null=True),
+        ),
+        migrations.AlterField(
+            model_name='brlayer',
+            name='giturl',
+            field=models.CharField(max_length=254, null=True),
+        ),
+    ]

bitbake/lib/toaster/orm/models.py

@@ -33,9 +33,10 @@ from django.conf import settings
 import django.db.models.signals
 
 import sys
-import os.path
+import os
 import re
 import itertools
+from signal import SIGUSR1
 
 import logging
 logger = logging.getLogger("toaster")
@@ -376,6 +377,8 @@ class Project(models.Model):
             except ProjectVariable.DoesNotExist:
                 pass
             br.save()
+            signal_runbuilds()
+
         except Exception:
             # revert the build request creation since we're not done cleanly
             br.delete()
@@ -1736,6 +1739,11 @@ def invalidate_cache(**kwargs):
     except Exception as e:
       logger.warning("Problem with cache backend: Failed to clear cache: %s" % e)
 
+def signal_runbuilds():
+    """Send SIGUSR1 to runbuilds process"""
+    with open(os.path.join(os.getenv('BUILDDIR'), '.runbuilds.pid')) as pidf:
+        os.kill(int(pidf.read()), SIGUSR1)
+
 django.db.models.signals.post_save.connect(invalidate_cache)
 django.db.models.signals.post_delete.connect(invalidate_cache)
 django.db.models.signals.m2m_changed.connect(invalidate_cache)

bitbake/lib/toaster/tests/db/test_db.py

@@ -0,0 +1,55 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2016 Damien Lespiau
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import sys
+
+try:
+    from StringIO import StringIO
+except ImportError:
+    from io import StringIO
+
+from contextlib import contextmanager
+
+from django.core import management
+from django.test import TestCase
+
+
+@contextmanager
+def capture(command, *args, **kwargs):
+    out, sys.stdout = sys.stdout, StringIO()
+    command(*args, **kwargs)
+    sys.stdout.seek(0)
+    yield sys.stdout.read()
+    sys.stdout = out
+
+
+def makemigrations():
+    management.call_command('makemigrations')
+
+
+class MigrationTest(TestCase):
+
+    def testPendingMigration(self):
+        """Make sure there's no pending migration."""
+
+        with capture(makemigrations) as output:
+            self.assertEqual(output, "No changes detected\n")

bitbake/lib/toaster/toastergui/templates/basebuildpage.html

@@ -38,10 +38,13 @@
             $(this).parent().addClass("active");
           } else {
             /* Special case the configvar as this is part of configuration
-             * page but is a separate url
+             * page but is a separate url, and the direct links to errors
+             * and warnings, which are part of the build dashboard
              */
             if (window.location.pathname === configVarUrl){
               $("#menu-configuration").addClass("active");
+            } else if (currentUrl.indexOf('error') > 1 || currentUrl.indexOf('warning') > 1){
+              $("#menu-dashboard").addClass("active");
             } else {
               $(this).parent().removeClass("active");
             }
@@ -57,7 +60,7 @@
     <div class="row">
       <div id="nav" class="col-md-2">
         <ul class="nav nav-pills nav-stacked" id="build-menu">
-          <li
+          <li id="menu-dashboard"
             {% if request.resolver_match.url_name == 'builddashboard'  %}
               class="active"
             {% endif %} >

bitbake/lib/toaster/toastergui/templates/builddashboard.html

@@ -235,7 +235,7 @@
                           </a>
 
                   {% elif exectask.count > 1%}
-                      <a class="text-danger" href="{% url "tasks" build.id %}?filter=outcome%3A4">{{exectask.count}}</a>
+                      <a class="text-danger" href="{% url "tasks" build.id %}?limit=25&amp;page=1&amp;orderby=order&amp;filter=task_outcome:failed&amp;default_orderby=order&amp;filter_value=on&amp;">{{exectask.count}}</a>
                   {% endif %}
                   </dd>
               {% endif %}
@@ -244,12 +244,12 @@
               Tasks executed
               <span class="glyphicon glyphicon-question-sign get-help" title="'Executed' tasks are those that need to be run in order to generate the task output"></span>
           </dt>
-          <dd><a href="{% url 'tasks' build.pk %}?filter=task_executed%3A1&amp;count=25&amp;search=&amp;page=1&amp;orderby=order%3A%2B">{% query build.task_build task_executed=1 order__gt=0 as exectask%}{{exectask.count}}</a></dd>
+          <dd><a href="{% url 'tasks' build.pk %}?limit=25&amp;page=1&amp;orderby=order&amp;filter=execution_outcome:executed&amp;default_orderby=order&amp;filter_value=on&amp;">{% query build.task_build task_executed=1 order__gt=0 as exectask%}{{exectask.count}}</a></dd>
           <dt>
               Tasks not executed
               <span class="glyphicon glyphicon-question-sign get-help" title="'Not executed' tasks don't need to run because their outcome is provided by another task"></span>
           </dt>
-          <dd><a href="{% url 'tasks' build.pk %}?filter=task_executed%3A0&amp;count=25&amp;search=&amp;page=1&amp;orderby=order%3A%2B">{% query build.task_build task_executed=0 order__gt=0 as noexectask%}{{noexectask.count}}</a></dd>
+          <dd><a href="{% url 'tasks' build.pk %}?limit=25&amp;page=1&amp;orderby=order&amp;filter=execution_outcome:not_executed&amp;default_orderby=order&amp;filter_value=on&amp;">{% query build.task_build task_executed=0 order__gt=0 as noexectask%}{{noexectask.count}}</a></dd>
           <dt>
               Reuse
               <span class="glyphicon glyphicon-question-sign get-help" title="The percentage of 'not executed' tasks over the total number of tasks, which is a measure of the efficiency of your build"></span>

bitbake/lib/toaster/toastergui/templates/customrecipe.html

@@ -176,8 +176,6 @@
         <a href="{% url 'projectbuilds' project.id%}">{{last_build.completed_on|date:"d/m/y H:i"}}</a>
         </dd>
         {% endif %}
-        <dt>Layer</dt>
-        <dd><a href="{% url 'layerdetails' project.id recipe.layer_version.pk %}">{{recipe.layer_version.layer.name}}</a></dd>
         <dt>Based on</dt>
         <dd><a href="{% url 'recipedetails' project.id recipe.base_recipe.pk %}">{{recipe.base_recipe.name}}</a></dd>
         {% if recipe.get_last_successful_built_target %}

documentation/dev-manual/dev-manual-common-tasks.xml

@@ -2297,6 +2297,8 @@
                         However, you might still want to make some adjustments.
                         For example, you can set
                         <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
+                        or
+                        <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
                         to pass any needed configure options that are specific
                         to the recipe.</para></listitem>
                     <listitem><para><emphasis>CMake:</emphasis>
@@ -3011,8 +3013,10 @@
                     You do not need to add a <filename>do_compile</filename> step since by default BitBake
                     starts the <filename>make</filename> command to compile the application.
                     If you need additional <filename>make</filename> options, you should store them in the
-                    <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OEMAKE'>EXTRA_OEMAKE</ulink></filename>
-                    variable.
+                    <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OEMAKE'><filename>EXTRA_OEMAKE</filename></ulink>
+                    or
+                    <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
+                    variables.
                     BitBake passes these options into the GNU <filename>make</filename> invocation.
                     Note that a <filename>do_install</filename> task is still required.
                     Otherwise, BitBake runs an empty <filename>do_install</filename> task by default.
@@ -3946,8 +3950,9 @@
                         <filename>webkit</filename>).
                         </para></listitem>
                     <listitem><para>
-                        Using QEMU in usermode might not work properly when
-                        running 64-bit binaries under 32-bit host machines.
+                        QEMU might not work properly when using QEMU in
+                        usermode and running 64-bit binaries under 32-bit
+                        host machines.
                         In particular, "qemumips64" is known to not work under
                         i686.
                         </para></listitem>

documentation/dev-manual/dev-manual-model.xml

@@ -649,13 +649,13 @@
         </para>
 
         <section id='use-devtool-to-integrate-new-code'>
-            <title>Use <filename>devtool add</filename> to Integrate New Code</title>
+            <title>Use <filename>devtool add</filename> to Add an Application</title>
 
             <para>
                 The <filename>devtool add</filename> command generates
                 a new recipe based on existing source code.
                 This command takes advantage of the
-                <link linkend='devtool-the-workspace-layer-structure'>workspace</link>
+                <ulink url='&YOCTO_DOCS_DEV_URL;#devtool-the-workspace-layer-structure'>workspace</ulink>
                 layer that many <filename>devtool</filename> commands
                 use.
                 The command is flexible enough to allow you to extract source
@@ -721,7 +721,8 @@
                                 and needs to be extracted to some
                                 local area - this time outside of the default
                                 workspace.
-                                As always, if required <filename>devtool</filename> creates
+                                If required, <filename>devtool</filename>
+                                always creates
                                 a Git repository locally during the extraction.
                                 Furthermore, the first positional argument
                                 <replaceable>srctree</replaceable> in this case
@@ -788,10 +789,6 @@
                         <para>If you need to take the build output and eventually
                         move it to the target hardware, you would use
                         <filename>devtool build</filename>:
-                        <note>
-                            You could use <filename>bitbake</filename> to build
-                            the recipe as well.
-                        </note>
                         <literallayout class='monospaced'>
      $ devtool build <replaceable>recipe</replaceable>
                         </literallayout></para>
@@ -831,49 +828,44 @@
                         However, <filename>devtool</filename> does not provide a
                         specific command that allows you to do this.
                         </para></listitem>
-                    <listitem><para><emphasis>Optionally Update the Recipe With Patch Files</emphasis>:
-                        Once you are satisfied with the recipe, if you have made
-                        any changes to the source tree that you want to have
-                        applied by the recipe, you need to generate patches
-                        from those changes.
-                        You do this before moving the recipe
-                        to its final layer and cleaning up the workspace area
-                        <filename>devtool</filename> uses.
-                        This optional step is especially relevant if you are
-                        using or adding third-party software.</para>
-                        <para>To convert commits created using Git to patch files,
-                        use the <filename>devtool update-recipe</filename> command.
+                    <listitem><para>
+                        <emphasis>Finish Your Work With the Recipe</emphasis>:
+                        The <filename>devtool finish</filename> command creates
+                        any patches corresponding to commits in the local
+                        Git repository, moves the new recipe to a more permanent
+                        layer, and then resets the recipe so that the recipe is
+                        built normally rather than from the workspace.
+                        <literallayout class='monospaced'>
+     $ devtool finish <replaceable>recipe layer</replaceable>
+                        </literallayout>
                         <note>
                             Any changes you want to turn into patches must be
                             committed to the Git repository in the source tree.
+                        </note></para>
+
+                        <para>As mentioned, the <filename>devtool finish</filename>
+                        command moves the final recipe to its permanent layer.
+                        </para>
+
+                        <para>As a final process of the
+                        <filename>devtool finish</filename> command, the state
+                        of the standard layers and the upstream source is
+                        restored so that you can build the recipe from those
+                        areas rather than the workspace.
+                        <note>
+                            You can use the <filename>devtool reset</filename>
+                            command to put things back should you decide you
+                            do not want to proceed with your work.
+                            If you do use this command, realize that the source
+                            tree is preserved.
                         </note>
-                        <literallayout class='monospaced'>
-     $ devtool update-recipe <replaceable>recipe</replaceable>
-                        </literallayout>
-                        </para></listitem>
-                    <listitem><para><emphasis>Move the Recipe to its Permanent Layer</emphasis>:
-                        Before cleaning up the workspace, you need to move the
-                        final recipe to its permanent layer.
-                        You must do this before using the
-                        <filename>devtool reset</filename> command if you want to
-                        retain the recipe.
-                        </para></listitem>
-                    <listitem><para><emphasis>Reset the Recipe</emphasis>:
-                        As a final step, you can restore the state such that
-                        standard layers and the upstream source is used to build
-                        the recipe rather than data in the workspace.
-                        To reset the recipe, use the <filename>devtool reset</filename>
-                        command:
-                        <literallayout class='monospaced'>
-     $ devtool reset <replaceable>recipe</replaceable>
-                        </literallayout>
                         </para></listitem>
                 </orderedlist>
             </para>
         </section>
 
         <section id='devtool-use-devtool-modify-to-enable-work-on-code-associated-with-an-existing-recipe'>
-            <title>Use <filename>devtool modify</filename> to Enable Work on Code Associated with an Existing Recipe</title>
+            <title>Use <filename>devtool modify</filename> to Modify the Source of an Existing Component</title>
 
             <para>
                 The <filename>devtool modify</filename> command prepares the
@@ -1028,17 +1020,12 @@
                     <listitem><para><emphasis>Build the Recipe</emphasis>:
                         Once you have updated the source files, you can build
                         the recipe.
-                        You can either use <filename>devtool build</filename> or
-                        <filename>bitbake</filename>.
-                        Either method produces build output that is stored
-                        in
-                        <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>.
                         </para></listitem>
                     <listitem><para><emphasis>Deploy the Build Output</emphasis>:
                         When you use the <filename>devtool build</filename>
-                        command or <filename>bitbake</filename> to build out your
-                        recipe, you probably want to see if the resulting build
-                        output works as expected on target hardware.
+                        command to build out your recipe, you probably want to see
+                        if the resulting build output works as expected on target
+                        hardware.
                         <note>
                             This step assumes you have a previously built
                             image that is already either running in QEMU or
@@ -1062,42 +1049,43 @@
                         However, <filename>devtool</filename> does not provide a
                         specific command that allows you to do this.
                         </para></listitem>
-                    <listitem><para><emphasis>Optionally Create Patch Files for Your Changes</emphasis>:
-                        After you have debugged your changes, you can
-                        use <filename>devtool update-recipe</filename> to
-                        generate patch files for all the commits you have
-                        made.
-                        <note>
-                            Patch files are generated only for changes
-                            you have committed.
-                        </note>
+                    <listitem><para>
+                        <emphasis>Finish Your Work With the Recipe</emphasis>:
+                        The <filename>devtool finish</filename> command creates
+                        any patches corresponding to commits in the local
+                        Git repository, updates the recipe to point to them
+                        (or creates a <filename>.bbappend</filename> file to do
+                        so, depending on the specified destination layer), and
+                        then resets the recipe so that the recipe is built normally
+                        rather than from the workspace.
                         <literallayout class='monospaced'>
-     $ devtool update-recipe <replaceable>recipe</replaceable>
+     $ devtool finish <replaceable>recipe layer</replaceable>
                         </literallayout>
-                        By default, the
-                        <filename>devtool update-recipe</filename> command
-                        creates the patch files in a folder named the same
-                        as the recipe beneath the folder in which the recipe
-                        resides, and updates the recipe's
-                        <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
-                        statement to point to the generated patch files.
                         <note>
-                            You can use the
-                            "--append <replaceable>LAYERDIR</replaceable>"
-                            option to cause the command to create append files
-                            in a specific layer rather than the default
-                            recipe layer.
+                            Any changes you want to turn into patches must be
+                            committed to the Git repository in the source tree.
+                        </note></para>
+
+                        <para>Because there is no need to move the recipe,
+                        <filename>devtool finish</filename> either updates the
+                        original recipe in the original layer or the command
+                        creates a <filename>.bbappend</filename> in a different
+                        layer as provided by <replaceable>layer</replaceable>.
+                        </para>
+
+                        <para>As a final process of the
+                        <filename>devtool finish</filename> command, the state
+                        of the standard layers and the upstream source is
+                        restored so that you can build the recipe from those
+                        areas rather than the workspace.
+                        <note>
+                            You can use the <filename>devtool reset</filename>
+                            command to put things back should you decide you
+                            do not want to proceed with your work.
+                            If you do use this command, realize that the source
+                            tree is preserved.
                         </note>
                         </para></listitem>
-                    <listitem><para><emphasis>Restore the Workspace</emphasis>:
-                        The <filename>devtool reset</filename> restores the
-                        state so that standard layers and upstream sources are
-                        used to build the recipe rather than what is in the
-                        workspace.
-                        <literallayout class='monospaced'>
-     $ devtool reset <replaceable>recipe</replaceable>
-                        </literallayout>
-                        </para></listitem>
                 </orderedlist>
             </para>
         </section>
@@ -1297,7 +1285,7 @@
 
      OpenEmbedded development tool
 
-     optional arguments:
+     options:
        --basepath BASEPATH  Base directory of SDK / build directory
        --bbpath BBPATH      Explicitly specify the BBPATH, rather than getting it
                             from the metadata
@@ -1315,11 +1303,12 @@
          status               Show workspace status
          search               Search available recipes
        Working on a recipe in the workspace:
-         build                Build a recipe
          edit-recipe          Edit a recipe file in your workspace
          configure-help       Get help on configure script options
+         build                Build a recipe
          update-recipe        Apply changes from external source tree to recipe
          reset                Remove a recipe from your workspace
+         finish               Finish working on a recipe in your workspace
        Testing changes on target:
          deploy-target        Deploy recipe output files to live target machine
          undeploy-target      Undeploy recipe output files in live target machine
@@ -1339,14 +1328,14 @@
                 <literallayout class='monospaced'>
      $ devtool add --help
      usage: devtool add [-h] [--same-dir | --no-same-dir] [--fetch URI]
-                        [--version VERSION] [--no-git] [--binary] [--also-native]
-                        [--src-subdir SUBDIR]
+                        [--version VERSION] [--no-git] [--autorev] [--binary]
+                        [--also-native] [--src-subdir SUBDIR]
                         [recipename] [srctree] [fetchuri]
 
      Adds a new recipe to the workspace to build a specified source tree. Can
      optionally fetch a remote URI and unpack it to create the source tree.
 
-     positional arguments:
+     arguments:
        recipename            Name for new recipe to add (just name - no version,
                              path or extension). If not specified, will attempt to
                              auto-detect it.
@@ -1357,7 +1346,7 @@
        fetchuri              Fetch the specified URI and extract it to create the
                              source tree
 
-     optional arguments:
+     options:
        -h, --help            show this help message and exit
        --same-dir, -s        Build in same directory as source
        --no-same-dir         Force build in a separate build directory
@@ -1368,6 +1357,8 @@
                              Version to use within recipe (PV)
        --no-git, -g          If fetching source, do not set up source tree as a git
                              repository
+       --autorev, -a         When fetching from a git repository, set SRCREV in the
+                             recipe to a floating revision instead of fixed
        --binary, -b          Treat the source tree as something that should be
                              installed verbatim (no compilation, same directory
                              structure). Useful with binary packages e.g. RPMs.

documentation/mega-manual/mega-manual.xml

@@ -123,6 +123,8 @@
         xmlns:xi="http://www.w3.org/2003/XInclude" href="../sdk-manual/sdk-appendix-obtain.xml"/>
     <xi:include
         xmlns:xi="http://www.w3.org/2003/XInclude" href="../sdk-manual/sdk-appendix-customizing.xml"/>
+    <xi:include
+        xmlns:xi="http://www.w3.org/2003/XInclude" href="../sdk-manual/sdk-appendix-mars.xml"/>
 
 <!-- Includes bsp-guide title image and then bsp-guide chapters -->
 

documentation/poky.ent

@@ -62,7 +62,7 @@
 <!ENTITY OE_INIT_FILE "oe-init-build-env">
 <!ENTITY UBUNTU_HOST_PACKAGES_ESSENTIAL "gawk wget git-core diffstat unzip texinfo gcc-multilib \
      build-essential chrpath socat">
-<!ENTITY FEDORA_HOST_PACKAGES_ESSENTIAL "gawk make wget tar bzip2 gzip python unzip perl patch \
+<!ENTITY FEDORA_HOST_PACKAGES_ESSENTIAL "gawk make wget tar bzip2 gzip python3 unzip perl patch \
      diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
      ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
      findutils which">

documentation/ref-manual/closer-look.xml

@@ -888,7 +888,9 @@
                         class,
                         you can add additional configuration options by using
                         the <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
-                        variable.
+                        or
+                        <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>
+                        variables.
                         For information on how this variable works within
                         that class, see the
                         <filename>meta/classes/autotools.bbclass</filename> file.
@@ -1210,6 +1212,67 @@
             </para>
         </section>
 
+        <section id='stamp-files-and-the-rerunning-of-tasks'>
+            <title>Stamp Files and the Rerunning of Tasks</title>
+
+            <para>
+                For each task that completes successfully, BitBake writes a
+                stamp file into the
+                <link linkend='var-STAMPS_DIR'><filename>STAMPS_DIR</filename></link>
+                directory.
+                The beginning of the stamp file's filename is determined by the
+                <link linkend='var-STAMP'><filename>STAMP</filename></link>
+                variable, and the end of the name consists of the task's name
+                and current
+                <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
+                <note>
+                    This naming scheme assumes that
+                    <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
+                    is "OEBasicHash", which is almost always the case in
+                    current OpenEmbedded.
+                </note>
+                To determine if a task needs to be rerun, BitBake checks if a
+                stamp file with a matching input checksum exists for the task.
+                If such a stamp file exists, the task's output is assumed to
+                exist and still be valid.
+                If the file does not exist, the task is rerun.
+                <note>
+                    <para>The stamp mechanism is more general than the shared
+                    state (sstate) cache mechanism described in the
+                    "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
+                    section.
+                    BitBake avoids rerunning any task that has a valid
+                    stamp file, not just tasks that can be accelerated through
+                    the sstate cache.</para>
+                    <para>However, you should realize that stamp files only
+                    serve as a marker that some work has been done and that
+                    these files do not record task output.
+                    The actual task output would usually be somewhere in
+                    <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>
+                    (e.g. in some recipe's
+                    <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.)
+                    What the sstate cache mechanism adds is a way to cache task
+                    output that can then be shared between build machines.
+                    </para>
+                </note>
+                Since <filename>STAMPS_DIR</filename> is usually a subdirectory
+                of <filename>TMPDIR</filename>, removing
+                <filename>TMPDIR</filename> will also remove
+                <filename>STAMPS_DIR</filename>, which means tasks will
+                properly be rerun to repopulate <filename>TMPDIR</filename>.
+            </para>
+
+            <para>
+                If you want some task to always be considered "out of date",
+                you can mark it with the
+                <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
+                varflag.
+                If some other task depends on such a task, then that task will
+                also always be considered out of date, which might not be what
+                you want.
+            </para>
+        </section>
+
         <section id='setscene-tasks-and-shared-state'>
             <title>Setscene Tasks and Shared State</title>
 
@@ -1528,6 +1591,10 @@
                     Specifies whether or not packagedata will be included in
                     the extensible SDK for all recipes in the "world" target.
                     </para></listitem>
+                <listitem><para><link linkend='var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></link>:
+                    Specifies whether or not the toolchain will be included
+                    when building the extensible SDK.
+                    </para></listitem>
                 <listitem><para><link linkend='var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></link>:
                     A list of variables allowed through from the build system
                     configuration into the extensible SDK configuration.

documentation/ref-manual/faq.xml

@@ -33,9 +33,7 @@
             <para id='faq-not-meeting-requirements'>
                 My development system does not meet the
                 required Git, tar, and Python versions.
-                In particular, I do not have Python 2.7.3 or greater, or
-                I do have Python 3.x, which is specifically not supported by
-                the Yocto Project.
+                In particular, I do not have Python 3.4.0 or greater.
                 Can I still use the Yocto Project?
             </para>
         </question>

documentation/ref-manual/introduction.xml

@@ -47,7 +47,7 @@
             <listitem><para><emphasis>
                 <link linkend='usingpoky'>Using the Yocto Project</link>:</emphasis>
                 Provides an overview of the components that make up the Yocto Project
-                followed by information about debugng images created in the Yocto Project.
+                followed by information about debugging images created in the Yocto Project.
                 </para></listitem>
             <listitem><para><emphasis>
                 <link linkend='closer-look'>A Closer Look at the Yocto Project Development Environment</link>:</emphasis>
@@ -306,7 +306,7 @@
                         Packages needed if you are going to run
                         <filename>oe-selftest</filename>:
                         <literallayout class='monospaced'>
-     $ sudo dnf install GitPython
+     $ sudo dnf install python3-GitPython
                         </literallayout>
                         </para></listitem>
                 </itemizedlist>
@@ -422,8 +422,7 @@
             <itemizedlist>
                 <listitem><para>Git 1.8.3.1 or greater</para></listitem>
                 <listitem><para>tar 1.24 or greater</para></listitem>
-                <listitem><para>Python 2.7.3 or greater not including
-                    Python 3.x, which is not supported.</para></listitem>
+                <listitem><para>Python 3.4.0 or greater</para></listitem>
             </itemizedlist>
         </para>
 

documentation/ref-manual/ref-classes.xml

@@ -161,13 +161,17 @@
                 cross-compilation.
                 You can pass additional parameters to
                 <filename>configure</filename> through the
-                <filename><link linkend='var-EXTRA_OECONF'>EXTRA_OECONF</link></filename> variable.
+                <filename><link linkend='var-EXTRA_OECONF'>EXTRA_OECONF</link></filename>
+                or
+                <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>
+                variables.
                 </para></listitem>
             <listitem><para><link linkend='ref-tasks-compile'><filename>do_compile</filename></link> -
                 Runs <filename>make</filename> with arguments that specify the
                 compiler and linker.
                 You can pass additional arguments through
-                the <filename><link linkend='var-EXTRA_OEMAKE'>EXTRA_OEMAKE</link></filename> variable.
+                the <filename><link linkend='var-EXTRA_OEMAKE'>EXTRA_OEMAKE</link></filename>
+                variable.
                 </para></listitem>
             <listitem><para><link linkend='ref-tasks-install'><filename>do_install</filename></link> -
                 Runs <filename>make install</filename> and passes in
@@ -3317,7 +3321,7 @@
 
     <para>
         You can also see the
-        <ulink url='https://www.freedesktop.org/wiki/Software/systemd/'>Systemd documentation</ulink>
+        <ulink url='http://www.freedesktop.org/wiki/Software/systemd/systemd-boot/'>Systemd-boot documentation</ulink>
         for more information.
     </para>
 </section>
@@ -3726,7 +3730,9 @@
         software that uses the Waf build system.
         You can use the
         <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
-        variable to specify additional configuration options to be passed on
+        or
+        <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>
+        variables to specify additional configuration options to be passed on
         the Waf command line.
     </para>
 </section>

documentation/ref-manual/ref-qa-checks.xml

@@ -918,7 +918,8 @@ can be found then it should be implemented.  I can't find one at the moment.
                     and the upstream change log or release notes.
                     Once you have worked out what the appropriate
                     change is, you can update
-                    <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
+                    <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>,
+                    <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>,
                     or the individual
                     <link linkend='var-PACKAGECONFIG'><filename>PACKAGECONFIG</filename></link>
                     option values accordingly.

documentation/ref-manual/ref-structure.xml

@@ -801,8 +801,9 @@
         <title><filename>build/tmp/stamps/</filename></title>
 
         <para>
-            This directory holds information that BitBake uses for accounting purposes
-            to track what tasks have run and when they have run.
+            This directory holds information that BitBake uses for
+            accounting purposes to track what tasks have run and when they
+            have run.
             The directory is sub-divided by architecture, package name, and
             version.
             Following is an example:
@@ -812,6 +813,13 @@
             Although the files in the directory are empty of data,
             BitBake uses the filenames and timestamps for tracking purposes.
         </para>
+
+        <para>
+            For information on how BitBake uses stamp files to determine if
+            a task should be rerun, see the
+            "<link linkend='stamp-files-and-the-rerunning-of-tasks'>Stamp Files and the Rerunning of Tasks</link>"
+            section.
+        </para>
     </section>
 
     <section id='structure-build-tmp-log'>

documentation/ref-manual/ref-tasks.xml

@@ -334,6 +334,17 @@
             <link linkend='var-FILES'><filename>FILES</filename></link>
             variables.
         </para>
+
+        <para>
+            The <filename>do_package</filename> task, in conjunction with the
+            <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
+            task, also saves some important package metadata.
+            For additional information, see the
+            <link linkend='var-PKGDESTWORK'><filename>PKGDESTWORK</filename></link>
+            variable and the
+            "<link linkend='automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</link>"
+            section.
+        </para>
     </section>
 
     <section id='ref-tasks-package_qa'>
@@ -406,8 +417,11 @@
         <title><filename>do_packagedata</filename></title>
 
         <para>
-            Creates package metadata used by the build system to generate the
-            final packages.
+            Saves package metadata generated by the
+            <link linkend='ref-tasks-package'><filename>do_package</filename></link>
+            task in
+            <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
+            to make it available globally.
         </para>
     </section>
 

documentation/ref-manual/ref-variables.xml

@@ -1992,22 +1992,36 @@
                     An internal variable specifying the special class override
                     that should currently apply (e.g. "class-target",
                     "class-native", and so forth).
-                    The classes that use this variable set it to
-                    appropriate values.
+                    The classes that use this variable (e.g.
+                    <link linkend='ref-classes-native'><filename>native</filename></link>,
+                    <link linkend='ref-classes-nativesdk'><filename>nativesdk</filename></link>,
+                    and so forth) set the variable to appropriate values.
+                    <note>
+                        <filename>CLASSOVERRIDE</filename> gets its default
+                        "class-target" value from the
+                        <filename>bitbake.conf</filename> file.
+                    </note>
                 </para>
 
                 <para>
-                    You do not normally directly interact with this variable.
-                    The value for the <filename>CLASSOVERRIDE</filename>
-                    variable goes into
-                    <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
-                    and then can be used as an override.
-                    Here is an example where "python-native" is added to
-                    <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
-                    only when building for the <filename>-native</filename> case:
+                    As an example, the following override allows you to install
+                    extra files, but only when building for the target:
                     <literallayout class='monospaced'>
-     DEPENDS_append_class-native = " python-native"
+     do_install_append_class-target() {
+         install my-extra-file ${D}${sysconfdir}
+     }
                     </literallayout>
+                    Here is an example where <filename>FOO</filename>
+                    is set to "native" when building for the build host, and
+                    to "other" when not building for the build host:
+                    <literallayout class='monospaced'>
+     FOO_class-native = "native"
+     FOO = "other"
+                    </literallayout>
+                    The underlying mechanism behind
+                    <filename>CLASSOVERRIDE</filename> is simply that it is
+                    included in the default value of
+                    <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
                 </para>
             </glossdef>
         </glossentry>
@@ -2813,6 +2827,18 @@
                                 those from <filename>foo-dev</filename>, into
                                 the sysroot.
                                 </para></listitem>
+                            <listitem><para>
+                                One recipe having another recipe in
+                                <filename>DEPENDS</filename> does not by itself
+                                add any runtime dependencies between the
+                                packages produced by the two recipes.
+                                However, as explained in the
+                                "<link linkend='automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</link>"
+                                section, runtime dependencies will often be
+                                added automatically, meaning
+                                <filename>DEPENDS</filename> alone is
+                                sufficient for most recipes.
+                                </para></listitem>
                             <listitem><para>
                                 Counterintuitively,
                                 <filename>DEPENDS</filename> is often necessary
@@ -3410,20 +3436,28 @@
 
         <glossentry id='var-DISTROOVERRIDES'><glossterm>DISTROOVERRIDES</glossterm>
             <info>
-                DISTROOVERRIDES[doc] = "Lists overrides specific to the current distribution. By default, the variable list includes the value of the DISTRO variable."
+                DISTROOVERRIDES[doc] = "A colon-separated list of overrides specific to the current distribution."
             </info>
             <glossdef>
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
-                    This variable lists overrides specific to the current
-                    distribution.
-                    By default, the variable list includes the value of the
-                    <filename><link linkend='var-DISTRO'>DISTRO</link></filename>
-                    variable.
-                    You can extend the variable to apply any variable overrides
-                    you want as part of the distribution and are not
-                    already in <filename>OVERRIDES</filename> through
-                    some other means.
+                    A colon-separated list of overrides specific to the
+                    current distribution.
+                    By default, this list includes the value of
+                    <link linkend='var-DISTRO'><filename>DISTRO</filename></link>.
+                </para>
+
+                <para>
+                    You can extend <filename>DISTROOVERRIDES</filename>
+                    to add extra overrides that should apply to
+                    the distribution.
+                </para>
+
+                <para>
+                    The underlying mechanism behind
+                    <filename>DISTROOVERRIDES</filename> is simply that it
+                    is included in the default value of
+                    <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
                 </para>
             </glossdef>
         </glossentry>
@@ -4003,6 +4037,10 @@
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
                     Additional <filename>configure</filename> script options.
+                    See
+                    <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>
+                    for additional information on passing configure script
+                    options.
                 </para>
             </glossdef>
         </glossentry>
@@ -4465,6 +4503,25 @@
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-FORCE_RO_REMOVE'><glossterm>FORCE_RO_REMOVE</glossterm>
+            <info>
+                FORCE_RO_REMOVE[doc] = "Forces the removal of the packages listed in ROOTFS_RO_UNNEEDED during the generation of the root filesystem."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    Forces the removal of the packages listed in
+                    <filename>ROOTFS_RO_UNNEEDED</filename> during the
+                    generation of the root filesystem.
+                </para>
+
+                <para>
+                    Set the variable to "1" to force the removal of these
+                    packages.
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-FULL_OPTIMIZATION'><glossterm>FULL_OPTIMIZATION</glossterm>
             <info>
                 FULL_OPTIMIZATION[doc]= "The options to pass in TARGET_CFLAGS and CFLAGS when compiling an optimized system. This variable defaults to '-fexpensive-optimizations -fomit-frame-pointer -frename-registers -O2'."
@@ -7795,36 +7852,41 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
 
         <glossentry id='var-MACHINEOVERRIDES'><glossterm>MACHINEOVERRIDES</glossterm>
             <info>
-                MACHINEOVERRIDES[doc] = "Lists overrides specific to the current machine. By default, this list includes the value of MACHINE."
+                MACHINEOVERRIDES[doc] = "A colon-separated list of overrides that apply to the current machine."
             </info>
             <glossdef>
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
-                    Lists overrides specific to the current machine.
-                    By default, this list includes the value
-                    of <filename><link linkend='var-MACHINE'>MACHINE</link></filename>.
-                    You can extend the list to apply variable overrides for
-                    classes of machines.
-                    For example, all QEMU emulated machines (e.g. qemuarm,
-                    qemux86, and so forth) include a common file named
-                    <filename>meta/conf/machine/include/qemu.inc</filename>
-                    that prepends <filename>MACHINEOVERRIDES</filename> with
-                    the following variable override:
-                    <literallayout class='monospaced'>
-    MACHINEOVERRIDES =. "qemuall:"
-                    </literallayout>
+                    A colon-separated list of overrides that apply to the
+                    current machine.
+                    By default, this list includes the value of
+                    <link linkend='var-MACHINE'><filename>MACHINE</filename></link>.
                 </para>
 
                 <para>
-                    Applying an override like <filename>qemuall</filename>
-                    affects all QEMU emulated machines elsewhere.
-                    Here is an example from the
-                    <filename>connman-conf</filename> recipe:
+                    You can extend <filename>MACHINEOVERRIDES</filename>
+                    to add extra overrides that should apply to a machine.
+                    For example, all machines emulated in QEMU (e.g.
+                    <filename>qemuarm</filename>, <filename>qemux86</filename>,
+                    and so forth) include a file named
+                    <filename>meta/conf/machine/include/qemu.inc</filename>
+                    that prepends the following override to
+                    <filename>MACHINEOVERRIDES</filename>:
                     <literallayout class='monospaced'>
-    SRC_URI_append_qemuall = "file://wired.config \
-                              file://wired-setup \
-                             "
+     MACHINEOVERRIDES =. "qemuall:"
                     </literallayout>
+                    This override allows variables to be overriden for all
+                    machines emulated in QEMU, like in the following example
+                    from the <filename>connman-conf</filename> recipe:
+                    <literallayout class='monospaced'>
+     SRC_URI_append_qemuall = "file://wired.config \
+                               file://wired-setup \
+                              "
+                    </literallayout>
+                    The underlying mechanism behind
+                    <filename>MACHINEOVERRIDES</filename> is simply that it is
+                    included in the default value of
+                    <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>.
                 </para>
             </glossdef>
         </glossentry>
@@ -7885,6 +7947,39 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     <link linkend='var-BPN'><filename>BPN</filename></link> variable).
                     <filename>MLPREFIX</filename> gets set when a prefix has been
                     added to <filename>PN</filename>.
+                    <note>
+                        The "ML" in <filename>MLPREFIX</filename> stands for
+                        "MultiLib".
+                        This representation is historical and comes from
+                        a time when <filename>nativesdk</filename> was a suffix
+                        rather than a prefix on the recipe name.
+                        When <filename>nativesdk</filename> was turned into a
+                        prefix, it made sense to set
+                        <filename>MLPREFIX</filename> for it as well.
+                    </note>
+                </para>
+
+                <para>
+                    To help understand when <filename>MLPREFIX</filename>
+                    might be needed, consider when
+                    <link linkend='var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
+                    is used to provide a <filename>nativesdk</filename> version
+                    of a recipe in addition to the target version.
+                    If that recipe declares build-time dependencies on tasks in
+                    other recipes by using
+                    <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>,
+                    then a dependency on "foo" will automatically get rewritten
+                    to a dependency on "nativesdk-foo".
+                    However, dependencies like the following will not get
+                    rewritten automatically:
+                    <literallayout class='monospaced'>
+     do_foo[depends] += "<replaceable>recipe</replaceable>:do_foo"
+                    </literallayout>
+                    If you want such a dependency to also get transformed,
+                    you can do the following:
+                    <literallayout class='monospaced'>
+     do_foo[depends] += "${MLPREFIX}<replaceable>recipe</replaceable>:do_foo"
+                    </literallayout>
                 </para>
             </glossdef>
         </glossentry>
@@ -8420,18 +8515,60 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
 
         <glossentry id='var-OVERRIDES'><glossterm>OVERRIDES</glossterm>
             <info>
-                OVERRIDES[doc] = "BitBake uses OVERRIDES to control what variables are overridden after BitBake parses recipes and configuration files."
+                OVERRIDES[doc] = "A colon-separated list of overrides that currently apply."
             </info>
             <glossdef>
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
-                    BitBake uses <filename>OVERRIDES</filename> to control
-                    what variables are overridden after BitBake parses
-                    recipes and configuration files.
-                    You can find more information on how overrides are handled
-                    in the
+                    A colon-separated list of overrides that currently apply.
+                    Overrides are a BitBake mechanism that allows variables to
+                    be selectively overridden at the end of parsing.
+                    The set of overrides in <filename>OVERRIDES</filename>
+                    represents the "state" during building, which includes
+                    the current recipe being built, the machine for which
+                    it is being built, and so forth.
+                </para>
+
+                <para>
+                    As an example, if the string "an-override" appears as an
+                    element in the colon-separated list in
+                    <filename>OVERRIDES</filename>, then the following
+                    assignment will override <filename>FOO</filename> with the
+                    value "overridden" at the end of parsing:
+                    <literallayout class='monospaced'>
+     FOO_an-override = "overridden"
+                    </literallayout>
+                    See the
                     "<ulink url='&YOCTO_DOCS_BB_URL;#conditional-syntax-overrides'>Conditional Syntax (Overrides)</ulink>"
-                    section of the BitBake User Manual.
+                    section in the BitBake User Manual for more information on
+                    the overrides mechanism.
+                </para>
+
+                <para>
+                    The default value of <filename>OVERRIDES</filename>
+                    includes the values of the
+                    <link linkend='var-CLASSOVERRIDE'><filename>CLASSOVERRIDE</filename></link>,
+                    <link linkend='var-MACHINEOVERRIDES'><filename>MACHINEOVERRIDES</filename></link>,
+                    and
+                    <link linkend='var-DISTROOVERRIDES'><filename>DISTROOVERRIDES</filename></link>
+                    variables.
+                    Another important override included by default is
+                    <filename>pn-${PN}</filename>.
+                    This override allows variables to be set for a single
+                    recipe within configuration (<filename>.conf</filename>)
+                    files.
+                    Here is an example:
+                    <literallayout class='monospaced'>
+     FOO_pn-myrecipe = "myrecipe-specific value"
+                    </literallayout>
+                    <note><title>Tip</title>
+                        An easy way to see what overrides apply is to search for
+                        <filename>OVERRIDES</filename> in the output of the
+                        <filename>bitbake -e</filename> command.
+                        See the
+                        "<link linkend='usingpoky-debugging-viewing-variable-values'>Viewing Variable Values</link>"
+                        section for more information.
+                    </note>
                 </para>
             </glossdef>
         </glossentry>
@@ -8979,10 +9116,13 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                         <listitem><para>Extra arguments
                             that should be added to the configure script
                             argument list
-                            (<link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>)
+                            (<link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
+                            or
+                            <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>)
                             if the feature is enabled.</para></listitem>
                         <listitem><para>Extra arguments
                             that should be added to <filename>EXTRA_OECONF</filename>
+                            or <filename>PACKAGECONFIG_CONFARGS</filename>
                             if the feature is disabled.
                             </para></listitem>
                         <listitem><para>Additional build dependencies
@@ -9066,6 +9206,50 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-PACKAGECONFIG_CONFARGS'><glossterm>PACKAGECONFIG_CONFARGS</glossterm>
+            <info>
+                PACKAGECONFIG_CONFARGS[doc] = "A space-separated list of configuration options generated from PACKAGECONFIG."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    A space-separated list of configuration options generated
+                    from the
+                    <link linkend='var-PACKAGECONFIG'><filename>PACKAGECONFIG</filename></link>
+                    setting.
+                    This list of options helps other classes and
+                    recipes take advantage of the
+                    <filename>PACKAGECONFIG</filename> mechanism without
+                    having to include options from
+                    <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>.
+                </para>
+
+                <para>
+                    To illustrate how to use
+                    <filename>PACKAGECONFIG_CONFARGS</filename>, consider the
+                    following example:
+                    <literallayout class='monospaced'>
+     PACKAGECONFIG_CONFARGS = " \
+          -prefix ${prefix} \
+          -sysroot ${STAGING_DIR_NATIVE} \
+          -no-gcc-sysroot
+          "
+                    </literallayout>
+                    In the previous example,
+                    <filename>PACKAGECONFIG_CONFARGS</filename> is set with
+                    three configuration options that can be passed using the
+                    <filename>PACKAGECONFIG</filename> mechanism, thus
+                    avoiding having to use <filename>EXTRA_OECONF</filename>.
+                </para>
+
+                <para>
+                    For additional information, see the
+                    <link linkend='var-PACKAGECONFIG'><filename>PACKAGECONFIG</filename></link>
+                    variable.
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-PACKAGEGROUP_DISABLE_COMPLEMENTARY'><glossterm>PACKAGEGROUP_DISABLE_COMPLEMENTARY</glossterm>
             <info>
                 PACKAGEGROUP_DISABLE_COMPLEMENTARY[doc] = "Prevents automatic creation of the normal complementary packages such as -dev and -dbg in a packagegroup recipe."
@@ -9467,14 +9651,16 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
                     task packages data for each recipe and installs it into
                     this temporary, shared area.
-                    This directory defaults to the following:
+                    This directory defaults to the following, which you should
+                    not change:
                     <literallayout class='monospaced'>
      ${STAGING_DIR_HOST}/pkgdata
                     </literallayout>
-                </para>
-
-                <para>
-                    Do not change this default.
+                    For examples of how this data is used, see the
+                    "<link linkend='automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</link>"
+                    section and the
+                    "<link linkend='viewing-package-information-with-oe-pkgdata-util'>Viewing Package Information with <filename>oe-pkgdata-util</filename></link>"
+                    section.
                 </para>
             </glossdef>
         </glossentry>
@@ -9505,33 +9691,30 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
 
         <glossentry id='var-PKGDESTWORK'><glossterm>PKGDESTWORK</glossterm>
             <info>
-                PKGDESTWORK[doc] = "Points to a temporary work area used by the do_package task to write output from the do_packagedata task."
+                PKGDESTWORK[doc] = "Points to a temporary work area where the do_package task saves package metadata."
             </info>
             <glossdef>
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
-                    Points to a temporary work area used by the
+                    Points to a temporary work area where the
                     <link linkend='ref-tasks-package'><filename>do_package</filename></link>
-                    task to write output from the
-                    <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
-                    task.
+                    task saves package metadata.
                     The <filename>PKGDESTWORK</filename> location defaults to
                     the following:
                     <literallayout class='monospaced'>
      ${WORKDIR}/pkgdata
                     </literallayout>
-                </para>
-
-                <para>
-                    The <filename>do_packagedata</filename> task then packages
-                    the data in the temporary work area and installs it into a
-                    shared directory pointed to by
-                    <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>.
-                </para>
-
-                <para>
                     Do not change this default.
                 </para>
+
+                <para>
+                    The
+                    <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
+                    task copies the package metadata from
+                    <filename>PKGDESTWORK</filename> to
+                    <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
+                    to make it available globally.
+                </para>
             </glossdef>
         </glossentry>
 
@@ -9810,9 +9993,34 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     numbers that could potentially change.
                     Here are two examples:
                     <literallayout class='monospaced'>
-     PREFERRED_VERSION_python = "2.7.3"
+     PREFERRED_VERSION_python = "3.4.0"
      PREFERRED_VERSION_linux-yocto = "3.19%"
                     </literallayout>
+                    <note>
+                        The specified version is matched against
+                        <link linkend='var-PV'><filename>PV</filename></link>,
+                        which does not necessarily match the version part of
+                        the recipe's filename.
+                        For example, consider two recipes
+                        <filename>foo_1.2.bb</filename> and
+                        <filename>foo_git.bb</filename> where
+                        <filename>foo_git.bb</filename> contains the following
+                        assignment:
+                        <literallayout class='monospaced'>
+     PV = "1.1+git${SRCPV}"
+                        </literallayout>
+                        In this case, the correct way to select
+                        <filename>foo_git.bb</filename> is by using an
+                        assignment such as the following:
+                        <literallayout class='monospaced'>
+     PREFERRED_VERSION_foo = "1.1+git%"
+                        </literallayout>
+                        Compare that previous example against the following
+                        incorrect example, which does not work:
+                        <literallayout class='monospaced'>
+     PREFERRED_VERSION_foo = "git"
+                        </literallayout>
+                    </note>
                     Sometimes the <filename>PREFERRED_VERSION</filename>
                     variable can be set by configuration files in a way that
                     is hard to change.
@@ -9830,6 +10038,13 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     <literallayout class='monospaced'>
      PREFERRED_VERSION_linux-yocto_forcevariable = "3.4%"
                     </literallayout>
+                    <note>
+                        The <filename>_forcevariable</filename> override is
+                        not handled specially.
+                        This override only works because the default value of
+                        <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
+                        includes "forcevariable".
+                    </note>
                 </para>
             </glossdef>
         </glossentry>
@@ -9942,6 +10157,12 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                      libplds4.so"
                     </literallayout>
                 </para>
+
+                <para>
+                    For more information, see the
+                    "<link linkend='automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</link>"
+                    section.
+                </para>
             </glossdef>
         </glossentry>
 
@@ -10236,40 +10457,59 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
 
         <glossentry id='var-RDEPENDS'><glossterm>RDEPENDS</glossterm>
             <info>
-                RDEPENDS[doc] = "Lists a package's runtime dependencies (i.e. other packages) that must be installed for the package to be built. They must be the names of other packages as listed in the PACKAGES variable, not recipe names (PN)."
+                RDEPENDS[doc] = "Lists runtime dependencies of a package."
             </info>
             <glossdef>
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
-                    Lists a package's runtime dependencies (i.e. other packages)
-                    that must be installed in order for the built package to run
-                    correctly.
-                    If a package in this list cannot be found during the build,
-                    you will get a build error.
+                    Lists runtime dependencies of a package.
+                    These dependencies are other packages that must be
+                    installed in order for the package to function correctly.
+                    As an example, the following assignment declares that the
+                    package <filename>foo</filename> needs the packages
+                    <filename>bar</filename> and <filename>baz</filename> to
+                    be installed:
+                    <literallayout class='monospaced'>
+     RDEPENDS_foo = "bar baz"
+                    </literallayout>
+                    The most common types of package runtime dependencies are
+                    automatically detected and added.
+                    Therefore, most recipes do not need to set
+                    <filename>RDEPENDS</filename>.
+                    For more information, see the
+                    "<link linkend='automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</link>"
+                    section.
                 </para>
 
                 <para>
-                    When you use the <filename>RDEPENDS</filename> variable
-                    in a recipe, you are essentially stating that the recipe's
+                    The practical effect of the above
+                    <filename>RDEPENDS</filename> assignment is that
+                    <filename>bar</filename> and <filename>baz</filename>
+                    will be declared as dependencies inside the package
+                    <filename>foo</filename> when it is written out by one of
+                    the
+                    <link linkend='ref-tasks-package_write_deb'><filename>do_package_write_*</filename></link>
+                    tasks.
+                    Exactly how this is done depends on which package format
+                    is used, which is determined by
+                    <link linkend='var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></link>.
+                    When the corresponding package manager installs the
+                    package, it will know to also install the packages on
+                    which it depends.
+                </para>
+
+                <para>
+                    To ensure that the packages <filename>bar</filename> and
+                    <filename>baz</filename> get built, the previous
+                    <filename>RDEPENDS</filename> assignment also causes a task
+                    dependency to be added.
+                    This dependency is from the recipe's
                     <link linkend='ref-tasks-build'><filename>do_build</filename></link>
-                    task depends on the existence of a specific package.
-                    Consider this simple example for two recipes named "a" and
-                    "b" that produce similarly named IPK packages.
-                    In this example, the <filename>RDEPENDS</filename>
-                    statement appears in the "a" recipe:
-                    <literallayout class='monospaced'>
-     RDEPENDS_${PN} = "b"
-                    </literallayout>
-                    Here, the dependency is such that the
-                    <filename>do_build</filename> task for recipe "a" depends
-                    on the
-                    <link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link>
-                    task of recipe "b".
-                    This means the package file for "b" must be available when
-                    the output for recipe "a" has been completely built.
-                    More importantly, package "a" will be marked as depending
-                    on package "b" in a manner that is understood by the
-                    package manager.
+                    (not to be confused with
+                    <link linkend='ref-tasks-compile'><filename>do_compile</filename></link>)
+                    task to the <filename>do_package_write_*</filename>
+                    task of the recipes that build <filename>bar</filename> and
+                    <filename>baz</filename>.
                 </para>
 
                 <para>
@@ -10289,7 +10529,8 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                 <para>
                     Because the <filename>RDEPENDS</filename> variable applies
                     to packages being built, you should always use the variable
-                    in a form with an attached package name.
+                    in a form with an attached package name (remember that a
+                    single recipe can build multiple packages).
                     For example, suppose you are building a development package
                     that depends on the <filename>perl</filename> package.
                     In this case, you would use the following
@@ -10317,41 +10558,18 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                 </para>
 
                 <para>
-                    The package name you attach to the
-                    <filename>RDEPENDS</filename> variable must appear
-                    as it would in the <filename>PACKAGES</filename>
-                    namespace before any renaming of the output package by
-                    classes like
-                    <link linkend='ref-classes-debian'><filename>debian</filename></link>.
-                </para>
-
-                <para>
-                    In many cases you do not need to explicitly add
-                    runtime dependencies using
-                    <filename>RDEPENDS</filename> since some automatic
-                    handling occurs:
-                    <itemizedlist>
-                        <listitem><para><emphasis><filename>shlibdeps</filename></emphasis>:
-                            If a runtime package contains a compiled binary,
-                            the build processes the binary in order to
-                            determine any shared libraries
-                            (<filename>.so</filename> files) to which it is
-                            dynamically linked.
-                            The build process adds the packages containing
-                            these libraries to <filename>RDEPENDS</filename>
-                            when creating the runtime package.
-                            In addition, if the shared library is versioned,
-                            the dependency will have the version appended to it
-                            in order to force an upgrade to the appropriate
-                            version if needed.
-                            </para></listitem>
-                        <listitem><para><emphasis><filename>pcdeps</filename></emphasis>:  If
-                            the package ships a <filename>pkg-config</filename>
-                            information file, the build process uses this file
-                            to add items to the <filename>RDEPENDS</filename>
-                            variable to create the runtime packages.
-                            </para></listitem>
-                    </itemizedlist>
+                    The package names you use with
+                    <filename>RDEPENDS</filename> must appear as they would in
+                    the <filename>PACKAGES</filename> variable.
+                    The
+                    <link linkend='var-PKG'><filename>PKG</filename></link>
+                    variable allows a different name to be used for
+                    the final package (e.g. the
+                    <link linkend='ref-classes-debian'><filename>debian</filename></link>
+                    class uses this to rename packages), but this final package
+                    name cannot be used with <filename>RDEPENDS</filename>,
+                    which makes sense as <filename>RDEPENDS</filename> is meant
+                    to be independent of the package format used.
                 </para>
 
                 <para>
@@ -11068,6 +11286,33 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-SDK_INCLUDE_TOOLCHAIN'><glossterm>SDK_INCLUDE_TOOLCHAIN</glossterm>
+            <info>
+                SDK_INCLUDE_TOOLCHAIN[doc] = "When set to "1", specifies to include the toolchain in the extensible SDK."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    When set to "1", specifies to include the toolchain in the
+                    extensible SDK.
+                    Including the toolchain is useful particularly when
+                    <link linkend='var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></link>
+                    is set to "minimal" to keep the SDK reasonably small
+                    but you still want to provide a usable toolchain.
+                    For example, suppose you want to use the toolchain from an
+                    IDE (e.g. Eclipse) or from other tools and you do not
+                    want to perform additional steps to install the toolchain.
+                </para>
+
+                <para>
+                    The <filename>SDK_INCLUDE_TOOLCHAIN</filename> variable
+                    defaults to "0" if <filename>SDK_EXT_TYPE</filename>
+                    is set to "minimal", and defaults to "1" if
+                    <filename>SDK_EXT_TYPE</filename> is set to "full".
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-SDK_INHERIT_BLACKLIST'><glossterm>SDK_INHERIT_BLACKLIST</glossterm>
             <info>
                 SDK_INHERIT_BLACKLIST[doc] = "A list of classes to remove from the INHERIT value globally within the extensible SDK configuration."
@@ -12619,6 +12864,13 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     </literallayout>
                 </para>
 
+                <para>
+                    For information on how BitBake uses stamp files to determine
+                    if a task should be rerun, see the
+                    "<link linkend='stamp-files-and-the-rerunning-of-tasks'>Stamp Files and the Rerunning of Tasks</link>"
+                    section.
+                </para>
+
                 <para>
                     See <link linkend='var-STAMPS_DIR'><filename>STAMPS_DIR</filename></link>,
                     <link linkend='var-MULTIMACH_TARGET_SYS'><filename>MULTIMACH_TARGET_SYS</filename></link>,
@@ -12873,7 +13125,7 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
 
                 <para>
                     For information on Systemd-boot, see the
-                    <ulink url='http://freedesktop.org/wiki/Software/systemd-boot/'>Systemd-boot documentation</ulink>.
+                    <ulink url='http://www.freedesktop.org/wiki/Software/systemd/systemd-boot/'>Systemd-boot documentation</ulink>.
                 </para>
             </glossdef>
         </glossentry>
@@ -12903,7 +13155,7 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
 
                 <para>
                     For information on Systemd-boot, see the
-                    <ulink url='http://freedesktop.org/wiki/Software/systemd-boot/'>Systemd-boot documentation</ulink>.
+                    <ulink url='http://www.freedesktop.org/wiki/Software/systemd/systemd-boot/'>Systemd-boot documentation</ulink>.
                 </para>
             </glossdef>
         </glossentry>
@@ -12931,7 +13183,7 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
 
                 <para>
                     For information on Systemd-boot, see the
-                    <ulink url='http://freedesktop.org/wiki/Software/systemd-boot/'>Systemd-boot documentation</ulink>.
+                    <ulink url='http://www.freedesktop.org/wiki/Software/systemd/systemd-boot/'>Systemd-boot documentation</ulink>.
                 </para>
             </glossdef>
         </glossentry>
@@ -14626,7 +14878,9 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     task.
                     Normally, invalid configure options are simply not passed
                     to the configure script (e.g. should be removed from
-                    <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>).
+                    <link linkend='var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></link>
+                    or
+                    <link linkend='var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></link>).
                     However, common options, for example, exist that are passed
                     to all configure scripts at a class level that might not
                     be valid for some configure scripts.

documentation/ref-manual/ref-varlocality.xml

@@ -176,16 +176,18 @@
             <para>
                 This section lists variables that define extra build information for recipes.
                 <itemizedlist>
+                    <listitem><para><filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE
+                        </link></filename></para></listitem>
                     <listitem><para><filename><link linkend='var-EXTRA_OECMAKE'>EXTRA_OECMAKE</link>
                         </filename></para></listitem>
                     <listitem><para><filename><link linkend='var-EXTRA_OECONF'>EXTRA_OECONF</link>
                         </filename></para></listitem>
                     <listitem><para><filename><link linkend='var-EXTRA_OEMAKE'>EXTRA_OEMAKE</link>
                         </filename></para></listitem>
+                    <listitem><para><filename><link linkend='var-PACKAGECONFIG_CONFARGS'>PACKAGECONFIG_CONFARGS</link>
+                        </filename></para></listitem>
                     <listitem><para><filename><link linkend='var-PACKAGES'>PACKAGES</link></filename>
                         </para></listitem>
-                    <listitem><para><filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE
-                        </link></filename></para></listitem>
                 </itemizedlist>
             </para>
         </section>

documentation/ref-manual/technical-details.xml

@@ -979,6 +979,115 @@
     </section>
 </section>
 
+<section id='automatically-added-runtime-dependencies'>
+    <title>Automatically Added Runtime Dependencies</title>
+
+    <para>
+        The OpenEmbedded build system automatically adds common types of
+        runtime dependencies between packages, which means you do not need to
+        explicitly declare the packages using
+        <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>.
+        Two automatic mechanisms exist (<filename>shlibdeps</filename> and
+        <filename>pcdeps</filename>) that handle shared libraries and
+        package configuration (pkg-config) modules, respectively.
+        For other types of runtime dependencies, you must manually declare
+        the dependencies.
+        <itemizedlist>
+            <listitem><para>
+                <filename>shlibdeps</filename>:
+                During the
+                <link linkend='ref-tasks-package'><filename>do_package</filename></link>
+                task of each recipe, all shared libraries installed by the
+                recipe are located.
+                For each shared library, the package that contains the shared
+                library is registered as providing the shared library.
+                More specifically, the package is registered as providing the
+                <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink>
+                of the library.
+                The resulting shared-library-to-package mapping
+                is saved globally in
+                <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
+                by the
+                <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
+                task.</para>
+
+                <para>Simultaneously, all executables and shared libraries
+                installed by the recipe are inspected to see what shared
+                libraries they link against.
+                For each shared library dependency that is found,
+                <filename>PKGDATA_DIR</filename> is queried to
+                see if some package (likely from a different recipe) contains
+                the shared library.
+                If such a package is found, a runtime dependency is added from
+                the package that depends on the shared library to the package
+                that contains the library.</para>
+
+                <para>The automatically added runtime dependency also includes
+                a version restriction.
+                This version restriction specifies that at least the current
+                version of the package that provides the shared library must be
+                used, as if
+                "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)"
+                had been added to
+                <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>.
+                This forces an upgrade of the package containing the shared
+                library when installing the package that depends on the
+                library, if needed.</para>
+
+                <para>If you want to avoid a package being registered as
+                providing a particular shared library (e.g. because the library
+                is for internal use only), then add the library to
+                <link linkend='var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></link>
+                inside the package's recipe.
+                </para></listitem>
+            <listitem><para>
+                <filename>pcdeps</filename>:
+                During the
+                <link linkend='ref-tasks-package'><filename>do_package</filename></link>
+                task of each recipe, all pkg-config modules
+                (<filename>*.pc</filename> files) installed by the recipe are
+                located.
+                For each module, the package that contains the module is
+                registered as providing the module.
+                The resulting module-to-package mapping is saved globally in
+                <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
+                by the
+                <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
+                task.</para>
+
+                <para>Simultaneously, all pkg-config modules installed by the
+                recipe are inspected to see what other pkg-config modules they
+                depend on.
+                A module is seen as depending on another module if it contains
+                a "Requires:" line that specifies the other module.
+                For each module dependency,
+                <filename>PKGDATA_DIR</filename> is queried to see if some
+                package contains the module.
+                If such a package is found, a runtime dependency is added from
+                the package that depends on the module to the package that
+                contains the module.
+                <note>
+                    The <filename>pcdeps</filename> mechanism most often infers
+                    dependencies between <filename>-dev</filename> packages.
+                </note>
+                </para></listitem>
+        </itemizedlist>
+    </para>
+
+    <para>
+        The <filename>do_package</filename> task depends on the
+        <link linkend='ref-tasks-packagedata'><filename>do_packagedata</filename></link>
+        task of each recipe in
+        <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+        through use of a
+        <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename>
+        declaration, which guarantees that the required
+        shared-library/module-to-package mapping information will be available
+        when needed as long as <filename>DEPENDS</filename> has been
+        correctly set.
+    </para>
+</section>
+
 <section id='fakeroot-and-pseudo'>
     <title>Fakeroot and Pseudo</title>
 

documentation/ref-manual/usingpoky.xml

@@ -248,6 +248,88 @@
         </para>
     </section>
 
+    <section id='viewing-package-information-with-oe-pkgdata-util'>
+        <title>Viewing Package Information with <filename>oe-pkgdata-util</filename></title>
+
+        <para>
+            You can use the <filename>oe-pkgdata-util</filename> command-line
+            utility to query
+            <link linkend='var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></link>
+            and display various package-related information.
+            When you use the utility, you must use it to view information
+            on packages that have already been built.
+        </para>
+
+        <para>
+            Following are a few of the available
+            <filename>oe-pkgdata-util</filename> subcommands.
+            <note>
+                You can use the standard * and ? globbing wildcards as part of
+                package names and paths.
+            </note>
+            <itemizedlist>
+                <listitem><para>
+                    <filename>oe-pkgdata-util list-pkgs [</filename><replaceable>pattern</replaceable><filename>]</filename>:
+                    Lists all packages that have been built, optionally
+                    limiting the match to packages that match
+                    <replaceable>pattern</replaceable>.
+                    </para></listitem>
+                <listitem><para>
+                    <filename>oe-pkgdata-util list-pkg-files&nbsp;</filename><replaceable>package</replaceable><filename>&nbsp;...</filename>:
+                    Lists the files and directories contained in the given
+                    packages.
+                    <note>
+                        <para>
+                        A different way to view the contents of a package is
+                        to look at the
+                        <filename>${</filename><link linkend='var-WORKDIR'><filename>WORKDIR</filename></link><filename>}/packages-split</filename>
+                        directory of the recipe that generates the
+                        package.
+                        This directory is created by the
+                        <link linkend='ref-tasks-package'><filename>do_package</filename></link>
+                        task and has one subdirectory for each package the
+                        recipe generates, which contains the files stored in
+                        that package.</para>
+                        <para>
+                        If you want to inspect the
+                        <filename>${WORKDIR}/packages-split</filename>
+                        directory, make sure that
+                        <link linkend='ref-classes-rm-work'><filename>rm_work</filename></link>
+                        is not enabled when you build the recipe.
+                        </para>
+                        </note>
+                    </para></listitem>
+                <listitem><para>
+                    <filename>oe-pkgdata-util find-path&nbsp;</filename><replaceable>path</replaceable><filename>&nbsp;...</filename>:
+                    Lists the names of the packages that contain the given
+                    paths.
+                    For example, the following tells us that
+                    <filename>/usr/share/man/man1/make.1</filename>
+                    is contained in the <filename>make-doc</filename>
+                    package:
+                    <literallayout class='monospaced'>
+     $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
+     make-doc: /usr/share/man/man1/make.1
+                    </literallayout>
+                    </para></listitem>
+                <listitem><para>
+                    <filename>oe-pkgdata-util lookup-recipe&nbsp;</filename><replaceable>package</replaceable><filename>&nbsp;...</filename>:
+                    Lists the name of the recipes that
+                    produce the given packages.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            For more information on the <filename>oe-pkgdata-util</filename>
+            command, use the help facility:
+            <literallayout class='monospaced'>
+     $ oe-pkgdata-util &dash;&dash;help
+     $ oe-pkgdata-util <replaceable>subcommand</replaceable> --help
+            </literallayout>
+        </para>
+    </section>
+
     <section id='usingpoky-viewing-dependencies-between-recipes-and-tasks'>
         <title>Viewing Dependencies Between Recipes and Tasks</title>
 

documentation/sdk-manual/sdk-appendix-customizing.xml

@@ -381,6 +381,19 @@
             section for more information.
         </note>
     </para>
+
+    <para>
+        You can explicitly control whether or not to include the toolchain
+        when you build an SDK by setting the
+        <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>
+        variable to "1".
+        In particular, it is useful to include the toolchain when you
+        have set <filename>SDK_EXT_TYPE</filename> to
+        "minimal", which by default, excludes the toolchain.
+        Also, it is helpful if you are building a small SDK for use with
+        an IDE, such as Eclipse, or some other tool where you do not want
+        to take extra steps to install a toolchain.
+    </para>
 </section>
 </appendix>
 <!--

documentation/sdk-manual/sdk-appendix-mars.xml

@@ -210,19 +210,6 @@
                             </para></listitem>
                         <listitem><para>Use Git to checkout the correct
                             tag:
-
-                            <note><title>Developer's Note</title>
-                                <para role='writernotes'>
-                                Because the 2.2 tag will not exist until after
-                                the release, I must first do the following
-                                before running the
-                                <filename>git checkout mars/yocto-&DISTRO;</filename>
-                                command in this step:
-                                <literallayout class='monospaced'>
-     $ git tag mars/yocto-2.2 origin/mars-master
-                                </literallayout></para>
-                            </note>
-
                             <literallayout class='monospaced'>
      $ cd ~/eclipse-poky
      $ git checkout mars/yocto-&DISTRO;
@@ -291,15 +278,6 @@
                         <listitem><para>Check the boxes that appear in
                             the installation window to install the
                             following:
-
-                            <note><title>Developer's Note</title>
-                                <para role='writernotes'>
-                                Right now, a check box for BitBake Commander
-                                is appearing.
-                                This probably needs removed.
-                                Do not check this box.</para>
-                            </note>
-
                             <literallayout class='monospaced'>
      Yocto Project SDK Plug-in
      Yocto Project Documentation plug-in
@@ -500,7 +478,7 @@
                             <ulink url='https://wiki.yoctoproject.org/wiki/TipsAndTricks/RunningEclipseAgainstBuiltImage'>wiki</ulink>.
                             In this case, you specify your Build Directory path
                             followed by the image (e.g.
-                            <filename>/home/scottrif/poky/tmp/deploy/images/qemux86/bzImage-qemux86.bin</filename>).
+                            <filename>/home/scottrif/poky/build/tmp/deploy/images/qemux86/bzImage-qemux86.bin</filename>).
                             </para>
                             <para>If you selected the standalone pre-built
                             toolchain, the pre-built image you downloaded is
@@ -812,8 +790,24 @@
                     </para></listitem>
                 <listitem><para>Click "Next".
                     </para></listitem>
-                <listitem><para>Clear out the "host name" field and enter
-                    the IP Address determined earlier (e.g. 192.168.7.2).
+                <listitem><para>Clear out the "Connection name" field and
+                    enter any name you want for the connection.
+                    </para></listitem>
+                <listitem><para>Put the IP address for the connection in
+                    the "Host" field.
+                    For QEMU, the default is <filename>192.168.7.2</filename>.
+                    However, if a previous QEMU session did not exit
+                    cleanly, the IP address increments (e.g.
+                    <filename>192.168.7.3</filename>).
+                    <note>
+                        You can find the IP address for the current QEMU
+                        session by looking in the xterm that opens when
+                        you launch QEMU.
+                    </note>
+                    </para></listitem>
+                <listitem><para>Enter <filename>root</filename>, which
+                    is the default for QEMU, for the "User" field.
+                    Be sure to leave the password field empty.
                     </para></listitem>
                 <listitem><para>Click "Finish" to close the
                     New Connections Dialog.

documentation/sdk-manual/sdk-extensible.xml

@@ -343,42 +343,37 @@
                     However, <filename>devtool</filename> does not provide a
                     specific command that allows you to do this.
                     </para></listitem>
-                <listitem><para><emphasis>Optionally Update the Recipe With Patch Files</emphasis>:
-                    Once you are satisfied with the recipe, if you have made
-                    any changes to the source tree that you want to have
-                    applied by the recipe, you need to generate patches
-                    from those changes.
-                    You do this before moving the recipe
-                    to its final layer and cleaning up the workspace area
-                    <filename>devtool</filename> uses.
-                    This optional step is especially relevant if you are
-                    using or adding third-party software.</para>
-                    <para>To convert commits created using Git to patch files,
-                    use the <filename>devtool update-recipe</filename> command.
+                <listitem><para>
+                    <emphasis>Finish Your Work With the Recipe</emphasis>:
+                    The <filename>devtool finish</filename> command creates
+                    any patches corresponding to commits in the local
+                    Git repository, moves the new recipe to a more permanent
+                    layer, and then resets the recipe so that the recipe is
+                    built normally rather than from the workspace.
+                    <literallayout class='monospaced'>
+     $ devtool finish <replaceable>recipe layer</replaceable>
+                    </literallayout>
                     <note>
                         Any changes you want to turn into patches must be
                         committed to the Git repository in the source tree.
+                    </note></para>
+
+                    <para>As mentioned, the <filename>devtool finish</filename>
+                    command moves the final recipe to its permanent layer.
+                    </para>
+
+                    <para>As a final process of the
+                    <filename>devtool finish</filename> command, the state
+                    of the standard layers and the upstream source is
+                    restored so that you can build the recipe from those
+                    areas rather than the workspace.
+                    <note>
+                        You can use the <filename>devtool reset</filename>
+                        command to put things back should you decide you
+                        do not want to proceed with your work.
+                        If you do use this command, realize that the source
+                        tree is preserved.
                     </note>
-                    <literallayout class='monospaced'>
-     $ devtool update-recipe <replaceable>recipe</replaceable>
-                    </literallayout>
-                    </para></listitem>
-                <listitem><para><emphasis>Move the Recipe to its Permanent Layer</emphasis>:
-                    Before cleaning up the workspace, you need to move the
-                    final recipe to its permanent layer.
-                    You must do this before using the
-                    <filename>devtool reset</filename> command if you want to
-                    retain the recipe.
-                    </para></listitem>
-                <listitem><para><emphasis>Reset the Recipe</emphasis>:
-                    As a final step, you can restore the state such that
-                    standard layers and the upstream source is used to build
-                    the recipe rather than data in the workspace.
-                    To reset the recipe, use the <filename>devtool reset</filename>
-                    command:
-                    <literallayout class='monospaced'>
-     $ devtool reset <replaceable>recipe</replaceable>
-                    </literallayout>
                     </para></listitem>
             </orderedlist>
         </para>
@@ -569,42 +564,43 @@
                     However, <filename>devtool</filename> does not provide a
                     specific command that allows you to do this.
                     </para></listitem>
-                <listitem><para><emphasis>Optionally Create Patch Files for Your Changes</emphasis>:
-                    After you have debugged your changes, you can
-                    use <filename>devtool update-recipe</filename> to
-                    generate patch files for all the commits you have
-                    made.
-                    <note>
-                        Patch files are generated only for changes
-                        you have committed.
-                    </note>
+                <listitem><para>
+                    <emphasis>Finish Your Work With the Recipe</emphasis>:
+                    The <filename>devtool finish</filename> command creates
+                    any patches corresponding to commits in the local
+                    Git repository, updates the recipe to point to them
+                    (or creates a <filename>.bbappend</filename> file to do
+                    so, depending on the specified destination layer), and
+                    then resets the recipe so that the recipe is built normally
+                    rather than from the workspace.
                     <literallayout class='monospaced'>
-     $ devtool update-recipe <replaceable>recipe</replaceable>
+     $ devtool finish <replaceable>recipe layer</replaceable>
                     </literallayout>
-                    By default, the
-                    <filename>devtool update-recipe</filename> command
-                    creates the patch files in a folder named the same
-                    as the recipe beneath the folder in which the recipe
-                    resides, and updates the recipe's
-                    <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
-                    statement to point to the generated patch files.
                     <note>
-                        You can use the
-                        "--append <replaceable>LAYERDIR</replaceable>"
-                        option to cause the command to create append files
-                        in a specific layer rather than the default
-                        recipe layer.
+                        Any changes you want to turn into patches must be
+                        committed to the Git repository in the source tree.
+                    </note></para>
+
+                    <para>Because there is no need to move the recipe,
+                    <filename>devtool finish</filename> either updates the
+                    original recipe in the original layer or the command
+                    creates a <filename>.bbappend</filename> in a different
+                    layer as provided by <replaceable>layer</replaceable>.
+                    </para>
+
+                    <para>As a final process of the
+                    <filename>devtool finish</filename> command, the state
+                    of the standard layers and the upstream source is
+                    restored so that you can build the recipe from those
+                    areas rather than the workspace.
+                    <note>
+                        You can use the <filename>devtool reset</filename>
+                        command to put things back should you decide you
+                        do not want to proceed with your work.
+                        If you do use this command, realize that the source
+                        tree is preserved.
                     </note>
                     </para></listitem>
-                <listitem><para><emphasis>Restore the Workspace</emphasis>:
-                    The <filename>devtool reset</filename> restores the
-                    state so that standard layers and upstream sources are
-                    used to build the recipe rather than what is in the
-                    workspace.
-                    <literallayout class='monospaced'>
-     $ devtool reset <replaceable>recipe</replaceable>
-                    </literallayout>
-                    </para></listitem>
             </orderedlist>
         </para>
     </section>
@@ -641,8 +637,7 @@
                 Binary package (i.e. "-b" option)
                 </para></listitem>
             <listitem><para>
-                Node.js module through
-                <filename>npm</filename>
+                Node.js module
                 </para></listitem>
             <listitem><para>
                 Python modules that use <filename>setuptools</filename>
@@ -781,7 +776,7 @@
             If the <filename>devtool add</filename> command cannot
             determine licensing information, the
             <filename>LICENSE</filename> value is set to "CLOSED" and the
-            <filename>LIC_FILES_CHKSUM</filename> vaule remains unset.
+            <filename>LIC_FILES_CHKSUM</filename> value remains unset.
             This behavior allows you to continue with development but is
             unlikely to be correct in all cases.
             Consequently, you should check the documentation or source
@@ -849,7 +844,10 @@
                     To force the value on the command line, add the
                     variable setting to
                     <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OEMAKE'><filename>EXTRA_OEMAKE</filename></ulink>
-                    within the recipe as follows:
+                    or
+                    <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
+                    within the recipe.
+                    Here is an example using <filename>EXTRA_OEMAKE</filename>:
                     <literallayout class='monospaced'>
      EXTRA_OEMAKE += "'CC=${CC}' 'CXX=${CXX}'"
                     </literallayout>
@@ -918,8 +916,15 @@
         <title>Adding Node.js Modules</title>
 
         <para>
-            You can use the <filename>devtool add</filename> command in the
-            following form to add Node.js modules:
+            You can use the <filename>devtool add</filename> command two
+            different ways to add Node.js modules: 1) Through
+            <filename>npm</filename> and, 2) from a repository or local
+            source.
+        </para>
+
+        <para>
+            Use the following form to add Node.js modules through
+            <filename>npm</filename>:
             <literallayout class='monospaced'>
      $ devtool add "npm://registry.npmjs.org;name=forever;version=0.15.1"
             </literallayout>
@@ -952,6 +957,21 @@
                 </itemizedlist>
             </note>
         </para>
+
+        <para>
+            As mentioned earlier, you can also add Node.js modules
+            directly from a repository or local source tree.
+            To add modules this way, use <filename>devtool add</filename> in
+            the following form:
+            <literallayout class='monospaced'>
+     $ devtool add https://github.com/diversario/node-ssdp
+            </literallayout>
+            In this example, <filename>devtool</filename> fetches the specified
+            Git repository, detects that the code is Node.js code, fetches
+            dependencies using <filename>npm</filename>, and sets
+            <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
+            accordingly.
+        </para>
     </section>
 </section>
 
@@ -1062,9 +1082,12 @@
             then a fixed set of arguments is passed to it to enable
             cross-compilation plus any extras specified by
             <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
+            or
+            <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
             set within the recipe.
             If you wish to pass additional options, add them to
-            <filename>EXTRA_OECONF</filename>.
+            <filename>EXTRA_OECONF</filename> or
+            <filename>PACKAGECONFIG_CONFARGS</filename>.
             Other supported build tools have similar variables
             (e.g.
             <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECMAKE'><filename>EXTRA_OECMAKE</filename></ulink>
@@ -1072,8 +1095,9 @@
             <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OESCONS'><filename>EXTRA_OESCONS</filename></ulink>
             for Scons, and so forth).
             If you need to pass anything on the <filename>make</filename>
-            command line, you can use <filename>EXTRA_OEMAKE</filename> to do
-            so.
+            command line, you can use <filename>EXTRA_OEMAKE</filename> or the
+            <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
+            variables to do so.
         </para>
 
         <para>
@@ -1081,7 +1105,8 @@
             to help you set the arguments listed in the previous paragraph.
             The command determines the exact options being passed, and shows
             them to you along with any custom arguments specified through
-            <filename>EXTRA_OECONF</filename>.
+            <filename>EXTRA_OECONF</filename> or
+            <filename>PACKAGECONFIG_CONFARGS</filename>.
             If applicable, the command also shows you the output of the
             configure script's "&dash;&dash;help" option as a reference.
         </para>

documentation/yocto-project-qs/yocto-project-qs.xml

@@ -255,8 +255,7 @@
                         tar 1.24 or greater
                         </para></listitem>
                     <listitem><para>
-                        Python 2.7.3 or greater excluding Python
-                        3.x, which is not supported.
+                        Python 3.4.0 or greater.
                         </para></listitem>
                 </itemizedlist>
                 If your build host does not meet any of these three listed

meta-poky/conf/distro/include/maintainers.inc

@@ -42,6 +42,7 @@ RECIPE_MAINTAINER_pn-apr-util = "Hongxu Jia <hongxu.jia@windriver.com>"
 RECIPE_MAINTAINER_pn-apt = "Aníbal Limón <anibal.limon@linux.intel.com>"
 RECIPE_MAINTAINER_pn-apt-native = "Aníbal Limón <anibal.limon@linux.intel.com>"
 RECIPE_MAINTAINER_pn-argp-standalone = "Khem Raj <raj.khem@gmail.com>"
+RECIPE_MAINTAINER_pn-asciidoc = "Dengke Du <dengke.du@windriver.com>"
 RECIPE_MAINTAINER_pn-aspell = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-at = "Chen Qi <Qi.Chen@windriver.com>"
 RECIPE_MAINTAINER_pn-at-spi2-atk = "Jussi Kukkonen <jussi.kukkonen@intel.com>"
@@ -503,6 +504,7 @@ RECIPE_MAINTAINER_pn-mesa-demos = "Jussi Kukkonen <jussi.kukkonen@intel.com>"
 RECIPE_MAINTAINER_pn-mesa-gl = "Jussi Kukkonen <jussi.kukkonen@intel.com>"
 RECIPE_MAINTAINER_pn-meta-environment = "Richard Purdie <richard.purdie@linuxfoundation.org>"
 RECIPE_MAINTAINER_pn-meta-environment-extsdk = "Richard Purdie <richard.purdie@linuxfoundation.org>"
+RECIPE_MAINTAINER_pn-meta-extsdk-toolchain = "Richard Purdie <richard.purdie@linuxfoundation.org>"
 RECIPE_MAINTAINER_pn-meta-ide-support = "Cristian Iorga <cristian.iorga@intel.com>"
 RECIPE_MAINTAINER_pn-meta-toolchain = "Cristian Iorga <cristian.iorga@intel.com>"
 RECIPE_MAINTAINER_pn-meta-world-pkgdata = "Richard Purdie <richard.purdie@linuxfoundation.org>"
@@ -641,6 +643,7 @@ RECIPE_MAINTAINER_pn-python-pyrex = "Alejandro Hernandez <alejandro.hernandez@li
 RECIPE_MAINTAINER_pn-python-scons = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python-scons-native = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python-setuptools = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
+RECIPE_MAINTAINER_pn-python-six = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python-smartpm = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python-smmap = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python3 = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
@@ -657,6 +660,7 @@ RECIPE_MAINTAINER_pn-python3-numpy = "Alejandro Hernandez <alejandro.hernandez@l
 RECIPE_MAINTAINER_pn-python3-pip = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python3-pygobject = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python3-setuptools = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
+RECIPE_MAINTAINER_pn-python3-six = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-python3-smmap = "Alejandro Hernandez <alejandro.hernandez@linux.intel.com>"
 RECIPE_MAINTAINER_pn-qemu = "Aníbal Limón <anibal.limon@linux.intel.com>"
 RECIPE_MAINTAINER_pn-qemu-helper-native = "Aníbal Limón <anibal.limon@linux.intel.com>"

meta-poky/conf/distro/poky.conf

@@ -81,6 +81,7 @@ SANITY_TESTED_DISTROS ?= " \
             Ubuntu-16.04 \n \
             Fedora-22 \n \
             Fedora-23 \n \
+            Fedora-24 \n \
             CentOSLinux-7.* \n \
             Debian-8.* \n \
             openSUSE-13.2 \n \

meta-selftest/recipes-test/images/wic-image-minimal.bb

@@ -7,7 +7,7 @@ IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
 IMAGE_FSTYPES = "wic"
 RM_OLD_IMAGE = "1"
 
-DEPENDS = "syslinux syslinux-native parted-native dosfstools-native mtools-native gptfdisk-native"
+DEPENDS = "syslinux syslinux-native dosfstools-native mtools-native gptfdisk-native"
 
 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 

meta-yocto-bsp/conf/machine/beaglebone.conf

@@ -22,7 +22,7 @@ EXTRA_IMAGECMD_jffs2 = "-lnp "
 SERIAL_CONSOLE = "115200 ttyO0"
 
 PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
-PREFERRED_VERSION_linux-yocto ?= "4.4%"
+PREFERRED_VERSION_linux-yocto ?= "4.8%"
 
 KERNEL_IMAGETYPE = "zImage"
 KERNEL_DEVICETREE = "am335x-bone.dtb am335x-boneblack.dtb"

meta-yocto-bsp/conf/machine/edgerouter.conf

@@ -11,7 +11,7 @@ KERNEL_ALT_IMAGETYPE = "vmlinux.bin"
 KERNEL_IMAGE_STRIP_EXTRA_SECTIONS  = ".comment"
 
 PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
-PREFERRED_VERSION_linux-yocto ?= "4.4%"
+PREFERRED_VERSION_linux-yocto ?= "4.8%"
 
 SERIAL_CONSOLE = "115200 ttyS0"
 USE_VT ?= "0"

meta-yocto-bsp/conf/machine/genericx86-64.conf

@@ -6,3 +6,5 @@
 DEFAULTTUNE ?= "core2-64"
 require conf/machine/include/tune-core2.inc
 require conf/machine/include/genericx86-common.inc
+
+PREFERRED_VERSION_linux-yocto_genericx86-64 ?= "4.8%"

meta-yocto-bsp/conf/machine/genericx86.conf

@@ -8,3 +8,5 @@ require conf/machine/include/tune-core2.inc
 require conf/machine/include/genericx86-common.inc
 
 MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "gma500-gfx-check"
+
+PREFERRED_VERSION_linux-yocto_genericx86 ?= "4.8%"

meta-yocto-bsp/conf/machine/include/genericx86-common.inc

@@ -1,4 +1,5 @@
 include conf/machine/include/x86-base.inc
+require conf/machine/include/qemuboot-x86.inc
 MACHINE_FEATURES += "wifi efi pcbios"
 
 PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
@@ -18,3 +19,7 @@ MACHINE_EXTRA_RRECOMMENDS += "linux-firmware eee-acpi-scripts"
 GLIBC_ADDONS = "nptl"
 
 EXTRA_OECONF_append_pn-matchbox-panel-2 = " --with-battery=acpi"
+
+IMAGE_FSTYPES += "wic"
+WKS_FILE = "directdisk.wks"
+do_image_wic[depends] += "mtools-native:do_populate_sysroot dosfstools-native:do_populate_sysroot syslinux-native:do_populate_sysroot syslinux:do_build"

meta-yocto-bsp/conf/machine/mpc8315e-rdb.conf

@@ -14,7 +14,7 @@ SERIAL_CONSOLE = "115200 ttyS0"
 
 MACHINE_FEATURES = "keyboard pci ext2 ext3 serial"
 
-PREFERRED_VERSION_linux-yocto ?= "4.4%"
+PREFERRED_VERSION_linux-yocto ?= "4.8%"
 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
 
 PREFERRED_PROVIDER_virtual/xserver ?= "xserver-xorg"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.1.bbappend

@@ -7,11 +7,11 @@ KBRANCH_mpc8315e-rdb = "standard/fsl-mpc8315e-rdb"
 KMACHINE_genericx86 ?= "common-pc"
 KMACHINE_genericx86-64 ?= "common-pc-64"
 
-SRCREV_machine_genericx86    ?= "0847ed67f89b5a8bbaaa0a7b6cfa2a99ef34834f"
-SRCREV_machine_genericx86-64 ?= "0847ed67f89b5a8bbaaa0a7b6cfa2a99ef34834f"
-SRCREV_machine_edgerouter ?= "403eda4633e9037fb715d0d1e8ae847b2bd0651a"
-SRCREV_machine_beaglebone ?= "859aecf685fcd9d30490a6da459fb76b48947075"
-SRCREV_machine_mpc8315e-rdb ?= "403eda4633e9037fb715d0d1e8ae847b2bd0651a"
+SRCREV_machine_genericx86    ?= "a6b3a8cc120640bf5e528c8558ce060675757fc1"
+SRCREV_machine_genericx86-64 ?= "a6b3a8cc120640bf5e528c8558ce060675757fc1"
+SRCREV_machine_edgerouter ?= "a6b3a8cc120640bf5e528c8558ce060675757fc1"
+SRCREV_machine_beaglebone ?= "2ccca89503ed8c3f37b486b29d2ec3a6699f1899"
+SRCREV_machine_mpc8315e-rdb ?= "a6b3a8cc120640bf5e528c8558ce060675757fc1"
 
 COMPATIBLE_MACHINE_genericx86 = "genericx86"
 COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
@@ -19,5 +19,8 @@ COMPATIBLE_MACHINE_edgerouter = "edgerouter"
 COMPATIBLE_MACHINE_beaglebone = "beaglebone"
 COMPATIBLE_MACHINE_mpc8315e-rdb = "mpc8315e-rdb"
 
-LINUX_VERSION_genericx86 = "4.1.30"
-LINUX_VERSION_genericx86-64 = "4.1.30"
+LINUX_VERSION_genericx86 = "4.1.32"
+LINUX_VERSION_genericx86-64 = "4.1.32"
+LINUX_VERSION_edgerouter = "4.1.32"
+LINUX_VERSION_beaglebone = "4.1.32"
+LINUX_VERSION_mpc8315e-rdb = "4.1.32"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.4.bbappend

@@ -7,11 +7,11 @@ KBRANCH_edgerouter = "standard/edgerouter"
 KBRANCH_beaglebone = "standard/beaglebone"
 KBRANCH_mpc8315e-rdb = "standard/fsl-mpc8315e-rdb"
 
-SRCREV_machine_genericx86    ?= "0a0c93f29c0d65c00abdd2f6d1eb89134fae9525"
-SRCREV_machine_genericx86-64 ?= "0a0c93f29c0d65c00abdd2f6d1eb89134fae9525"
-SRCREV_machine_edgerouter ?= "628bf627561c6285d99fb978e11d4c15fc29324b"
-SRCREV_machine_beaglebone ?= "628bf627561c6285d99fb978e11d4c15fc29324b"
-SRCREV_machine_mpc8315e-rdb ?= "94ac8da44990afd2d43c0ccd713420fb1cfa0792"
+SRCREV_machine_genericx86    ?= "7d1401a0dd9bebfe49937ca7d9785972e0cc76d0"
+SRCREV_machine_genericx86-64 ?= "7d1401a0dd9bebfe49937ca7d9785972e0cc76d0"
+SRCREV_machine_edgerouter ?= "7d1401a0dd9bebfe49937ca7d9785972e0cc76d0"
+SRCREV_machine_beaglebone ?= "7d1401a0dd9bebfe49937ca7d9785972e0cc76d0"
+SRCREV_machine_mpc8315e-rdb ?= "2d7b24e9c0d40d8317a0b9e1b0eefb8280e50d5f"
 
 COMPATIBLE_MACHINE_genericx86 = "genericx86"
 COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
@@ -19,5 +19,8 @@ COMPATIBLE_MACHINE_edgerouter = "edgerouter"
 COMPATIBLE_MACHINE_beaglebone = "beaglebone"
 COMPATIBLE_MACHINE_mpc8315e-rdb = "mpc8315e-rdb"
 
-LINUX_VERSION_genericx86 = "4.4.18"
-LINUX_VERSION_genericx86-64 = "4.4.18"
+LINUX_VERSION_genericx86 = "4.4.20"
+LINUX_VERSION_genericx86-64 = "4.4.20"
+LINUX_VERSION_edgerouter = "4.4.20"
+LINUX_VERSION_beaglebone = "4.4.20"
+LINUX_VERSION_mpc8315e-rdb = "4.4.20"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.8.bbappend

@@ -0,0 +1,26 @@
+KBRANCH_genericx86  = "standard/base"
+KBRANCH_genericx86-64  = "standard/base"
+
+KMACHINE_genericx86 ?= "common-pc"
+KMACHINE_genericx86-64 ?= "common-pc-64"
+KBRANCH_edgerouter = "standard/edgerouter"
+KBRANCH_beaglebone = "standard/beaglebone"
+KBRANCH_mpc8315e-rdb = "standard/fsl-mpc8315e-rdb"
+
+SRCREV_machine_genericx86    ?= "eb9c19c72d40a4bb8a6a3b7b949e41e5c31c0d94"
+SRCREV_machine_genericx86-64 ?= "eb9c19c72d40a4bb8a6a3b7b949e41e5c31c0d94"
+SRCREV_machine_edgerouter ?= "eb9c19c72d40a4bb8a6a3b7b949e41e5c31c0d94"
+SRCREV_machine_beaglebone ?= "eb9c19c72d40a4bb8a6a3b7b949e41e5c31c0d94"
+SRCREV_machine_mpc8315e-rdb ?= "b5c48c7a5d6473373fe7f1e2e8d7b0cf57929908"
+
+COMPATIBLE_MACHINE_genericx86 = "genericx86"
+COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
+COMPATIBLE_MACHINE_edgerouter = "edgerouter"
+COMPATIBLE_MACHINE_beaglebone = "beaglebone"
+COMPATIBLE_MACHINE_mpc8315e-rdb = "mpc8315e-rdb"
+
+LINUX_VERSION_genericx86 = "4.8-rc7"
+LINUX_VERSION_genericx86-64 = "4.8-rc7"
+LINUX_VERSION_edgerouter = "4.8-rc7"
+LINUX_VERSION_beaglebone = "4.8-rc7"
+LINUX_VERSION_mpc8315e-rdb = "4.8-rc7"

meta/classes/autotools.bbclass

@@ -25,7 +25,7 @@ inherit siteinfo
 
 # Space separated list of shell scripts with variables defined to supply test
 # results for autoconf tests we cannot run at build time.
-export CONFIG_SITE = "${@siteinfo_get_files(d, False)}"
+export CONFIG_SITE = "${@siteinfo_get_files(d)}"
 
 acpaths = "default"
 EXTRA_AUTORECONF = "--exclude=autopoint"
@@ -137,6 +137,8 @@ do_configure[postfuncs] += "autotools_postconfigure"
 ACLOCALDIR = "${WORKDIR}/aclocal-copy"
 
 python autotools_copy_aclocals () {
+    import copy
+
     s = d.getVar("AUTOTOOLS_SCRIPT_PATH", True)
     if not os.path.exists(s + "/configure.in") and not os.path.exists(s + "/configure.ac"):
         if not d.getVar("AUTOTOOLS_COPYACLOCAL", False):
@@ -165,36 +167,63 @@ python autotools_copy_aclocals () {
     if start is None:
         bb.fatal("Couldn't find ourself in BB_TASKDEPDATA?")
 
-    # We need to find configure tasks which are either from <target> -> <target>
-    # or <native> -> <native> but not <target> -> <native> unless they're direct
-    # dependencies. This mirrors what would get restored from sstate.
-    done = [start]
-    next = [start]
+    # We need to figure out which m4 files we need to expose to this do_configure task.
+    # This needs to match what would get restored from sstate, which is controlled 
+    # ultimately by calls from bitbake to setscene_depvalid().
+    # That function expects a setscene dependency tree. We build a dependency tree 
+    # condensed to do_populate_sysroot -> do_populate_sysroot dependencies, similar to 
+    # that used by setscene tasks. We can then call into setscene_depvalid() and decide
+    # which dependencies we can "see" and should expose the m4 files for.
+    setscenedeps = copy.deepcopy(taskdepdata)
+
+    start = set([start])
+
+    # Create collapsed do_populate_sysroot -> do_populate_sysroot tree
+    for dep in taskdepdata:
+        data = setscenedeps[dep]        
+        if data[1] != "do_populate_sysroot":
+            for dep2 in setscenedeps:
+                data2 = setscenedeps[dep2]
+                if dep in data2[3]:
+                    data2[3].update(setscenedeps[dep][3])
+                    data2[3].remove(dep)
+            if dep in start:
+                start.update(setscenedeps[dep][3])
+                start.remove(dep)
+            del setscenedeps[dep]
+
+    # Remove circular references
+    for dep in setscenedeps:
+        if dep in setscenedeps[dep][3]:
+            setscenedeps[dep][3].remove(dep)
+
+    # Direct dependencies should be present and can be depended upon
+    for dep in start:
+        configuredeps.append(setscenedeps[dep][0])
+
+    # Call into setscene_depvalid for each sub-dependency and only copy m4 files
+    # for ones that would be restored from sstate.
+    done = list(start)
+    next = list(start)
     while next:
         new = []
         for dep in next:
-            data = taskdepdata[dep]
+            data = setscenedeps[dep]
             for datadep in data[3]:
                 if datadep in done:
                     continue
-                if (not data[0].endswith("-native")) and taskdepdata[datadep][0].endswith("-native") and dep != start:
+                taskdeps = {}
+                taskdeps[dep] = setscenedeps[dep][:2]
+                taskdeps[datadep] = setscenedeps[datadep][:2]
+                retval = setscene_depvalid(datadep, taskdeps, [], d)
+                if retval:
+                    bb.note("Skipping setscene dependency %s for m4 macro copying" % datadep)
                     continue
                 done.append(datadep)
                 new.append(datadep)
-                if taskdepdata[datadep][1] == "do_configure":
-                    configuredeps.append(taskdepdata[datadep][0])
+                configuredeps.append(setscenedeps[datadep][0])
         next = new
 
-    #configuredeps2 = []
-    #for dep in taskdepdata:
-    #    data = taskdepdata[dep]
-    #    if data[1] == "do_configure" and data[0] != pn:
-    #        configuredeps2.append(data[0])
-    #configuredeps.sort()
-    #configuredeps2.sort()
-    #bb.warn(str(configuredeps))
-    #bb.warn(str(configuredeps2))
-
     cp = []
     if nodeps:
         bb.warn("autotools: Unable to find task dependencies, -b being used? Pulling in all m4 files")
@@ -224,8 +253,9 @@ python autotools_copy_aclocals () {
         t = os.path.join(aclocaldir, os.path.basename(c))
         if not os.path.exists(t):
             os.symlink(c, t)
-            
-    d.setVar("CONFIG_SITE", siteinfo_get_files(d, False))
+
+    # Refresh variable with cache files
+    d.setVar("CONFIG_SITE", siteinfo_get_files(d, aclocalcache=True))
 }
 autotools_copy_aclocals[vardepsexclude] += "MACHINE SDK_ARCH BUILD_ARCH SDK_OS BB_TASKDEPDATA"
 

meta/classes/base.bbclass

@@ -309,7 +309,7 @@ base_do_compile() {
 }
 
 addtask install after do_compile
-do_install[dirs] = "${D} ${B}"
+do_install[dirs] = "${B}"
 # Remove and re-create ${D} so that is it guaranteed to be empty
 do_install[cleandirs] = "${D}"
 
@@ -472,7 +472,7 @@ python () {
         else:
             raise bb.parse.SkipPackage("incompatible with machine %s (not in COMPATIBLE_MACHINE)" % d.getVar('MACHINE', True))
 
-    source_mirror_fetch = d.getVar('SOURCE_MIRROR_FETCH', 0)
+    source_mirror_fetch = d.getVar('SOURCE_MIRROR_FETCH', False)
     if not source_mirror_fetch:
         need_host = d.getVar('COMPATIBLE_HOST', True)
         if need_host:

meta/classes/cross.bbclass

@@ -68,4 +68,16 @@ do_install () {
 
 USE_NLS = "no"
 
+export CC = "${BUILD_CC}"
+export CXX = "${BUILD_CXX}"
+export FC = "${BUILD_FC}"
+export CPP = "${BUILD_CPP}"
+export LD = "${BUILD_LD}"
+export CCLD = "${BUILD_CCLD}"
+export AR = "${BUILD_AR}"
+export AS = "${BUILD_AS}"
+export RANLIB = "${BUILD_RANLIB}"
+export STRIP = "${BUILD_STRIP}"
+export NM = "${BUILD_NM}"
+
 inherit nopackages

meta/classes/cve-check.bbclass

@@ -0,0 +1,267 @@
+# This class is used to check recipes against public CVEs.
+#
+# In order to use this class just inherit the class in the
+# local.conf file and it will add the cve_check task for
+# every recipe. The task can be used per recipe, per image,
+# or using the special cases "world" and "universe". The
+# cve_check task will print a warning for every unpatched
+# CVE found and generate a file in the recipe WORKDIR/cve
+# directory. If an image is build it will generate a report
+# in DEPLOY_DIR_IMAGE for all the packages used.
+#
+# Example:
+#   bitbake -c cve_check openssl
+#   bitbake core-image-sato
+#   bitbake -k -c cve_check universe
+#
+# DISCLAIMER
+#
+# This class/tool is meant to be used as support and not
+# the only method to check against CVEs. Running this tool
+# doesn't guarantee your packages are free of CVEs.
+
+CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"
+
+CVE_CHECK_LOCAL_DIR ?= "${WORKDIR}/cve"
+CVE_CHECK_LOCAL_FILE ?= "${CVE_CHECK_LOCAL_DIR}/cve.log"
+CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
+
+CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"
+CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cve"
+CVE_CHECK_COPY_FILES ??= "1"
+CVE_CHECK_CREATE_MANIFEST ??= "1"
+
+# Whitelist for packages (PN)
+CVE_CHECK_PN_WHITELIST = "\
+    glibc-locale \
+"
+
+# Whitelist for CVE and version of package
+CVE_CHECK_CVE_WHITELIST = "{\
+    'CVE-2014-2524': ('6.3',), \
+}"
+
+python do_cve_check () {
+    """
+    Check recipe for patched and unpatched CVEs
+    """
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
+        patched_cves = get_patches_cves(d)
+        patched, unpatched = check_cves(d, patched_cves)
+        if patched or unpatched:
+            cve_data = get_cve_info(d, patched + unpatched)
+            cve_write_data(d, patched, unpatched, cve_data)
+    else:
+        bb.note("Failed to update CVE database, skipping CVE check")
+}
+
+addtask cve_check after do_unpack before do_build
+do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
+do_cve_check[nostamp] = "1"
+
+python cve_check_cleanup () {
+    """
+    Delete the file used to gather all the CVE information.
+    """
+
+    bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE", True))
+}
+
+addhandler cve_check_cleanup
+cve_check_cleanup[eventmask] = "bb.cooker.CookerExit"
+
+python cve_check_write_rootfs_manifest () {
+    """
+    Create CVE manifest when building an image
+    """
+
+    import shutil
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
+        bb.note("Writing rootfs CVE manifest")
+        deploy_dir = d.getVar("DEPLOY_DIR_IMAGE", True)
+        link_name = d.getVar("IMAGE_LINK_NAME", True)
+        manifest_name = d.getVar("CVE_CHECK_MANIFEST", True)
+        cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE", True)
+
+        shutil.copyfile(cve_tmp_file, manifest_name)
+
+        if manifest_name and os.path.exists(manifest_name):
+            manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name)
+            # If we already have another manifest, update symlinks
+            if os.path.exists(os.path.realpath(manifest_link)):
+                if d.getVar('RM_OLD_IMAGE', True) == "1":
+                    os.remove(os.path.realpath(manifest_link))
+                os.remove(manifest_link)
+            os.symlink(os.path.basename(manifest_name), manifest_link)
+            bb.plain("Image CVE report stored in: %s" % manifest_name)
+}
+
+ROOTFS_POSTPROCESS_COMMAND_prepend = "${@'cve_check_write_rootfs_manifest; ' if d.getVar('CVE_CHECK_CREATE_MANIFEST', True) == '1' else ''}"
+
+def get_patches_cves(d):
+    """
+    Get patches that solve CVEs using the "CVE: " tag.
+    """
+
+    import re
+
+    pn = d.getVar("PN", True)
+    cve_match = re.compile("CVE:( CVE\-\d{4}\-\d+)+")
+    patched_cves = set()
+    bb.debug(2, "Looking for patches that solves CVEs for %s" % pn)
+    for url in src_patches(d):
+        patch_file = bb.fetch.decodeurl(url)[2]
+        with open(patch_file, "r", encoding="utf-8") as f:
+            try:
+                patch_text = f.read()
+            except UnicodeDecodeError:
+                bb.debug(1, "Failed to read patch %s using UTF-8 encoding"
+                        " trying with iso8859-1" %  patch_file)
+                f.close()
+                with open(patch_file, "r", encoding="iso8859-1") as f:
+                    patch_text = f.read()
+
+        # Search for the "CVE: " line
+        match = cve_match.search(patch_text)
+        if match:
+            # Get only the CVEs without the "CVE: " tag
+            cves = patch_text[match.start()+5:match.end()]
+            for cve in cves.split():
+                bb.debug(2, "Patch %s solves %s" % (patch_file, cve))
+                patched_cves.add(cve)
+        else:
+            bb.debug(2, "Patch %s doesn't solve CVEs" % patch_file)
+
+    return patched_cves
+
+def check_cves(d, patched_cves):
+    """
+    Run cve-check-tool looking for patched and unpatched CVEs.
+    """
+
+    import ast, csv, tempfile, subprocess, io
+
+    cves_patched = []
+    cves_unpatched = []
+    bpn = d.getVar("BPN", True)
+    pv = d.getVar("PV", True).split("git+")[0]
+    cves = " ".join(patched_cves)
+    cve_db_dir = d.getVar("CVE_CHECK_DB_DIR", True)
+    cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST", True))
+    cve_cmd = "cve-check-tool"
+    cmd = [cve_cmd, "--no-html", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir]
+
+    # If the recipe has been whitlisted we return empty lists
+    if d.getVar("PN", True) in d.getVar("CVE_CHECK_PN_WHITELIST", True).split():
+        bb.note("Recipe has been whitelisted, skipping check")
+        return ([], [])
+
+    # It is needed to export the proxies to download the database using HTTP
+    bb.utils.export_proxies(d)
+
+    try:
+        # Write the faux CSV file to be used with cve-check-tool
+        fd, faux = tempfile.mkstemp(prefix="cve-faux-")
+        with os.fdopen(fd, "w") as f:
+            f.write("%s,%s,%s," % (bpn, pv, cves))
+        cmd.append(faux)
+
+        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode("utf-8")
+        bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output))
+    except subprocess.CalledProcessError as e:
+        bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output))
+    finally:
+        os.remove(faux)
+
+    for row in csv.reader(io.StringIO(output)):
+        # Third row has the unpatched CVEs
+        if row[2]:
+            for cve in row[2].split():
+                # Skip if the CVE has been whitlisted for the current version
+                if pv in cve_whitelist.get(cve,[]):
+                    bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
+                else:
+                    cves_unpatched.append(cve)
+                    bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
+        # Fourth row has patched CVEs
+        if row[3]:
+            for cve in row[3].split():
+                cves_patched.append(cve)
+                bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve))
+
+    return (cves_patched, cves_unpatched)
+
+def get_cve_info(d, cves):
+    """
+    Get CVE information from the database used by cve-check-tool.
+
+    Unfortunately the only way to get CVE info is set the output to
+    html (hard to parse) or query directly the database.
+    """
+
+    try:
+        import sqlite3
+    except ImportError:
+        from pysqlite2 import dbapi2 as sqlite3
+
+    cve_data = {}
+    db_file = d.getVar("CVE_CHECK_DB_FILE", True)
+    placeholder = ",".join("?" * len(cves))
+    query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder
+    conn = sqlite3.connect(db_file)
+    cur = conn.cursor()
+    for row in cur.execute(query, tuple(cves)):
+        cve_data[row[0]] = {}
+        cve_data[row[0]]["summary"] = row[1]
+        cve_data[row[0]]["score"] = row[2]
+        cve_data[row[0]]["modified"] = row[3]
+        cve_data[row[0]]["vector"] = row[4]
+    conn.close()
+
+    return cve_data
+
+def cve_write_data(d, patched, unpatched, cve_data):
+    """
+    Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
+    CVE manifest if enabled.
+    """
+
+    cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True)
+    nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId="
+    write_string = ""
+    first_alert = True
+    bb.utils.mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True))
+
+    for cve in sorted(cve_data):
+        write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True)
+        write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV", True)
+        write_string += "CVE: %s\n" % cve
+        if cve in patched:
+            write_string += "CVE STATUS: Patched\n"
+        else:
+            write_string += "CVE STATUS: Unpatched\n"
+            if first_alert:
+                bb.warn("Found unpatched CVE, for more information check %s" % cve_file)
+                first_alert = False
+        write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
+        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"]
+        write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
+        write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
+
+    with open(cve_file, "w") as f:
+        bb.note("Writing file %s with CVE information" % cve_file)
+        f.write(write_string)
+
+    if d.getVar("CVE_CHECK_COPY_FILES", True) == "1":
+        cve_dir = d.getVar("CVE_CHECK_DIR", True)
+        bb.utils.mkdirhier(cve_dir)
+        deploy_file = os.path.join(cve_dir, d.getVar("PN", True))
+        with open(deploy_file, "w") as f:
+            f.write(write_string)
+
+    if d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1":
+        with open(d.getVar("CVE_CHECK_TMP_FILE", True), "a") as f:
+            f.write("%s" % write_string)

meta/classes/externalsrc.bbclass

@@ -29,6 +29,23 @@ EXTERNALSRC_SYMLINKS ?= "oe-workdir:${WORKDIR} oe-logs:${T}"
 
 python () {
     externalsrc = d.getVar('EXTERNALSRC', True)
+
+    # If this is the base recipe and EXTERNALSRC is set for it or any of its
+    # derivatives, then enable BB_DONT_CACHE to force the recipe to always be
+    # re-parsed so that the file-checksums function for do_compile is run every
+    # time.
+    bpn = d.getVar('BPN', True)
+    if bpn == d.getVar('PN', True):
+        classextend = (d.getVar('BBCLASSEXTEND', True) or '').split()
+        if (externalsrc or
+                ('native' in classextend and
+                 d.getVar('EXTERNALSRC_pn-%s-native' % bpn, True)) or
+                ('nativesdk' in classextend and
+                 d.getVar('EXTERNALSRC_pn-nativesdk-%s' % bpn, True)) or
+                ('cross' in classextend and
+                 d.getVar('EXTERNALSRC_pn-%s-cross' % bpn, True))):
+            d.setVar('BB_DONT_CACHE', '1')
+
     if externalsrc:
         d.setVar('S', externalsrc)
         externalsrcbuild = d.getVar('EXTERNALSRC_BUILD', True)
@@ -85,9 +102,6 @@ python () {
         d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
         d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ")
 
-        # Force the recipe to be always re-parsed so that the file_checksums
-        # function is run every time
-        d.setVar('BB_DONT_CACHE', '1')
         d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}')
 
         # We don't want the workdir to go away

meta/classes/gobject-introspection-data.bbclass

@@ -3,7 +3,5 @@
 #
 # It should be used in recipes to determine whether introspection data should be built,
 # so that qemu use can be avoided when necessary.
-GI_DATA_ENABLED = "${@bb.utils.contains('DISTRO_FEATURES', 'gobject-introspection-data', \
+GI_DATA_ENABLED ?= "${@bb.utils.contains('DISTRO_FEATURES', 'gobject-introspection-data', \
                       bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'True', 'False', d), 'False', d)}"
-
-

meta/classes/gtk-doc.bbclass

@@ -7,7 +7,7 @@
 #
 # It should be used in recipes to determine whether gtk-doc based documentation should be built,
 # so that qemu use can be avoided when necessary.
-GTKDOC_ENABLED = "${@bb.utils.contains('DISTRO_FEATURES', 'api-documentation', \
+GTKDOC_ENABLED ?= "${@bb.utils.contains('DISTRO_FEATURES', 'api-documentation', \
                       bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'True', 'False', d), 'False', d)}"
 
 EXTRA_OECONF_prepend_class-target = "${@bb.utils.contains('GTKDOC_ENABLED', 'True', '--enable-gtk-doc --enable-gtk-doc-html --disable-gtk-doc-pdf', \

meta/classes/image.bbclass

@@ -116,7 +116,7 @@ python () {
 
 def rootfs_variables(d):
     from oe.rootfs import variable_depends
-    variables = ['IMAGE_DEVICE_TABLES','BUILD_IMAGES_FROM_FEEDS','IMAGE_TYPES_MASKED','IMAGE_ROOTFS_ALIGNMENT','IMAGE_OVERHEAD_FACTOR','IMAGE_ROOTFS_SIZE','IMAGE_ROOTFS_EXTRA_SPACE',
+    variables = ['IMAGE_DEVICE_TABLE','IMAGE_DEVICE_TABLES','BUILD_IMAGES_FROM_FEEDS','IMAGE_TYPES_MASKED','IMAGE_ROOTFS_ALIGNMENT','IMAGE_OVERHEAD_FACTOR','IMAGE_ROOTFS_SIZE','IMAGE_ROOTFS_EXTRA_SPACE',
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','RM_OLD_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
@@ -359,7 +359,7 @@ python () {
     # Without de-duplication, gen_conversion_cmds() below
     # would create the same compression command multiple times.
     ctypes = set(d.getVar('CONVERSIONTYPES', True).split())
-    old_overrides = d.getVar('OVERRIDES', 0)
+    old_overrides = d.getVar('OVERRIDES', False)
 
     def _image_base_type(type):
         basetype = type
@@ -415,6 +415,7 @@ python () {
     d.appendVarFlag('do_image', 'vardeps', ' '.join(vardeps))
 
     maskedtypes = (d.getVar('IMAGE_TYPES_MASKED', True) or "").split()
+    maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
 
     for t in basetypes:
         vardeps = set()
@@ -537,6 +538,12 @@ def get_rootfs_size(d):
     base_size += rootfs_alignment - 1
     base_size -= base_size % rootfs_alignment
 
+    # Do not check image size of the debugfs image. This is not supposed
+    # to be deployed, etc. so it doesn't make sense to limit the size
+    # of the debug.
+    if (d.getVar('IMAGE_BUILDING_DEBUGFS', True) or "") == "true":
+        return base_size
+
     # Check the rootfs size against IMAGE_ROOTFS_MAXSIZE (if set)
     if rootfs_maxsize:
         rootfs_maxsize_int = int(rootfs_maxsize)

meta/classes/image_types.bbclass

@@ -218,6 +218,7 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
+do_image_wic[depends] += "parted-native:do_populate_sysroot"
 
 python do_write_wks_template () {
     """Write out expanded template contents to WKS_FULL_PATH."""

meta/classes/insane.bbclass

@@ -690,7 +690,7 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf, messages):
 do_populate_lic[postfuncs] += "populate_lic_qa_checksum"
 python populate_lic_qa_checksum() {
     """
-    Check for changes in the license files 
+    Check for changes in the license files.
     """
     import tempfile
     sane = True
@@ -703,8 +703,7 @@ python populate_lic_qa_checksum() {
         return
 
     if not lic_files and d.getVar('SRC_URI', True):
-        package_qa_handle_error("license-checksum", pn + ": Recipe file fetches files and does not have license file information (LIC_FILES_CHKSUM)", d)
-        return
+        sane = package_qa_handle_error("license-checksum", pn + ": Recipe file fetches files and does not have license file information (LIC_FILES_CHKSUM)", d)
 
     srcdir = d.getVar('S', True)
 
@@ -712,7 +711,7 @@ python populate_lic_qa_checksum() {
         try:
             (type, host, path, user, pswd, parm) = bb.fetch.decodeurl(url)
         except bb.fetch.MalformedUrl:
-            package_qa_handle_error("license-checksum", pn + ": LIC_FILES_CHKSUM contains an invalid URL: " + url, d)
+            sane = package_qa_handle_error("license-checksum", pn + ": LIC_FILES_CHKSUM contains an invalid URL: " + url, d)
             continue
         srclicfile = os.path.join(srcdir, path)
         if not os.path.isfile(srclicfile):
@@ -736,7 +735,7 @@ python populate_lic_qa_checksum() {
             linesout = 0
             for line in fi:
                 lineno += 1
-                if (lineno >= beginline): 
+                if (lineno >= beginline):
                     if ((lineno <= endline) or not endline):
                         fo.write(line)
                         linesout += 1
@@ -768,7 +767,10 @@ python populate_lic_qa_checksum() {
             else:
                 msg = pn + ": LIC_FILES_CHKSUM is not specified for " +  url
                 msg = msg + "\n" + pn + ": The md5 checksum is " + md5chksum
-            package_qa_handle_error("license-checksum", msg, d)
+            sane = package_qa_handle_error("license-checksum", msg, d)
+
+    if not sane:
+        bb.fatal("Fatal QA errors found, failing task.")
 }
 
 def package_qa_check_staged(path,d):

meta/classes/kernel-yocto.bbclass

@@ -1,5 +1,5 @@
 # remove tasks that modify the source tree in case externalsrc is inherited
-SRCTREECOVEREDTASKS += "do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_shared_workdir do_fetch do_unpack do_patch"
+SRCTREECOVEREDTASKS += "do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_fetch do_unpack do_patch"
 
 # returns local (absolute) path names for all valid patches in the
 # src_uri

meta/classes/kernel.bbclass

@@ -127,9 +127,9 @@ PACKAGES_DYNAMIC += "^kernel-firmware-.*"
 export OS = "${TARGET_OS}"
 export CROSS_COMPILE = "${TARGET_PREFIX}"
 
-KERNEL_PRIORITY ?= "${@int(d.getVar('PV',1).split('-')[0].split('+')[0].split('.')[0]) * 10000 + \
-                       int(d.getVar('PV',1).split('-')[0].split('+')[0].split('.')[1]) * 100 + \
-                       int(d.getVar('PV',1).split('-')[0].split('+')[0].split('.')[-1])}"
+KERNEL_PRIORITY ?= "${@int(d.getVar('PV', True).split('-')[0].split('+')[0].split('.')[0]) * 10000 + \
+                       int(d.getVar('PV', True).split('-')[0].split('+')[0].split('.')[1]) * 100 + \
+                       int(d.getVar('PV', True).split('-')[0].split('+')[0].split('.')[-1])}"
 
 KERNEL_RELEASE ?= "${KERNEL_VERSION}"
 
@@ -140,7 +140,7 @@ KERNEL_IMAGEDEST = "boot"
 #
 # configuration
 #
-export CMDLINE_CONSOLE = "console=${@d.getVar("KERNEL_CONSOLE",1) or "ttyS0"}"
+export CMDLINE_CONSOLE = "console=${@d.getVar("KERNEL_CONSOLE", True) or "ttyS0"}"
 
 KERNEL_VERSION = "${@get_kernelversion_headers('${B}')}"
 
@@ -156,6 +156,7 @@ UBOOT_LOADADDRESS ?= "${UBOOT_ENTRYPOINT}"
 # Some Linux kernel configurations need additional parameters on the command line
 KERNEL_EXTRA_ARGS ?= ""
 
+EXTRA_OEMAKE = " HOSTCC="${BUILD_CC}" HOSTCPP="${BUILD_CPP}""
 KERNEL_ALT_IMAGETYPE ??= ""
 
 copy_initramfs() {
@@ -199,7 +200,7 @@ copy_initramfs() {
 	echo "Finished copy of initramfs into ./usr"
 }
 
-INITRAMFS_BASE_NAME = "initramfs-${PV}-${PR}-${MACHINE}-${DATETIME}"
+INITRAMFS_BASE_NAME ?= "initramfs-${PV}-${PR}-${MACHINE}-${DATETIME}"
 INITRAMFS_BASE_NAME[vardepsexclude] = "DATETIME"
 do_bundle_initramfs () {
 	if [ ! -z "${INITRAMFS_IMAGE}" -a x"${INITRAMFS_IMAGE_BUNDLE}" = x1 ]; then
@@ -326,6 +327,36 @@ kernel_do_install() {
 }
 do_install[prefuncs] += "package_get_auto_pr"
 
+# Must be ran no earlier than after do_kernel_checkout or else Makefile won't be in ${S}/Makefile
+do_kernel_version_sanity_check() {
+	# The Makefile determines the kernel version shown at runtime
+	# Don't use KERNEL_VERSION because the headers it grabs the version from aren't generated until do_compile
+	VERSION=$(grep "^VERSION =" ${S}/Makefile | sed s/.*=\ *//)
+	PATCHLEVEL=$(grep "^PATCHLEVEL =" ${S}/Makefile | sed s/.*=\ *//)
+	SUBLEVEL=$(grep "^SUBLEVEL =" ${S}/Makefile | sed s/.*=\ *//)
+	EXTRAVERSION=$(grep "^EXTRAVERSION =" ${S}/Makefile | sed s/.*=\ *//)
+
+	# Build a string for regex and a plain version string
+	reg="^${VERSION}\.${PATCHLEVEL}"
+	vers="${VERSION}.${PATCHLEVEL}"
+	if [ -n "${SUBLEVEL}" ]; then
+		# Ignoring a SUBLEVEL of zero is fine
+		if [ "${SUBLEVEL}" = "0" ]; then
+			reg="${reg}(\.${SUBLEVEL})?"
+		else
+			reg="${reg}\.${SUBLEVEL}"
+			vers="${vers}.${SUBLEVEL}"
+		fi
+	fi
+	vers="${vers}${EXTRAVERSION}"
+	reg="${reg}${EXTRAVERSION}"
+
+	if [ -z `echo ${PV} | grep -E "${reg}"` ]; then
+		bbfatal "Package Version (${PV}) does not match of kernel being built (${vers}). Please update the PV variable to match the kernel source."
+	fi
+	exit 0
+}
+
 addtask shared_workdir after do_compile before do_compile_kernelmodules
 addtask shared_workdir_setscene
 

meta/classes/package.bbclass

@@ -1490,7 +1490,7 @@ python package_do_shlibs() {
     import re, pipes
     import subprocess as sub
 
-    exclude_shlibs = d.getVar('EXCLUDE_FROM_SHLIBS', 0)
+    exclude_shlibs = d.getVar('EXCLUDE_FROM_SHLIBS', False)
     if exclude_shlibs:
         bb.note("not generating shlibs")
         return

meta/classes/package_ipk.bbclass

@@ -245,7 +245,7 @@ python do_package_ipk () {
 
         os.chdir(basedir)
         ret = subprocess.call("PATH=\"%s\" %s %s %s" % (localdata.getVar("PATH", True),
-                                                          d.getVar("OPKGBUILDCMD",1), pkg, pkgoutdir), shell=True)
+                                                          d.getVar("OPKGBUILDCMD", True), pkg, pkgoutdir), shell=True)
         if ret != 0:
             bb.utils.unlockfile(lf)
             raise bb.build.FuncFailed("opkg-build execution failed")

meta/classes/populate_sdk_base.bbclass

@@ -26,7 +26,7 @@ SDK_DIR = "${WORKDIR}/sdk"
 SDK_OUTPUT = "${SDK_DIR}/image"
 SDK_DEPLOY = "${DEPLOY_DIR}/sdk"
 
-SDKDEPLOYDIR = "${WORKDIR}/deploy-${PN}-populate-sdk"
+SDKDEPLOYDIR = "${WORKDIR}/${SDKMACHINE}-deploy-${PN}-populate-sdk"
 
 B_task-populate-sdk = "${SDK_DIR}"
 
@@ -125,7 +125,7 @@ SSTATE_SKIP_CREATION_task-populate-sdk = '1'
 do_populate_sdk[cleandirs] = "${SDKDEPLOYDIR}"
 do_populate_sdk[sstate-inputdirs] = "${SDKDEPLOYDIR}"
 do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
-do_populate_sdk[stamp-extra-info] = "${MACHINE}"
+do_populate_sdk[stamp-extra-info] = "${MACHINE}${SDKMACHINE}"
 
 fakeroot create_sdk_files() {
 	cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/

meta/classes/populate_sdk_ext.bbclass

@@ -43,7 +43,7 @@ SDK_TARGETS ?= "${PN}"
 
 def get_sdk_install_targets(d, images_only=False):
     sdk_install_targets = ''
-    if d.getVar('SDK_EXT_TYPE', True) != 'minimal':
+    if images_only or d.getVar('SDK_EXT_TYPE', True) != 'minimal':
         sdk_install_targets = d.getVar('SDK_TARGETS', True)
 
         depd = d.getVar('BB_TASKDEPDATA', False)
@@ -88,8 +88,7 @@ SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar(
 def clean_esdk_builddir(d, sdkbasepath):
     """Clean up traces of the fake build for create_filtered_tasklist()"""
     import shutil
-    cleanpaths = 'cache conf/sanity_info conf/templateconf.cfg'.split()
-    cleanpaths.append(os.path.basename(d.getVar('TMPDIR', True)))
+    cleanpaths = 'cache conf/sanity_info conf/templateconf.cfg tmp'.split()
     for pth in cleanpaths:
         fullpth = os.path.join(sdkbasepath, pth)
         if os.path.isdir(fullpth):
@@ -109,10 +108,12 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
     # Create a temporary build directory that we can pass to the env setup script
     shutil.copyfile(sdkbasepath + '/conf/local.conf', sdkbasepath + '/conf/local.conf.bak')
     try:
-        # Force the use of sstate from the build system
         with open(sdkbasepath + '/conf/local.conf', 'a') as f:
+            # Force the use of sstate from the build system
             f.write('\nSSTATE_DIR_forcevariable = "%s"\n' % d.getVar('SSTATE_DIR', True))
             f.write('SSTATE_MIRRORS_forcevariable = ""\n')
+            # Ensure TMPDIR is the default so that clean_esdk_builddir() can delete it
+            f.write('TMPDIR_forcevariable = "${TOPDIR}/tmp"\n')
             # Drop uninative if the build isn't using it (or else NATIVELSBSTRING will
             # be different and we won't be able to find our native sstate)
             if not bb.data.inherits_class('uninative', d):

meta/classes/qemuboot.bbclass

@@ -48,8 +48,9 @@ python write_qemuboot_conf() {
     import configparser
 
     build_vars = ['MACHINE', 'TUNE_ARCH', 'DEPLOY_DIR_IMAGE', \
-                'IMAGE_NAME', 'IMAGE_LINK_NAME', 'STAGING_DIR_NATIVE', \
-                'STAGING_BINDIR_NATIVE', 'STAGING_DIR_HOST']
+                'KERNEL_IMAGETYPE', 'IMAGE_NAME', 'IMAGE_LINK_NAME', \
+                'STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE', \
+                'STAGING_DIR_HOST']
 
     # Vars from bsp
     qb_vars = []
@@ -63,6 +64,15 @@ python write_qemuboot_conf() {
     cf.add_section('config_bsp')
     for k in build_vars + qb_vars:
         cf.set('config_bsp', k, '%s' % d.getVar(k, True))
+
+    # QB_DEFAULT_KERNEL's value of KERNEL_IMAGETYPE is the name of a symlink
+    # to the kernel file, which hinders relocatability of the qb conf.
+    # Read the link and replace it with the full filename of the target.
+    kernel_link = os.path.join(d.getVar('DEPLOY_DIR_IMAGE', True), d.getVar('QB_DEFAULT_KERNEL', True))
+    kernel = os.readlink(kernel_link)
+    cf.set('config_bsp', 'QB_DEFAULT_KERNEL', kernel)
+
+    bb.utils.mkdirhier(os.path.dirname(qemuboot))
     with open(qemuboot, 'w') as f:
         cf.write(f)
 

meta/classes/recipe_sanity.bbclass

@@ -8,7 +8,7 @@ def bad_runtime_vars(cfgdata, d):
         return
 
     for var in d.getVar("__recipe_sanity_badruntimevars", True).split():
-        val = d.getVar(var, 0)
+        val = d.getVar(var, False)
         if val and val != cfgdata.get(var):
             __note("%s should be %s_${PN}" % (var, var), d)
 
@@ -16,11 +16,11 @@ __recipe_sanity_reqvars = "DESCRIPTION"
 __recipe_sanity_reqdiffvars = ""
 def req_vars(cfgdata, d):
     for var in d.getVar("__recipe_sanity_reqvars", True).split():
-        if not d.getVar(var, 0):
+        if not d.getVar(var, False):
             __note("%s should be set" % var, d)
 
     for var in d.getVar("__recipe_sanity_reqdiffvars", True).split():
-        val = d.getVar(var, 0)
+        val = d.getVar(var, False)
         cfgval = cfgdata.get(var)
 
         if not val:
@@ -29,7 +29,7 @@ def req_vars(cfgdata, d):
             __note("%s should be defined to something other than default (%s)" % (var, cfgval), d)
 
 def var_renames_overwrite(cfgdata, d):
-    renames = d.getVar("__recipe_sanity_renames", 0)
+    renames = d.getVar("__recipe_sanity_renames", False)
     if renames:
         for (key, newkey, oldvalue, newvalue) in renames:
             if oldvalue != newvalue and oldvalue != cfgdata.get(newkey):
@@ -50,7 +50,7 @@ def can_use_autotools_base(cfgdata, d):
         if cfg.find(i) != -1:
             return False
 
-    for clsfile in d.getVar("__inherit_cache", 0):
+    for clsfile in d.getVar("__inherit_cache", False):
         (base, _) = os.path.splitext(os.path.basename(clsfile))
         if cfg.find("%s_do_configure" % base) != -1:
             __note("autotools_base usage needs verification, spotted %s_do_configure" % base, d)
@@ -60,7 +60,7 @@ def can_use_autotools_base(cfgdata, d):
 def can_delete_FILESPATH(cfgdata, d):
     expected = cfgdata.get("FILESPATH")
     expectedpaths = d.expand(expected)
-    unexpanded = d.getVar("FILESPATH", 0)
+    unexpanded = d.getVar("FILESPATH", False)
     filespath = d.getVar("FILESPATH", True).split(":")
     filespath = [os.path.normpath(f) for f in filespath if os.path.exists(f)]
     for fp in filespath:
@@ -73,7 +73,7 @@ def can_delete_FILESPATH(cfgdata, d):
 def can_delete_FILESDIR(cfgdata, d):
     expected = cfgdata.get("FILESDIR")
     #expected = "${@bb.utils.which(d.getVar('FILESPATH', True), '.')}"
-    unexpanded = d.getVar("FILESDIR", 0)
+    unexpanded = d.getVar("FILESDIR", False)
     if unexpanded is None:
         return False
 
@@ -90,7 +90,7 @@ def can_delete_others(p, cfgdata, d):
     for k in ["S", "PV", "PN", "DESCRIPTION", "DEPENDS",
               "SECTION", "PACKAGES", "EXTRA_OECONF", "EXTRA_OEMAKE"]:
     #for k in cfgdata:
-        unexpanded = d.getVar(k, 0)
+        unexpanded = d.getVar(k, False)
         cfgunexpanded = cfgdata.get(k)
         if not cfgunexpanded:
             continue
@@ -117,7 +117,7 @@ python do_recipe_sanity () {
         #(can_use_autotools_base, "candidate for use of autotools_base"),
         (incorrect_nonempty_PACKAGES, "native or cross recipe with non-empty PACKAGES"),
     ]
-    cfgdata = d.getVar("__recipe_sanity_cfgdata", 0)
+    cfgdata = d.getVar("__recipe_sanity_cfgdata", False)
 
     for (func, msg) in sanitychecks:
         if func(cfgdata, d):
@@ -143,8 +143,8 @@ python recipe_sanity_eh () {
 
     cfgdata = {}
     for k in d.keys():
-        if not isinstance(d.getVar(k, 0), bb.data_smart.DataSmart):
-            cfgdata[k] = d.getVar(k, 0)
+        if not isinstance(d.getVar(k, False), bb.data_smart.DataSmart):
+            cfgdata[k] = d.getVar(k, False)
 
     d.setVar("__recipe_sanity_cfgdata", cfgdata)
     #d.setVar("__recipe_sanity_cfgdata", d)

meta/classes/sanity.bbclass

@@ -135,8 +135,7 @@ is a good way to visualise the changes."""
         bb.note("Your conf/bblayers.conf has been automatically updated.")
         return
 
-        if not status.reparse:
-            status.addresult()
+        status.addresult()
 
     elif current_lconf == 6 and lconf_version > 6:
         # Handle rename of meta-yocto -> meta-poky
@@ -459,19 +458,19 @@ def check_gcc_march(sanity_data):
 
         # Check if GCC could work without march
         if not result:
-            status,res = oe.utils.getstatusoutput(sanity_data.expand("${BUILD_PREFIX}gcc gcc_test.c -o gcc_test"))
+            status,res = oe.utils.getstatusoutput(sanity_data.expand("${BUILD_CC} gcc_test.c -o gcc_test"))
             if status == 0:
                 result = True;
 
         if not result:
-            status,res = oe.utils.getstatusoutput(sanity_data.expand("${BUILD_PREFIX}gcc -march=native gcc_test.c -o gcc_test"))
+            status,res = oe.utils.getstatusoutput(sanity_data.expand("${BUILD_CC} -march=native gcc_test.c -o gcc_test"))
             if status == 0:
                 message = "BUILD_CFLAGS_append = \" -march=native\""
                 result = True;
 
         if not result:
             build_arch = sanity_data.getVar('BUILD_ARCH', True)
-            status,res = oe.utils.getstatusoutput(sanity_data.expand("${BUILD_PREFIX}gcc -march=%s gcc_test.c -o gcc_test" % build_arch))
+            status,res = oe.utils.getstatusoutput(sanity_data.expand("${BUILD_CC} -march=%s gcc_test.c -o gcc_test" % build_arch))
             if status == 0:
                 message = "BUILD_CFLAGS_append = \" -march=%s\"" % build_arch
                 result = True;
@@ -557,20 +556,17 @@ def check_perl_modules(sanity_data):
         return "Required perl module(s) not found: %s\n\n%s\n" % (ret, errresult)
     return None
 
-def sanity_check_conffiles(status, d):
+def sanity_check_conffiles(d):
     funcs = d.getVar('BBLAYERS_CONF_UPDATE_FUNCS', True).split()
     for func in funcs:
         conffile, current_version, required_version, func = func.split(":")
         if check_conf_exists(conffile, d) and d.getVar(current_version, True) is not None and \
                 d.getVar(current_version, True) != d.getVar(required_version, True):
-            success = True
             try:
                 bb.build.exec_func(func, d, pythonexception=True)
             except NotImplementedError as e:
-                success = False
-                status.addresult(str(e))
-            if success:
-                status.reparse = True
+                bb.fatal(e)
+            d.setVar("BB_INVALIDCONF", True)
 
 def sanity_handle_abichanges(status, d):
     #
@@ -673,11 +669,11 @@ def check_sanity_version_change(status, d):
     if not check_app_exists("${MAKE}", d):
         missing = missing + "GNU make,"
 
-    if not check_app_exists('${BUILD_PREFIX}gcc', d):
-        missing = missing + "C Compiler (%sgcc)," % d.getVar("BUILD_PREFIX", True)
+    if not check_app_exists('${BUILD_CC}', d):
+        missing = missing + "C Compiler (%s)," % d.getVar("BUILD_CC", True)
 
-    if not check_app_exists('${BUILD_PREFIX}g++', d):
-        missing = missing + "C++ Compiler (%sg++)," % d.getVar("BUILD_PREFIX", True)
+    if not check_app_exists('${BUILD_CXX}', d):
+        missing = missing + "C++ Compiler (%s)," % d.getVar("BUILD_CXX", True)
 
     required_utilities = d.getVar('SANITY_REQUIRED_UTILITIES', True)
 
@@ -746,7 +742,7 @@ def check_sanity_version_change(status, d):
             status.addresult("You have a 32-bit libc, but no 32-bit headers.  You must install the 32-bit libc headers.\n")
 
     bbpaths = d.getVar('BBPATH', True).split(":")
-    if ("." in bbpaths or "./" in bbpaths or "" in bbpaths) and not status.reparse:
+    if ("." in bbpaths or "./" in bbpaths or "" in bbpaths):
         status.addresult("BBPATH references the current directory, either through "    \
                 "an empty entry, a './' or a '.'.\n\t This is unsafe and means your "\
                 "layer configuration is adding empty elements to BBPATH.\n\t "\
@@ -796,8 +792,6 @@ def check_sanity_everybuild(status, d):
 
     sanity_check_locale(d)
 
-    sanity_check_conffiles(status, d)
-
     paths = d.getVar('PATH', True).split(":")
     if "." in paths or "./" in paths or "" in paths:
         status.addresult("PATH contains '.', './' or '' (empty element), which will break the build, please remove this.\nParsed PATH is " + str(paths) + "\n")
@@ -943,7 +937,6 @@ def check_sanity(sanity_data):
         def __init__(self):
             self.messages = ""
             self.network_error = False
-            self.reparse = False
 
         def addresult(self, message):
             if message:
@@ -999,7 +992,6 @@ def check_sanity(sanity_data):
 
     if status.messages != "":
         raise_sanity_error(sanity_data.expand(status.messages), sanity_data, status.network_error)
-    return status.reparse
 
 # Create a copy of the datastore and finalise it to ensure appends and 
 # overrides are set - the datastore has yet to be finalised at ConfigParsed
@@ -1008,15 +1000,20 @@ def copy_data(e):
     sanity_data.finalize()
     return sanity_data
 
+addhandler config_reparse_eventhandler
+config_reparse_eventhandler[eventmask] = "bb.event.ConfigParsed"
+python config_reparse_eventhandler() {
+    sanity_check_conffiles(e.data)
+}
+
 addhandler check_sanity_eventhandler
 check_sanity_eventhandler[eventmask] = "bb.event.SanityCheck bb.event.NetworkTest"
 python check_sanity_eventhandler() {
     if bb.event.getName(e) == "SanityCheck":
         sanity_data = copy_data(e)
+        check_sanity(sanity_data)
         if e.generateevents:
             sanity_data.setVar("SANITY_USE_EVENTS", "1")
-        reparse = check_sanity(sanity_data)
-        e.data.setVar("BB_INVALIDCONF", reparse)
         bb.event.fire(bb.event.SanityCheckPassed(), e.data)
     elif bb.event.getName(e) == "NetworkTest":
         sanity_data = copy_data(e)

meta/classes/siteinfo.bbclass

@@ -153,7 +153,7 @@ python () {
         bb.fatal("Please add your architecture to siteinfo.bbclass")
 }
 
-def siteinfo_get_files(d, no_cache = False):
+def siteinfo_get_files(d, aclocalcache = False):
     sitedata = siteinfo_data(d)
     sitefiles = ""
     for path in d.getVar("BBPATH", True).split(":"):
@@ -162,11 +162,17 @@ def siteinfo_get_files(d, no_cache = False):
             if os.path.exists(filename):
                 sitefiles += filename + " "
 
-    if no_cache: return sitefiles
+    if not aclocalcache:
+        return sitefiles
 
-    # Now check for siteconfig cache files
-    # Use the files copied to the aclocal cache generated by autotools.bbclass
-    # to avoid races
+    # Now check for siteconfig cache files in the directory setup by autotools.bbclass to
+    # avoid races.
+    #
+    # ACLOCALDIR may or may not exist so cache should only be set to True from autotools.bbclass
+    # after files have been copied into this location. To do otherwise risks parsing/signature
+    # issues and the directory being created/removed whilst this code executes. This can happen
+    # when a multilib recipe is parsed along with its base variant which may be running at the time
+    # causing rare but nasty failures
     path_siteconfig = d.getVar('ACLOCALDIR', True)
     if path_siteconfig and os.path.isdir(path_siteconfig):
         for i in os.listdir(path_siteconfig):
@@ -174,7 +180,6 @@ def siteinfo_get_files(d, no_cache = False):
                 continue
             filename = os.path.join(path_siteconfig, i)
             sitefiles += filename + " "
-
     return sitefiles
 
 #

meta/classes/sstate.bbclass

@@ -175,6 +175,8 @@ def sstate_install(ss, d):
     if os.access(manifest, os.R_OK):
         bb.fatal("Package already staged (%s)?!" % manifest)
 
+    d.setVar("SSTATE_INST_POSTRM", manifest + ".postrm")
+
     locks = []
     for lock in ss['lockfiles-shared']:
         locks.append(bb.utils.lockfile(lock, True))
@@ -205,6 +207,7 @@ def sstate_install(ss, d):
             f = os.path.normpath(f)
             realmatch = True
             for w in whitelist:
+                w = os.path.normpath(w)
                 if f.startswith(w):
                     realmatch = False
                     break
@@ -407,6 +410,13 @@ def sstate_clean_manifest(manifest, d):
         except OSError:
             pass
 
+    postrm = manifest + ".postrm"
+    if os.path.exists(manifest + ".postrm"):
+        import subprocess
+        os.chmod(postrm, 0o755)
+        subprocess.call(postrm, shell=True)
+        oe.path.remove(postrm)
+
     oe.path.remove(manifest)
 
 def sstate_clean(ss, d):
@@ -1011,6 +1021,8 @@ python sstate_eventhandler2() {
         for r in toremove:
             (stamp, manifest, workdir) = r.split()
             for m in glob.glob(manifest + ".*"):
+                if m.endswith(".postrm"):
+                    continue
                 sstate_clean_manifest(m, d)
             bb.utils.remove(stamp + "*")
             if removeworkdir:

meta/classes/testimage.bbclass

@@ -67,6 +67,11 @@ TESTIMAGEDEPENDS = ""
 TESTIMAGEDEPENDS_qemuall = "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot"
 TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'cpio-native:do_populate_sysroot', '', d)}"
 TESTIMAGEDEPENDS_qemuall += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'cpio-native:do_populate_sysroot', '', d)}"
+TESTIMAGEDEPENDS_qemuall += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'createrepo-native:do_populate_sysroot', '', d)}"
+TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python-smartpm-native:do_populate_sysroot', '', d)}"
+TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'ipk', 'opkg-utils-native:do_populate_sysroot', '', d)}"
+TESTIMAGEDEPENDS += "${@bb.utils.contains('IMAGE_PKGTYPE', 'deb', 'apt-native:do_populate_sysroot', '', d)}"
+
 
 TESTIMAGELOCK = "${TMPDIR}/testimage.lock"
 TESTIMAGELOCK_qemuall = ""

meta/classes/toolchain-scripts-base.bbclass

@@ -0,0 +1,11 @@
+#This function create a version information file
+toolchain_create_sdk_version () {
+	local versionfile=$1
+	rm -f $versionfile
+	touch $versionfile
+	echo 'Distro: ${DISTRO}' >> $versionfile
+	echo 'Distro Version: ${DISTRO_VERSION}' >> $versionfile
+	echo 'Metadata Revision: ${METADATA_REVISION}' >> $versionfile
+	echo 'Timestamp: ${DATETIME}' >> $versionfile
+}
+toolchain_create_sdk_version[vardepsexclude] = "DATETIME"

meta/classes/toolchain-scripts.bbclass

@@ -1,4 +1,4 @@
-inherit siteinfo kernel-arch
+inherit toolchain-scripts-base siteinfo kernel-arch
 
 # We want to be able to change the value of MULTIMACH_TARGET_SYS, because it
 # doesn't always match our expectations... but we default to the stock value
@@ -105,7 +105,7 @@ EOF
 }
 
 #we get the cached site config in the runtime
-TOOLCHAIN_CONFIGSITE_NOCACHE = "${@siteinfo_get_files(d, True)}"
+TOOLCHAIN_CONFIGSITE_NOCACHE = "${@siteinfo_get_files(d)}"
 TOOLCHAIN_CONFIGSITE_SYSROOTCACHE = "${STAGING_DIR}/${MLPREFIX}${MACHINE}/${target_datadir}/${TARGET_SYS}_config_site.d"
 TOOLCHAIN_NEED_CONFIGSITE_CACHE ??= "virtual/${MLPREFIX}libc ncurses"
 
@@ -136,18 +136,6 @@ toolchain_create_sdk_siteconfig () {
 # The immediate expansion above can result in unwanted path dependencies here
 toolchain_create_sdk_siteconfig[vardepsexclude] = "TOOLCHAIN_CONFIGSITE_SYSROOTCACHE"
 
-#This function create a version information file
-toolchain_create_sdk_version () {
-	local versionfile=$1
-	rm -f $versionfile
-	touch $versionfile
-	echo 'Distro: ${DISTRO}' >> $versionfile
-	echo 'Distro Version: ${DISTRO_VERSION}' >> $versionfile
-	echo 'Metadata Revision: ${METADATA_REVISION}' >> $versionfile
-	echo 'Timestamp: ${DATETIME}' >> $versionfile
-}
-toolchain_create_sdk_version[vardepsexclude] = "DATETIME"
-
 python __anonymous () {
     import oe.classextend
     deps = ""

meta/classes/useradd.bbclass

@@ -125,7 +125,7 @@ useradd_sysroot_sstate () {
 
 userdel_sysroot_sstate () {
 if test "x${STAGING_DIR_TARGET}" != "x"; then
-    if [ "${BB_CURRENTTASK}" = "configure" -o "${BB_CURRENTTASK}" = "clean" ]; then
+    if [ "${BB_CURRENTTASK}" = "clean" ]; then
         export PSEUDO="${FAKEROOTENV} PSEUDO_LOCALSTATEDIR=${STAGING_DIR_TARGET}${localstatedir}/pseudo ${STAGING_DIR_NATIVE}${bindir_native}/pseudo"
         OPT="--root ${STAGING_DIR_TARGET}"
 

meta/classes/useradd_base.bbclass

@@ -51,14 +51,6 @@ perform_groupmems () {
 	local groupname=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-g" || $i == "--group") print $(i+1) }'`
 	local username=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-a" || $i == "--add") print $(i+1) }'`
 	bbnote "${PN}: Running groupmems command with group $groupname and user $username"
-	# groupmems fails if /etc/gshadow does not exist
-	local gshadow=""
-	if [ -f $rootdir${sysconfdir}/gshadow ]; then
-		gshadow="yes"
-	else
-		gshadow="no"
-		touch $rootdir${sysconfdir}/gshadow
-	fi
 	local mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`"
 	if test "x$mem_exists" = "x"; then
 		eval flock -x $rootdir${sysconfdir} -c \"$PSEUDO groupmems \$opts\" || true
@@ -69,10 +61,6 @@ perform_groupmems () {
 	else
 		bbnote "${PN}: group $groupname already contains $username, not re-adding it"
 	fi
-	if test "x$gshadow" = "xno"; then
-		rm -f $rootdir${sysconfdir}/gshadow
-		rm -f $rootdir${sysconfdir}/gshadow-
-	fi
 }
 
 perform_groupdel () {

meta/conf/bitbake.conf

@@ -129,7 +129,7 @@ TARGET_CC_ARCH = "${TUNE_CCARGS}"
 TARGET_LD_ARCH = "${TUNE_LDARGS}"
 TARGET_AS_ARCH = "${TUNE_ASARGS}"
 
-SDK_ARCH = "${BUILD_ARCH}"
+SDKMACHINE ??= "x86_64"
 SDK_OS = "${BUILD_OS}"
 SDK_VENDOR = "-oesdk"
 SDK_SYS = "${SDK_ARCH}${SDK_VENDOR}${@['-' + d.getVar('SDK_OS', True), ''][d.getVar('SDK_OS', True) == ('' or 'custom')]}"
@@ -802,7 +802,7 @@ BB_CONSOLELOG ?= "${LOG_DIR}/cooker/${MACHINE}/${DATETIME}.log"
 
 # Setup our default hash policy
 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
-BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
+BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
     SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
     USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
     PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \

meta/conf/distro/include/default-distrovars.inc

@@ -23,7 +23,7 @@ DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT} ${DISTRO_FEATURES_LIBC}"
 IMAGE_FEATURES ?= ""
 
 WHITELIST_GPL-3.0 ?= ""
-LGPLv2_WHITELIST_GPL-3.0 ?= "libidn"
+LGPLv2_WHITELIST_GPL-3.0 ?= ""
 
 COMMERCIAL_AUDIO_PLUGINS ?= ""
 # COMMERCIAL_AUDIO_PLUGINS ?= "gst-plugins-ugly-mad gst-plugins-ugly-mpegaudioparse"

meta/conf/machine/include/qemuboot-mips.inc

@@ -3,5 +3,5 @@ IMAGE_CLASSES += "qemuboot"
 QB_MEM = "-m 256"
 QB_MACHINE = "-machine malta"
 QB_KERNEL_CMDLINE_APPEND = "console=ttyS0 console=tty"
-QB_OPT_APPEND = "-vga cirrus -show-cursor -usb -usbdevice tablet"
+QB_OPT_APPEND = "-vga cirrus -show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
 QB_SYSTEM_NAME = "qemu-system-${TUNE_ARCH}"

meta/conf/machine/include/qemuboot-x86.inc

@@ -10,5 +10,5 @@ QB_CPU_KVM_x86-64 = "-cpu kvm64"
 
 QB_AUDIO_DRV = "alsa"
 QB_AUDIO_OPT = "-soundhw ac97,es1370"
-QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1"
-QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet"
+QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1"
+QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci"

meta/conf/machine/qemuarm.conf

@@ -14,6 +14,6 @@ SERIAL_CONSOLES = "115200;ttyAMA0 115200;ttyAMA1"
 QB_SYSTEM_NAME = "qemu-system-arm"
 QB_MACHINE = "-machine versatilepb"
 QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,115200 console=tty"
-QB_OPT_APPEND = "-show-cursor -usb -usbdevice tablet"
+QB_OPT_APPEND = "-show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
 PREFERRED_VERSION_linux-yocto ??= "4.8%"
 QB_DTB = "${@base_version_less_or_equal('PREFERRED_VERSION_linux-yocto', '4.7', '', 'zImage-versatile-pb.dtb', d)}"

meta/conf/machine/qemuarm64.conf

@@ -15,7 +15,7 @@ QB_MEM = "-m 512"
 QB_MACHINE = "-machine virt"
 QB_CPU = "-cpu cortex-a57"
 QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,38400"
-QB_OPT_APPEND = "-show-cursor"
+QB_OPT_APPEND = "-show-cursor -device virtio-rng-pci"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device virtio-net-device,netdev=net0,mac=@MAC@"
 QB_SLIRP_OPT = "-netdev user,id=net0 -device virtio-net-device,netdev=net0"
 QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=raw -device virtio-blk-device,drive=disk0"

meta/conf/machine/qemuppc.conf

@@ -15,6 +15,6 @@ SERIAL_CONSOLES = "115200;ttyS0 115200;ttyS1"
 QB_SYSTEM_NAME = "qemu-system-ppc"
 QB_MACHINE = "-machine mac99"
 QB_CPU = "-cpu G4"
-QB_KERNEL_CMDLINE_APPEND = "console=ttyS0 console=tty"
-QB_OPT_APPEND = "-show-cursor -usb -usbdevice tablet"
+QB_KERNEL_CMDLINE_APPEND = "console=tty console=ttyS0"
+QB_OPT_APPEND = "-show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=@MAC@"

meta/files/toolchain-shar-relocate.sh

@@ -1,3 +1,8 @@
+if ! xargs --version > /dev/null 2>&1; then
+	echo "xargs is required by the relocation script, please install it first. Abort!"
+	exit 1
+fi
+
 # fix dynamic loader paths in all ELF SDK binaries
 native_sysroot=$($SUDO_EXEC cat $env_setup_script |grep 'OECORE_NATIVE_SYSROOT='|cut -d'=' -f2|tr -d '"')
 dl_path=$($SUDO_EXEC find $native_sysroot/lib -name "ld-linux*")
@@ -7,6 +12,10 @@ if [ "$dl_path" = "" ] ; then
 fi
 executable_files=$($SUDO_EXEC find $native_sysroot -type f \
 	\( -perm -0100 -o -perm -0010 -o -perm -0001 \) -printf "'%h/%f' ")
+if [ "x$executable_files" = "x" ]; then
+   echo "SDK relocate failed, could not get executalbe files"
+   exit 1
+fi
 
 tdir=`mktemp -d`
 if [ x$tdir = x ] ; then
@@ -14,7 +23,7 @@ if [ x$tdir = x ] ; then
    exit 1
 fi
 cat <<EOF >> $tdir/relocate_sdk.sh
-#!/bin/bash
+#!/bin/sh
 for py in python python2 python3
 do
 	PYTHON=\`which \${py} 2>/dev/null\`
@@ -53,9 +62,18 @@ done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
         -e "s:^#! */usr/bin/perl.*:#! /usr/bin/env perl:g" \
         -e "s: /usr/bin/perl: /usr/bin/env perl:g"
 
+if [ $? -ne 0 ]; then
+	echo "Failed to replace perl. Relocate script failed. Abort!"
+	exit 1
+fi
+
 # change all symlinks pointing to @SDKPATH@
 for l in $($SUDO_EXEC find $native_sysroot -type l); do
 	$SUDO_EXEC ln -sfn $(readlink $l|$SUDO_EXEC sed -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:") $l
+	if [ $? -ne 0 ]; then
+		echo "Failed to setup symlinks. Relocate script failed. Abort!"
+		exit 1
+    fi
 done
 
 echo done

meta/lib/oe/copy_buildsystem.py

@@ -17,7 +17,7 @@ class BuildSystem(object):
     def __init__(self, context, d):
         self.d = d
         self.context = context
-        self.layerdirs = d.getVar('BBLAYERS', True).split()
+        self.layerdirs = [os.path.abspath(pth) for pth in d.getVar('BBLAYERS', True).split()]
         self.layers_exclude = (d.getVar('SDK_LAYERS_EXCLUDE', True) or "").split()
 
     def copy_bitbake_and_layers(self, destdir, workspace_name=None):
@@ -26,7 +26,7 @@ class BuildSystem(object):
         bb.utils.mkdirhier(destdir)
         layers = list(self.layerdirs)
 
-        corebase = self.d.getVar('COREBASE', True)
+        corebase = os.path.abspath(self.d.getVar('COREBASE', True))
         layers.append(corebase)
 
         # Exclude layers

meta/lib/oe/package_manager.py

@@ -176,7 +176,7 @@ class RpmIndexer(Indexer):
             dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb', arch)
             if os.path.exists(dbpath):
                 bb.utils.remove(dbpath, True)
-            arch_dir = os.path.join(self.d.getVar('WORKDIR', True), 'rpms', arch)
+            arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
                 continue
 
@@ -686,7 +686,8 @@ class RpmPM(PackageManager):
         if not os.path.exists(self.d.expand('${T}/saved')):
             bb.utils.mkdirhier(self.d.expand('${T}/saved'))
 
-        self.indexer = RpmIndexer(self.d, self.deploy_dir)
+        packageindex_dir = os.path.join(self.d.getVar('WORKDIR', True), 'rpms')
+        self.indexer = RpmIndexer(self.d, packageindex_dir)
         self.pkgs_list = RpmPkgsList(self.d, self.target_rootfs, arch_var, os_var)
 
         self.ml_prefix_list, self.ml_os_list = self.indexer.get_ml_prefix_and_os_list(arch_var, os_var)
@@ -838,11 +839,13 @@ class RpmPM(PackageManager):
                     new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs)
                     if not new_pkg:
                         # Failed to translate, package not found!
-                        err_msg = '%s not found in the %s feeds (%s).\n' % \
-                                  (pkg, mlib, " ".join(feed_archs))
+                        err_msg = '%s not found in the %s feeds (%s) in %s.' % \
+                                  (pkg, mlib, " ".join(feed_archs), self.d.getVar('DEPLOY_DIR_RPM', True))
                         if not attempt_only:
-                            err_msg += " ".join(self.fullpkglist)
-                            bb.fatal(err_msg)
+                            bb.error(err_msg)
+                            bb.fatal("This is often caused by an empty package declared " \
+                                     "in a recipe's PACKAGES variable. (Empty packages are " \
+                                     "not constructed unless ALLOW_EMPTY_<pkg> = '1' is used.)")
                         bb.warn(err_msg)
                     else:
                         new_pkgs.append(new_pkg)
@@ -855,11 +858,13 @@ class RpmPM(PackageManager):
                 default_archs = self.ml_prefix_list['default']
                 new_pkg = self._search_pkg_name_in_feeds(pkg, default_archs)
                 if not new_pkg:
-                    err_msg = '%s not found in the base feeds (%s).\n' % \
-                              (pkg, ' '.join(default_archs))
+                    err_msg = '%s not found in the feeds (%s) in %s.' % \
+                                  (pkg, " ".join(default_archs), self.d.getVar('DEPLOY_DIR_RPM', True))
                     if not attempt_only:
-                        err_msg += " ".join(self.fullpkglist)
-                        bb.fatal(err_msg)
+                        bb.error(err_msg)
+                        bb.fatal("This is often caused by an empty package declared " \
+                                 "in a recipe's PACKAGES variable. (Empty packages are " \
+                                 "not constructed unless ALLOW_EMPTY_<pkg> = '1' is used.)")
                     bb.warn(err_msg)
                 else:
                     new_pkgs.append(new_pkg)

meta/lib/oe/recipeutils.py

@@ -449,8 +449,8 @@ def get_recipe_patched_files(d):
 def validate_pn(pn):
     """Perform validation on a recipe name (PN) for a new recipe."""
     reserved_names = ['forcevariable', 'append', 'prepend', 'remove']
-    if not re.match('[0-9a-z-.]+', pn):
-        return 'Recipe name "%s" is invalid: only characters 0-9, a-z, - and . are allowed' % pn
+    if not re.match('^[0-9a-z-.+]+$', pn):
+        return 'Recipe name "%s" is invalid: only characters 0-9, a-z, -, + and . are allowed' % pn
     elif pn in reserved_names:
         return 'Recipe name "%s" is invalid: is a reserved keyword' % pn
     elif pn.startswith('pn-'):

meta/lib/oe/utils.py

@@ -23,13 +23,13 @@ def ifelse(condition, iftrue = True, iffalse = False):
         return iffalse
 
 def conditional(variable, checkvalue, truevalue, falsevalue, d):
-    if d.getVar(variable,1) == checkvalue:
+    if d.getVar(variable, True) == checkvalue:
         return truevalue
     else:
         return falsevalue
 
 def less_or_equal(variable, checkvalue, truevalue, falsevalue, d):
-    if float(d.getVar(variable,1)) <= float(checkvalue):
+    if float(d.getVar(variable, True)) <= float(checkvalue):
         return truevalue
     else:
         return falsevalue

meta/lib/oeqa/buildperf/base.py

@@ -428,6 +428,11 @@ class BuildPerfTestCase(unittest.TestCase):
             n_e_v, revision = nevr.rsplit('-', 1)
             match = re.match(r'^(?P<name>\S+)-((?P<epoch>[0-9]{1,5})_)?(?P<version>[0-9]\S*)$',
                              n_e_v)
+            if not match:
+                # If we're not able to parse a version starting with a number, just
+                # take the part after last dash
+                match = re.match(r'^(?P<name>\S+)-((?P<epoch>[0-9]{1,5})_)?(?P<version>[^-]+)$',
+                                 n_e_v)
             name = match.group('name')
             version = match.group('version')
             epoch = match.group('epoch')

meta/lib/oeqa/oetest.py

@@ -152,13 +152,12 @@ class oeSDKTest(oeTest):
 
     @classmethod
     def hasHostPackage(self, pkg):
-
         if re.search(pkg, oeTest.tc.hostpkgmanifest):
             return True
         return False
 
     def _run(self, cmd):
-        return subprocess.check_output(". %s > /dev/null; %s;" % (self.tc.sdkenv, cmd), shell=True).decode("utf-8")
+        return subprocess.check_output(". %s > /dev/null; %s;" % (self.tc.sdkenv, cmd), shell=True, stderr=subprocess.STDOUT).decode("utf-8")
 
 class oeSDKExtTest(oeSDKTest):
     def _run(self, cmd):
@@ -170,7 +169,7 @@ class oeSDKExtTest(oeSDKTest):
         env['PATH'] = avoid_paths_in_environ(paths_to_avoid)
 
         return subprocess.check_output(". %s > /dev/null;"\
-            " %s;" % (self.tc.sdkenv, cmd), shell=True, env=env).decode("utf-8")
+            " %s;" % (self.tc.sdkenv, cmd), stderr=subprocess.STDOUT, shell=True, env=env).decode("utf-8")
 
 def getmodule(pos=2):
     # stack returns a list of tuples containg frame information

meta/lib/oeqa/runtime/parselogs.py

@@ -68,6 +68,7 @@ qemux86_common = [
     'Getting VBE info block failed',
     'vbe_init() failed with -22',
     'uvesafb: mode switch failed',
+    'tsc: HPET/PMTIMER calibration failed',
 ] + common_errors
 
 ignore_errors = { 
@@ -181,6 +182,9 @@ class ParseLogsTest(oeRuntimeTest):
     def getMachine(self):
         return oeRuntimeTest.tc.d.getVar("MACHINE", True)
 
+    def getWorkdir(self):
+        return oeRuntimeTest.tc.d.getVar("WORKDIR", True)
+
     #get some information on the CPU of the machine to display at the beginning of the output. This info might be useful in some cases.
     def getHardwareInfo(self):
         hwi = ""
@@ -218,16 +222,19 @@ class ParseLogsTest(oeRuntimeTest):
 
     #copy the log files to be parsed locally
     def transfer_logs(self, log_list):
-        target_logs = 'target_logs'
+        workdir = self.getWorkdir()
+        self.target_logs = workdir + '/' + 'target_logs'
+        target_logs = self.target_logs
         if not os.path.exists(target_logs):
             os.makedirs(target_logs)
+        bb.utils.remove(self.target_logs + "/*")
         for f in log_list:
             self.target.copy_from(f, target_logs)
 
     #get the local list of logs
     def get_local_log_list(self, log_locations):
         self.transfer_logs(self.getLogList(log_locations))
-        logs = [ os.path.join('target_logs',f) for f in os.listdir('target_logs') if os.path.isfile(os.path.join('target_logs',f)) ]
+        logs = [ os.path.join(self.target_logs, f) for f in os.listdir(self.target_logs) if os.path.isfile(os.path.join(self.target_logs, f)) ]
         return logs
 
     #build the grep command to be used with filters and exclusions

meta/lib/oeqa/runtime/rpm.py

@@ -51,12 +51,32 @@ class RpmInstallRemoveTest(oeRuntimeTest):
     @testcase(1096)
     @skipUnlessPassed('test_ssh')
     def test_rpm_query_nonroot(self):
-        (status, output) = self.target.run('useradd test1')
-        self.assertTrue(status == 0, msg="Failed to create new user: " + output)
-        (status, output) = self.target.run('su -c id test1')
-        self.assertTrue('(test1)' in output, msg="Failed to execute as new user")
-        (status, output) = self.target.run('su -c "rpm -qa" test1 ')
-        self.assertEqual(status, 0, msg="status: %s. Cannot run rpm -qa: %s" % (status, output))
+
+        def set_up_test_user(u):
+            (status, output) = self.target.run("id -u %s" % u)
+            if status == 0:
+                pass
+            else:
+                (status, output) = self.target.run("useradd %s" % u)
+                self.assertTrue(status == 0, msg="Failed to create new user: " + output)
+
+        def exec_as_test_user(u):
+            (status, output) = self.target.run("su -c id %s" % u)
+            self.assertTrue("({0})".format(u) in output, msg="Failed to execute as new user")
+            (status, output) = self.target.run("su -c \"rpm -qa\" %s " % u)
+            self.assertEqual(status, 0, msg="status: %s. Cannot run rpm -qa: %s" % (status, output))
+
+        def unset_up_test_user(u):
+            (status, output) = self.target.run("userdel -r %s" % u)
+            self.assertTrue(status == 0, msg="Failed to erase user: %s" % output)
+
+        tuser = 'test1'
+
+        try:
+            set_up_test_user(tuser)
+            exec_as_test_user(tuser)
+        finally:
+            unset_up_test_user(tuser)
 
     @testcase(195)
     @skipUnlessPassed('test_rpm_install')