recipes: Move out stale GPLv2 versions to a seperate layer

These are recipes where the upstream has moved to GPLv3 and these old
versions are the last ones under the GPLv2 license.

There are several reasons for making this move. There is a different
quality of service with these recipes in that they don't get security
fixes and upstream no longer care about them, in fact they're actively
hostile against people using old versions. The recipes tend to need a
different kind of maintenance to work with changes in the wider ecosystem
and there needs to be isolation between changes made in the v3 versions
and those in the v2 versions.

There are probably better ways to handle a "non-GPLv3" system but right
now having these in OE-Core makes them look like a first class citizen
when I believe they have potential for a variety of undesireable issues.

Moving them into a separate layer makes their different needs clearer, it
also makes it clear how many of these there are. Some are probably not
needed (e.g. mc), I also wonder whether some are useful (e.g. gmp)
since most things that use them are GPLv3 only already. Someone could
now more clearly see how to streamline the list of recipes here.

I'm proposing we mmove to this separate layer for 2.3 with its future
maintinership and testing to be determined in 2.4 and beyond.

(From OE-Core rev: 19b7e950346fb1dde6505c45236eba6cd9b33b4b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

bitbake/bin/bitbake

@@ -38,7 +38,7 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException
 if sys.getfilesystemencoding() != "utf-8":
     sys.exit("Please use a locale setting which supports utf-8.\nPython can't change the filesystem locale after loading so we need a utf-8 when python starts or things won't work.")
 
-__version__ = "1.33.0"
+__version__ = "1.33.2"
 
 if __name__ == "__main__":
     if __version__ != bb.__version__:

bitbake/bin/bitbake-layers

@@ -31,15 +31,6 @@ sys.path[0:0] = [os.path.join(topdir, 'lib')]
 
 import bb.tinfoil
 
-
-def tinfoil_init(parserecipes):
-    import bb.tinfoil
-    tinfoil = bb.tinfoil.Tinfoil(tracking=True)
-    tinfoil.prepare(not parserecipes)
-    tinfoil.logger.setLevel(logger.getEffectiveLevel())
-    return tinfoil
-
-
 def logger_create(name, output=sys.stderr):
     logger = logging.getLogger(name)
     loggerhandler = logging.StreamHandler(output)
@@ -86,8 +77,10 @@ def main():
     logger_setup_color(logger, global_args.color)
 
     plugins = []
-    tinfoil = tinfoil_init(False)
+    tinfoil = bb.tinfoil.Tinfoil(tracking=True)
+    tinfoil.logger.setLevel(logger.getEffectiveLevel())
     try:
+        tinfoil.prepare(True)
         for path in ([topdir] +
                     tinfoil.config_data.getVar('BBPATH').split(':')):
             pluginpath = os.path.join(path, 'lib', 'bblayers')

bitbake/bin/toaster

@@ -151,15 +151,7 @@ fi
 
 unset OE_ROOT
 
-# this defines the dir toaster will use for
-# 1) clones of layers (in _toaster_clones )
-# 2) the build dir (in build)
-# 3) the sqlite db if that is being used.
-# 4) pid's we need to clean up on exit/shutdown
-# note: for future. in order to make this an arbitrary directory, we need to
-# make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
-export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE TOASTER_DIR"
-export TOASTER_DIR=`pwd`
+
 
 WEBSERVER=1
 ADDR_PORT="localhost:8000"
@@ -215,10 +207,8 @@ fi
 # 2) the build dir (in build)
 # 3) the sqlite db if that is being used.
 # 4) pid's we need to clean up on exit/shutdown
-# note: for future. in order to make this an arbitrary directory, we need to
-# make sure that the toaster.sqlite file doesn't default to `pwd`
-# like it currently does.
 export TOASTER_DIR=`dirname $BUILDDIR`
+export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE TOASTER_DIR"
 
 # Determine the action. If specified by arguments, fine, if not, toggle it
 if [ "$CMD" = "start" ] ; then
@@ -255,7 +245,10 @@ case $CMD in
             return 4
         fi
         export BITBAKE_UI='toasterui'
-        $MANAGE runbuilds &
+        $MANAGE runbuilds \
+           </dev/null >>${BUILDDIR}/toaster_runbuilds.log 2>&1 \
+           & echo $! >${BUILDDIR}/.runbuilds.pid
+
         # set fail safe stop system on terminal exit
         trap stop_system SIGHUP
         echo "Successful ${CMD}."

bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml

@@ -385,7 +385,8 @@
                 The supported parameters are as follows:
                 <itemizedlist>
                     <listitem><para><emphasis>"method":</emphasis>
-                        The protocol over which to communicate with the CVS server.
+                        The protocol over which to communicate with the CVS
+                        server.
                         By default, this protocol is "pserver".
                         If "method" is set to "ext", BitBake examines the
                         "rsh" parameter and sets <filename>CVS_RSH</filename>.
@@ -469,25 +470,29 @@
                         You can think of this parameter as the top-level
                         directory of the repository data you want.
                         </para></listitem>
+                    <listitem><para><emphasis>"path_spec":</emphasis>
+                        A specific directory in which to checkout the
+                        specified svn module.
+                        </para></listitem>
                     <listitem><para><emphasis>"protocol":</emphasis>
                         The protocol to use, which defaults to "svn".
-                        Other options are "svn+ssh" and "rsh".
-                        For "rsh", the "rsh" parameter is also used.
+                        If "protocol" is set to "svn+ssh", the "ssh"
+                        parameter is also used.
                         </para></listitem>
                     <listitem><para><emphasis>"rev":</emphasis>
                         The revision of the source code to checkout.
                         </para></listitem>
-                    <listitem><para><emphasis>"date":</emphasis>
-                        The date of the source code to checkout.
-                        Specific revisions are generally much safer to checkout
-                        rather than by date as they do not involve timezones
-                        (e.g. they are much more deterministic).
-                        </para></listitem>
                     <listitem><para><emphasis>"scmdata":</emphasis>
                         Causes the “.svn” directories to be available during
                         compile-time when set to "keep".
                         By default, these directories are removed.
                         </para></listitem>
+                    <listitem><para><emphasis>"ssh":</emphasis>
+                        An optional parameter used when "protocol" is set
+                        to "svn+ssh".
+                        You can use this parameter to specify the ssh
+                        program used by svn.
+                        </para></listitem>
                     <listitem><para><emphasis>"transportuser":</emphasis>
                         When required, sets the username for the transport.
                         By default, this parameter is empty.
@@ -496,10 +501,11 @@
                         command.
                         </para></listitem>
                 </itemizedlist>
-                Following are two examples using svn:
+                Following are three examples using svn:
                 <literallayout class='monospaced'>
-     SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
-     SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
+     SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"
+     SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"
+     SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"
                 </literallayout>
             </para>
         </section>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml

@@ -529,7 +529,7 @@
        -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
                              Show debug logging for the specified logging domains
        -P, --profile         Profile the command and save reports.
-       -u UI, --ui=UI        The user interface to use (depexp, goggle, knotty or
+       -u UI, --ui=UI        The user interface to use (taskexp, knotty or
                              ncurses - default knotty).
        -t SERVERTYPE, --servertype=SERVERTYPE
                              Choose which server type to use (process or xmlrpc -
@@ -666,21 +666,21 @@
                 </para>
 
                 <para>
-                    When you generate a dependency graph, BitBake writes four files
+                    When you generate a dependency graph, BitBake writes three files
                     to the current working directory:
                     <itemizedlist>
-                        <listitem><para><emphasis><filename>package-depends.dot</filename>:</emphasis>
-                            Shows BitBake's knowledge of dependencies between
-                            runtime targets.
+                        <listitem><para>
+                            <emphasis><filename>recipe-depends.dot</filename>:</emphasis>
+                            Shows dependencies between recipes (i.e. a collapsed version of
+                            <filename>task-depends.dot</filename>).
                             </para></listitem>
-                        <listitem><para><emphasis><filename>pn-depends.dot</filename>:</emphasis>
-                            Shows dependencies between build-time targets
-                            (i.e. recipes).
-                            </para></listitem>
-                        <listitem><para><emphasis><filename>task-depends.dot</filename>:</emphasis>
+                        <listitem><para>
+                            <emphasis><filename>task-depends.dot</filename>:</emphasis>
                             Shows dependencies between tasks.
+                            These dependencies match BitBake's internal task execution list.
                             </para></listitem>
-                        <listitem><para><emphasis><filename>pn-buildlist</filename>:</emphasis>
+                        <listitem><para>
+                            <emphasis><filename>pn-buildlist</filename>:</emphasis>
                             Shows a simple list of targets that are to be built.
                             </para></listitem>
                     </itemizedlist>

bitbake/doc/bitbake.1

@@ -105,7 +105,7 @@ Show debug logging for the specified logging domains
 profile the command and print a report
 .TP
 .B \-uUI, \-\-ui=UI
-User interface to use. Currently, hob, depexp, goggle or ncurses can be specified as UI.
+User interface to use. Currently, knotty, taskexp or ncurses can be specified as UI.
 .TP
 .B \-tSERVERTYPE, \-\-servertype=SERVERTYPE
 Choose which server to use, none, process or xmlrpc.

bitbake/lib/bb/__init__.py

@@ -21,7 +21,7 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-__version__ = "1.33.0"
+__version__ = "1.33.2"
 
 import sys
 if sys.version_info < (3, 4, 0):

bitbake/lib/bb/build.py

@@ -878,3 +878,30 @@ def preceedtask(task, with_recrdeptasks, d):
         if recrdeptask:
             preceed.update(recrdeptask.split())
     return preceed
+
+def tasksbetween(task_start, task_end, d):
+    """
+    Return the list of tasks between two tasks in the current recipe,
+    where task_start is to start at and task_end is the task to end at
+    (and task_end has a dependency chain back to task_start).
+    """
+    outtasks = []
+    tasks = list(filter(lambda k: d.getVarFlag(k, "task"), d.keys()))
+    def follow_chain(task, endtask, chain=None):
+        if not chain:
+            chain = []
+        chain.append(task)
+        for othertask in tasks:
+            if othertask == task:
+                continue
+            if task == endtask:
+                for ctask in chain:
+                    if ctask not in outtasks:
+                        outtasks.append(ctask)
+            else:
+                deps = d.getVarFlag(othertask, 'deps', False)
+                if task in deps:
+                    follow_chain(othertask, endtask, chain)
+        chain.pop()
+    follow_chain(task_start, task_end)
+    return outtasks

bitbake/lib/bb/cooker.py

@@ -642,7 +642,6 @@ class BBCooker:
             logger.plain(env.getvalue())
 
         # emit variables and shell functions
-        data.update_data(envdata)
         with closing(StringIO()) as env:
             data.emit_env(env, envdata, True)
             logger.plain(env.getvalue())
@@ -703,7 +702,6 @@ class BBCooker:
         for mc in self.multiconfigs:
             taskdata[mc] = bb.taskdata.TaskData(abort, skiplist=self.skiplist, allowincomplete=allowincomplete)
             localdata[mc] = data.createCopy(self.databuilder.mcdata[mc])
-            bb.data.update_data(localdata[mc])
             bb.data.expandKeys(localdata[mc])
 
         current = 0
@@ -949,62 +947,54 @@ class BBCooker:
 
         depgraph = self.generateTaskDepTreeData(pkgs_to_build, task)
 
-        # Prints a flattened form of package-depends below where subpackages of a package are merged into the main pn
-        depends_file = open('pn-depends.dot', 'w' )
-        buildlist_file = open('pn-buildlist', 'w' )
-        print("digraph depends {", file=depends_file)
-        for pn in depgraph["pn"]:
-            fn = depgraph["pn"][pn]["filename"]
-            version = depgraph["pn"][pn]["version"]
-            print('"%s" [label="%s %s\\n%s"]' % (pn, pn, version, fn), file=depends_file)
-            print("%s" % pn, file=buildlist_file)
-        buildlist_file.close()
+        with open('pn-buildlist', 'w') as f:
+            for pn in depgraph["pn"]:
+                f.write(pn + "\n")
         logger.info("PN build list saved to 'pn-buildlist'")
-        for pn in depgraph["depends"]:
-            for depend in depgraph["depends"][pn]:
-                print('"%s" -> "%s" [style=solid]' % (pn, depend), file=depends_file)
-        for pn in depgraph["rdepends-pn"]:
-            for rdepend in depgraph["rdepends-pn"][pn]:
-                print('"%s" -> "%s" [style=dashed]' % (pn, rdepend), file=depends_file)
-        print("}", file=depends_file)
-        depends_file.close()
-        logger.info("PN dependencies saved to 'pn-depends.dot'")
 
-        depends_file = open('package-depends.dot', 'w' )
-        print("digraph depends {", file=depends_file)
-        for package in depgraph["packages"]:
-            pn = depgraph["packages"][package]["pn"]
-            fn = depgraph["packages"][package]["filename"]
-            version = depgraph["packages"][package]["version"]
-            if package == pn:
-                print('"%s" [label="%s %s\\n%s"]' % (pn, pn, version, fn), file=depends_file)
-            else:
-                print('"%s" [label="%s(%s) %s\\n%s"]' % (package, package, pn, version, fn), file=depends_file)
-            for depend in depgraph["depends"][pn]:
-                print('"%s" -> "%s" [style=solid]' % (package, depend), file=depends_file)
-        for package in depgraph["rdepends-pkg"]:
-            for rdepend in depgraph["rdepends-pkg"][package]:
-                print('"%s" -> "%s" [style=dashed]' % (package, rdepend), file=depends_file)
-        for package in depgraph["rrecs-pkg"]:
-            for rdepend in depgraph["rrecs-pkg"][package]:
-                print('"%s" -> "%s" [style=dotted]' % (package, rdepend), file=depends_file)
-        print("}", file=depends_file)
-        depends_file.close()
-        logger.info("Package dependencies saved to 'package-depends.dot'")
+        # Remove old format output files to ensure no confusion with stale data
+        try:
+            os.unlink('pn-depends.dot')
+        except FileNotFoundError:
+            pass
+        try:
+            os.unlink('package-depends.dot')
+        except FileNotFoundError:
+            pass
 
-        tdepends_file = open('task-depends.dot', 'w' )
-        print("digraph depends {", file=tdepends_file)
-        for task in depgraph["tdepends"]:
-            (pn, taskname) = task.rsplit(".", 1)
-            fn = depgraph["pn"][pn]["filename"]
-            version = depgraph["pn"][pn]["version"]
-            print('"%s.%s" [label="%s %s\\n%s\\n%s"]' % (pn, taskname, pn, taskname, version, fn), file=tdepends_file)
-            for dep in depgraph["tdepends"][task]:
-                print('"%s" -> "%s"' % (task, dep), file=tdepends_file)
-        print("}", file=tdepends_file)
-        tdepends_file.close()
+        with open('task-depends.dot', 'w') as f:
+            f.write("digraph depends {\n")
+            for task in depgraph["tdepends"]:
+                (pn, taskname) = task.rsplit(".", 1)
+                fn = depgraph["pn"][pn]["filename"]
+                version = depgraph["pn"][pn]["version"]
+                f.write('"%s.%s" [label="%s %s\\n%s\\n%s"]\n' % (pn, taskname, pn, taskname, version, fn))
+                for dep in depgraph["tdepends"][task]:
+                    f.write('"%s" -> "%s"\n' % (task, dep))
+            f.write("}\n")
         logger.info("Task dependencies saved to 'task-depends.dot'")
 
+        with open('recipe-depends.dot', 'w') as f:
+            f.write("digraph depends {\n")
+            pndeps = {}
+            for task in depgraph["tdepends"]:
+                (pn, taskname) = task.rsplit(".", 1)
+                if pn not in pndeps:
+                    pndeps[pn] = set()
+                for dep in depgraph["tdepends"][task]:
+                    (deppn, deptaskname) = dep.rsplit(".", 1)
+                    pndeps[pn].add(deppn)
+            for pn in pndeps:
+                fn = depgraph["pn"][pn]["filename"]
+                version = depgraph["pn"][pn]["version"]
+                f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn, version, fn))
+                for dep in pndeps[pn]:
+                    if dep == pn:
+                        continue
+                    f.write('"%s" -> "%s"\n' % (pn, dep))
+            f.write("}\n")
+        logger.info("Flatened recipe dependencies saved to 'recipe-depends.dot'")
+
     def show_appends_with_no_recipes(self):
         # Determine which bbappends haven't been applied
 
@@ -1035,7 +1025,6 @@ class BBCooker:
 
         for mc in self.multiconfigs:
             localdata = data.createCopy(self.databuilder.mcdata[mc])
-            bb.data.update_data(localdata)
             bb.data.expandKeys(localdata)
 
             # Handle PREFERRED_PROVIDERS
@@ -1641,7 +1630,8 @@ class BBCooker:
         if self.state != state.parsing and not self.parsecache_valid:
             self.parseConfiguration ()
             if CookerFeatures.SEND_SANITYEVENTS in self.featureset:
-                bb.event.fire(bb.event.SanityCheck(False), self.data)
+                for mc in self.multiconfigs:
+                    bb.event.fire(bb.event.SanityCheck(False), self.databuilder.mcdata[mc])
 
             for mc in self.multiconfigs:
                 ignore = self.databuilder.mcdata[mc].getVar("ASSUME_PROVIDED") or ""
@@ -1910,6 +1900,11 @@ class CookerCollectFiles(object):
             # that do not compile
             bbmasks = []
             for mask in bbmask.split():
+                # When constructing an older style single regex, it's possible for BBMASK
+                # to end up beginning with '|', which matches and masks _everything_.
+                if mask.startswith("|"):
+                    collectlog.warn("BBMASK contains regular expression beginning with '|', fixing: %s" % mask)
+                    mask = mask[1:]
                 try:
                     re.compile(mask)
                     bbmasks.append(mask)

bitbake/lib/bb/data_smart.py

@@ -976,7 +976,6 @@ class DataSmart(MutableMapping):
         data = {}
         d = self.createCopy()
         bb.data.expandKeys(d)
-        bb.data.update_data(d)
 
         config_whitelist = set((d.getVar("BB_HASHCONFIG_WHITELIST") or "").split())
         keys = set(key for key in iter(d) if not key.startswith("__"))

bitbake/lib/bb/fetch2/__init__.py

@@ -35,9 +35,9 @@ import operator
 import collections
 import subprocess
 import pickle
+import errno
 import bb.persist_data, bb.utils
 import bb.checksum
-from bb import data
 import bb.process
 
 __version__ = "2"
@@ -355,7 +355,7 @@ def decodeurl(url):
     user, password, parameters).
     """
 
-    m = re.compile('(?P<type>[^:]*)://((?P<user>[^/]+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
+    m = re.compile('(?P<type>[^:]*)://((?P<user>[^/;]+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
     if not m:
         raise MalformedUrl(url)
 
@@ -537,7 +537,11 @@ def fetcher_compare_revisions():
     return False
 
 def mirror_from_string(data):
-    return [ i.split() for i in (data or "").replace('\\n','\n').split('\n') if i ]
+    mirrors = (data or "").replace('\\n',' ').split()
+    # Split into pairs
+    if len(mirrors) % 2 != 0:
+        bb.warn('Invalid mirror data %s, should have paired members.' % data)
+    return list(zip(*[iter(mirrors)]*2))
 
 def verify_checksum(ud, d, precomputed={}):
     """
@@ -724,7 +728,7 @@ def get_autorev(d):
 
 def get_srcrev(d, method_name='sortable_revision'):
     """
-    Return the revsion string, usually for use in the version string (PV) of the current package
+    Return the revision string, usually for use in the version string (PV) of the current package
     Most packages usually only have one SCM so we just pass on the call.
     In the multi SCM case, we build a value based on SRCREV_FORMAT which must
     have been set.
@@ -998,6 +1002,12 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
     except bb.fetch2.NetworkAccess:
         raise
 
+    except IOError as e:
+        if e.errno in [os.errno.ESTALE]:
+            logger.warn("Stale Error Observed %s." % ud.url)
+            return False
+        raise
+
     except bb.fetch2.BBFetchException as e:
         if isinstance(e, ChecksumError):
             logger.warning("Mirror checksum failure for url %s (original url: %s)\nCleaning and trying again." % (ud.url, origud.url))
@@ -1172,7 +1182,7 @@ class FetchData(object):
         self.mirrortarball = None
         self.basename = None
         self.basepath = None
-        (self.type, self.host, self.path, self.user, self.pswd, self.parm) = decodeurl(data.expand(url, d))
+        (self.type, self.host, self.path, self.user, self.pswd, self.parm) = decodeurl(d.expand(url))
         self.date = self.getSRCDate(d)
         self.url = url
         if not self.user and "user" in self.parm:
@@ -1189,13 +1199,13 @@ class FetchData(object):
             self.sha256_name = "sha256sum"
         if self.md5_name in self.parm:
             self.md5_expected = self.parm[self.md5_name]
-        elif self.type not in ["http", "https", "ftp", "ftps", "sftp"]:
+        elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3"]:
             self.md5_expected = None
         else:
             self.md5_expected = d.getVarFlag("SRC_URI", self.md5_name)
         if self.sha256_name in self.parm:
             self.sha256_expected = self.parm[self.sha256_name]
-        elif self.type not in ["http", "https", "ftp", "ftps", "sftp"]:
+        elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3"]:
             self.sha256_expected = None
         else:
             self.sha256_expected = d.getVarFlag("SRC_URI", self.sha256_name)
@@ -1247,7 +1257,7 @@ class FetchData(object):
         self.donestamp = basepath + '.done'
         self.lockfile = basepath + '.lock'
 
-    def setup_revisons(self, d):
+    def setup_revisions(self, d):
         self.revisions = {}
         for name in self.names:
             self.revisions[name] = srcrev_internal_helper(self, d, name)
@@ -1394,6 +1404,10 @@ class FetchMethod(object):
                 cmd = 'lzip -dc %s | tar x --no-same-owner -f -' % file
             elif file.endswith('.lz'):
                 cmd = 'lzip -dc %s > %s' % (file, efile)
+            elif file.endswith('.tar.7z'):
+                cmd = '7z x -so %s | tar x --no-same-owner -f -' % file
+            elif file.endswith('.7z'):
+                cmd = '7za x -y %s 1>/dev/null' % file
             elif file.endswith('.zip') or file.endswith('.jar'):
                 try:
                     dos = bb.utils.to_boolean(urldata.parm.get('dos'), False)
@@ -1425,10 +1439,6 @@ class FetchMethod(object):
                 else:
                     raise UnpackError("Unable to unpack deb/ipk package - could not list contents", urldata.url)
                 cmd = 'ar x %s %s && tar --no-same-owner -xpf %s && rm %s' % (file, datafile, datafile, datafile)
-            elif file.endswith('.tar.7z'):
-                cmd = '7z x -so %s | tar xf - ' % file
-            elif file.endswith('.7z'):
-                cmd = '7za x -y %s 1>/dev/null' % file
 
         # If 'subdir' param exists, create a dir and use it as destination for unpack cmd
         if 'subdir' in urldata.parm:
@@ -1646,6 +1656,11 @@ class Fetch(object):
 
                 update_stamp(ud, self.d)
 
+            except IOError as e:
+                if e.errno in [os.errno.ESTALE]:
+                    logger.error("Stale Error Observed %s." % u)
+                    raise ChecksumError("Stale Error Detected")
+
             except BBFetchException as e:
                 if isinstance(e, ChecksumError):
                     logger.error("Checksum failure fetching %s" % u)
@@ -1775,6 +1790,7 @@ from . import svn
 from . import wget
 from . import ssh
 from . import sftp
+from . import s3
 from . import perforce
 from . import bzr
 from . import hg
@@ -1792,6 +1808,7 @@ methods.append(gitannex.GitANNEX())
 methods.append(cvs.Cvs())
 methods.append(ssh.SSH())
 methods.append(sftp.SFTP())
+methods.append(s3.S3())
 methods.append(perforce.Perforce())
 methods.append(bzr.Bzr())
 methods.append(hg.Hg())

bitbake/lib/bb/fetch2/bzr.py

@@ -27,7 +27,6 @@ import os
 import sys
 import logging
 import bb
-from bb import data
 from bb.fetch2 import FetchMethod
 from bb.fetch2 import FetchError
 from bb.fetch2 import runfetchcmd
@@ -43,14 +42,14 @@ class Bzr(FetchMethod):
         """
         # Create paths to bzr checkouts
         relpath = self._strip_leading_slashes(ud.path)
-        ud.pkgdir = os.path.join(data.expand('${BZRDIR}', d), ud.host, relpath)
+        ud.pkgdir = os.path.join(d.expand('${BZRDIR}'), ud.host, relpath)
 
-        ud.setup_revisons(d)
+        ud.setup_revisions(d)
 
         if not ud.revision:
             ud.revision = self.latest_revision(ud, d)
 
-        ud.localfile = data.expand('bzr_%s_%s_%s.tar.gz' % (ud.host, ud.path.replace('/', '.'), ud.revision), d)
+        ud.localfile = d.expand('bzr_%s_%s_%s.tar.gz' % (ud.host, ud.path.replace('/', '.'), ud.revision))
 
     def _buildbzrcommand(self, ud, d, command):
         """
@@ -58,7 +57,7 @@ class Bzr(FetchMethod):
         command is "fetch", "update", "revno"
         """
 
-        basecmd = data.expand('${FETCHCMD_bzr}', d)
+        basecmd = d.expand('${FETCHCMD_bzr}')
 
         proto =  ud.parm.get('protocol', 'http')
 

bitbake/lib/bb/fetch2/clearcase.py

@@ -65,7 +65,6 @@ import os
 import sys
 import shutil
 import bb
-from   bb import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import FetchError
 from   bb.fetch2 import runfetchcmd

bitbake/lib/bb/fetch2/cvs.py

@@ -63,7 +63,7 @@ class Cvs(FetchMethod):
         if 'fullpath' in ud.parm:
             fullpath = '_fullpath'
 
-        ud.localfile = bb.data.expand('%s_%s_%s_%s%s%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.tag, ud.date, norecurse, fullpath), d)
+        ud.localfile = d.expand('%s_%s_%s_%s%s%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.tag, ud.date, norecurse, fullpath))
 
     def need_update(self, ud, d):
         if (ud.date == "now"):

bitbake/lib/bb/fetch2/git.py

@@ -50,7 +50,7 @@ Supported SRC_URI options are:
    The default is "0", set nobranch=1 if needed.
 
 - usehead
-   For local git:// urls to use the current branch HEAD as the revsion for use with
+   For local git:// urls to use the current branch HEAD as the revision for use with
    AUTOREV. Implies nobranch.
 
 """
@@ -76,7 +76,6 @@ import re
 import bb
 import errno
 import bb.progress
-from   bb    import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger
@@ -186,7 +185,7 @@ class Git(FetchMethod):
 
         ud.write_tarballs = ((d.getVar("BB_GENERATE_MIRROR_TARBALLS") or "0") != "0") or ud.rebaseable
 
-        ud.setup_revisons(d)
+        ud.setup_revisions(d)
 
         for name in ud.names:
             # Ensure anything that doesn't look like a sha256 checksum/revision is translated into one

bitbake/lib/bb/fetch2/gitannex.py

@@ -22,7 +22,6 @@ BitBake 'Fetch' git annex implementation
 
 import os
 import bb
-from   bb import data
 from   bb.fetch2.git import Git
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger

bitbake/lib/bb/fetch2/gitsm.py

@@ -31,7 +31,6 @@ NOTE: Switching a SRC_URI from "git://" to "gitsm://" requires a clean of your r
 
 import os
 import bb
-from   bb    import data
 from   bb.fetch2.git import Git
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger

bitbake/lib/bb/fetch2/hg.py

@@ -29,7 +29,6 @@ import sys
 import logging
 import bb
 import errno
-from bb import data
 from bb.fetch2 import FetchMethod
 from bb.fetch2 import FetchError
 from bb.fetch2 import MissingParameterError
@@ -67,7 +66,7 @@ class Hg(FetchMethod):
         else:
             ud.proto = "hg"
 
-        ud.setup_revisons(d)
+        ud.setup_revisions(d)
 
         if 'rev' in ud.parm:
             ud.revision = ud.parm['rev']

bitbake/lib/bb/fetch2/local.py

@@ -29,7 +29,6 @@ import os
 import urllib.request, urllib.parse, urllib.error
 import bb
 import bb.utils
-from   bb import data
 from   bb.fetch2 import FetchMethod, FetchError
 from   bb.fetch2 import logger
 

bitbake/lib/bb/fetch2/npm.py

@@ -25,7 +25,6 @@ import json
 import subprocess
 import signal
 import bb
-from   bb import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import FetchError
 from   bb.fetch2 import ChecksumError
@@ -80,6 +79,7 @@ class Npm(FetchMethod):
         if not ud.version:
             raise ParameterError("NPM fetcher requires a version parameter", ud.url)
         ud.bbnpmmanifest = "%s-%s.deps.json" % (ud.pkgname, ud.version)
+        ud.bbnpmmanifest = ud.bbnpmmanifest.replace('/', '-')
         ud.registry = "http://%s" % (ud.url.replace('npm://', '', 1).split(';'))[0]
         prefixdir = "npm/%s" % ud.pkgname
         ud.pkgdatadir = d.expand("${DL_DIR}/%s" % prefixdir)
@@ -92,6 +92,7 @@ class Npm(FetchMethod):
 
         ud.write_tarballs = ((d.getVar("BB_GENERATE_MIRROR_TARBALLS") or "0") != "0")
         ud.mirrortarball = 'npm_%s-%s.tar.xz' % (ud.pkgname, ud.version)
+        ud.mirrortarball = ud.mirrortarball.replace('/', '-')
         ud.fullmirror = os.path.join(d.getVar("DL_DIR"), ud.mirrortarball)
 
     def need_update(self, ud, d):
@@ -133,8 +134,7 @@ class Npm(FetchMethod):
 
     def unpack(self, ud, destdir, d):
         dldir = d.getVar("DL_DIR")
-        depdumpfile = "%s-%s.deps.json" % (ud.pkgname, ud.version)
-        with open("%s/npm/%s" % (dldir, depdumpfile)) as datafile:
+        with open("%s/npm/%s" % (dldir, ud.bbnpmmanifest)) as datafile:
             workobj = json.load(datafile)
         dldir = "%s/%s" % (os.path.dirname(ud.localpath), ud.pkgname)
 
@@ -182,7 +182,12 @@ class Npm(FetchMethod):
             if pkg_os:
                 if not isinstance(pkg_os, list):
                     pkg_os = [pkg_os]
-                if 'linux' not in pkg_os or '!linux' in pkg_os:
+                blacklist = False
+                for item in pkg_os:
+                    if item.startswith('!'):
+                        blacklist = True
+                        break
+                if (not blacklist and 'linux' not in pkg_os) or '!linux' in pkg_os:
                     logger.debug(2, "Skipping %s since it's incompatible with Linux" % pkg)
                     return
         #logger.debug(2, "Output URL is %s - %s - %s" % (ud.basepath, ud.basename, ud.localfile))
@@ -195,6 +200,7 @@ class Npm(FetchMethod):
 
         dependencies = pdata.get('dependencies', {})
         optionalDependencies = pdata.get('optionalDependencies', {})
+        dependencies.update(optionalDependencies)
         depsfound = {}
         optdepsfound = {}
         data[pkg]['deps'] = {}

bitbake/lib/bb/fetch2/osc.py

@@ -10,7 +10,6 @@ import  os
 import  sys
 import logging
 import  bb
-from    bb       import data
 from    bb.fetch2 import FetchMethod
 from    bb.fetch2 import FetchError
 from    bb.fetch2 import MissingParameterError
@@ -47,7 +46,7 @@ class Osc(FetchMethod):
             else:
                 ud.revision = ""
 
-        ud.localfile = data.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.path.replace('/', '.'), ud.revision), d)
+        ud.localfile = d.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.path.replace('/', '.'), ud.revision))
 
     def _buildosccommand(self, ud, d, command):
         """
@@ -55,7 +54,7 @@ class Osc(FetchMethod):
         command is "fetch", "update", "info"
         """
 
-        basecmd = data.expand('${FETCHCMD_osc}', d)
+        basecmd = d.expand('${FETCHCMD_osc}')
 
         proto = ud.parm.get('protocol', 'ocs')
 

bitbake/lib/bb/fetch2/perforce.py

@@ -26,7 +26,6 @@ BitBake 'Fetch' implementation for perforce
 import os
 import logging
 import bb
-from   bb import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import FetchError
 from   bb.fetch2 import logger
@@ -87,9 +86,9 @@ class Perforce(FetchMethod):
         cleanedhost = ud.host.replace(':', '.')
         ud.pkgdir = os.path.join(ud.dldir, cleanedhost, cleanedpath)
 
-        ud.setup_revisons(d)
+        ud.setup_revisions(d)
 
-        ud.localfile = data.expand('%s_%s_%s.tar.gz' % (cleanedhost, cleanedpath, ud.revision), d)
+        ud.localfile = d.expand('%s_%s_%s.tar.gz' % (cleanedhost, cleanedpath, ud.revision))
 
     def _buildp4command(self, ud, d, command, depot_filename=None):
         """

bitbake/lib/bb/fetch2/repo.py

@@ -25,7 +25,6 @@ BitBake "Fetch" repo (git) implementation
 
 import os
 import bb
-from   bb    import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import runfetchcmd
 

bitbake/lib/bb/fetch2/s3.py

@@ -0,0 +1,96 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementation for Amazon AWS S3.
+
+Class for fetching files from Amazon S3 using the AWS Command Line Interface.
+The aws tool must be correctly installed and configured prior to use.
+
+"""
+
+# Copyright (C) 2017, Andre McCurdy <armccurdy@gmail.com>
+#
+# Based in part on bb.fetch2.wget:
+#    Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import bb
+import urllib.request, urllib.parse, urllib.error
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import FetchError
+from bb.fetch2 import runfetchcmd
+
+class S3(FetchMethod):
+    """Class to fetch urls via 'aws s3'"""
+
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with s3.
+        """
+        return ud.type in ['s3']
+
+    def recommends_checksum(self, urldata):
+        return True
+
+    def urldata_init(self, ud, d):
+        if 'downloadfilename' in ud.parm:
+            ud.basename = ud.parm['downloadfilename']
+        else:
+            ud.basename = os.path.basename(ud.path)
+
+        ud.localfile = d.expand(urllib.parse.unquote(ud.basename))
+
+    def download(self, ud, d):
+        """
+        Fetch urls
+        Assumes localpath was called first
+        """
+
+        cmd = 'aws s3 cp s3://%s%s %s' % (ud.host, ud.path, ud.localpath)
+        bb.fetch2.check_network_access(d, cmd, ud.url)
+        runfetchcmd(cmd, d)
+
+        # Additional sanity checks copied from the wget class (although there
+        # are no known issues which mean these are required, treat the aws cli
+        # tool with a little healthy suspicion).
+
+        if not os.path.exists(ud.localpath):
+            raise FetchError("The aws cp command returned success for s3://%s%s but %s doesn't exist?!" % (ud.host, ud.path, ud.localpath))
+
+        if os.path.getsize(ud.localpath) == 0:
+            os.remove(ud.localpath)
+            raise FetchError("The aws cp command for s3://%s%s resulted in a zero size file?! Deleting and failing since this isn't right." % (ud.host, ud.path))
+
+        return True
+
+    def checkstatus(self, fetch, ud, d):
+        """
+        Check the status of a URL
+        """
+
+        cmd = 'aws s3 ls s3://%s%s' % (ud.host, ud.path)
+        bb.fetch2.check_network_access(d, cmd, ud.url)
+        output = runfetchcmd(cmd, d)
+
+        # "aws s3 ls s3://mybucket/foo" will exit with success even if the file
+        # is not found, so check output of the command to confirm success.
+
+        if not output:
+            raise FetchError("The aws ls command for s3://%s%s gave empty output" % (ud.host, ud.path))
+
+        return True

bitbake/lib/bb/fetch2/sftp.py

@@ -62,12 +62,10 @@ SRC_URI = "sftp://user@host.example.com/dir/path.file.txt"
 import os
 import bb
 import urllib.request, urllib.parse, urllib.error
-from bb import data
 from bb.fetch2 import URI
 from bb.fetch2 import FetchMethod
 from bb.fetch2 import runfetchcmd
 
-
 class SFTP(FetchMethod):
     """Class to fetch urls via 'sftp'"""
 

bitbake/lib/bb/fetch2/ssh.py

@@ -43,7 +43,6 @@ IETF secsh internet draft:
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import re, os
-from   bb import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import FetchError
 from   bb.fetch2 import logger

bitbake/lib/bb/fetch2/svn.py

@@ -28,7 +28,6 @@ import sys
 import logging
 import bb
 import re
-from   bb import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import FetchError
 from   bb.fetch2 import MissingParameterError
@@ -61,15 +60,15 @@ class Svn(FetchMethod):
 
         # Create paths to svn checkouts
         relpath = self._strip_leading_slashes(ud.path)
-        ud.pkgdir = os.path.join(data.expand('${SVNDIR}', d), ud.host, relpath)
+        ud.pkgdir = os.path.join(d.expand('${SVNDIR}'), ud.host, relpath)
         ud.moddir = os.path.join(ud.pkgdir, ud.module)
 
-        ud.setup_revisons(d)
+        ud.setup_revisions(d)
 
         if 'rev' in ud.parm:
             ud.revision = ud.parm['rev']
 
-        ud.localfile = data.expand('%s_%s_%s_%s_.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.path.replace('/', '.'), ud.revision), d)
+        ud.localfile = d.expand('%s_%s_%s_%s_.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.path.replace('/', '.'), ud.revision))
 
     def _buildsvncommand(self, ud, d, command):
         """
@@ -79,9 +78,9 @@ class Svn(FetchMethod):
 
         proto = ud.parm.get('protocol', 'svn')
 
-        svn_rsh = None
-        if proto == "svn+ssh" and "rsh" in ud.parm:
-            svn_rsh = ud.parm["rsh"]
+        svn_ssh = None
+        if proto == "svn+ssh" and "ssh" in ud.parm:
+            svn_ssh = ud.parm["ssh"]
 
         svnroot = ud.host + ud.path
 
@@ -113,8 +112,8 @@ class Svn(FetchMethod):
             else:
                 raise FetchError("Invalid svn command %s" % command, ud.url)
 
-        if svn_rsh:
-            svncmd = "svn_RSH=\"%s\" %s" % (svn_rsh, svncmd)
+        if svn_ssh:
+            svncmd = "SVN_SSH=\"%s\" %s" % (svn_ssh, svncmd)
 
         return svncmd
 

bitbake/lib/bb/fetch2/wget.py

@@ -33,7 +33,6 @@ import logging
 import bb
 import bb.progress
 import urllib.request, urllib.parse, urllib.error
-from   bb import data
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import FetchError
 from   bb.fetch2 import logger
@@ -84,9 +83,9 @@ class Wget(FetchMethod):
         else:
             ud.basename = os.path.basename(ud.path)
 
-        ud.localfile = data.expand(urllib.parse.unquote(ud.basename), d)
+        ud.localfile = d.expand(urllib.parse.unquote(ud.basename))
         if not ud.localfile:
-            ud.localfile = data.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", "."), d)
+            ud.localfile = d.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", "."))
 
         self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate"
 
@@ -547,7 +546,7 @@ class Wget(FetchMethod):
 
         # src.rpm extension was added only for rpm package. Can be removed if the rpm
         # packaged will always be considered as having to be manually upgraded
-        psuffix_regex = "(tar\.gz|tgz|tar\.bz2|zip|xz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
+        psuffix_regex = "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
 
         # match name, version and archive type of a package
         package_regex_comp = re.compile("(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"

bitbake/lib/bb/parse/ast.py

@@ -130,7 +130,6 @@ class DataNode(AstNode):
                 val = groupd["value"]
         elif "colon" in groupd and groupd["colon"] != None:
             e = data.createCopy()
-            bb.data.update_data(e)
             op = "immediate"
             val = e.expand(groupd["value"], key + "[:=]")
         elif "append" in groupd and groupd["append"] != None:
@@ -350,12 +349,10 @@ def finalize(fn, d, variant = None):
     bb.event.fire(bb.event.RecipePreFinalise(fn), d)
 
     bb.data.expandKeys(d)
-    bb.data.update_data(d)
     code = []
     for funcname in d.getVar("__BBANONFUNCS", False) or []:
         code.append("%s(d)" % funcname)
     bb.utils.better_exec("\n".join(code), {"d": d})
-    bb.data.update_data(d)
 
     tasklist = d.getVar('__BBTASKS', False) or []
     bb.event.fire(bb.event.RecipeTaskPreProcess(fn, list(tasklist)), d)

bitbake/lib/bb/process.py

@@ -162,9 +162,9 @@ def run(cmd, input=None, log=None, extrafiles=None, **options):
         stdout, stderr = _logged_communicate(pipe, log, input, extrafiles)
     else:
         stdout, stderr = pipe.communicate(input)
-        if stdout:
+        if not stdout is None:
             stdout = stdout.decode("utf-8")
-        if stderr:
+        if not stderr is None:
             stderr = stderr.decode("utf-8")
 
     if pipe.returncode != 0:

bitbake/lib/bb/providers.py

@@ -48,7 +48,6 @@ def findProviders(cfgData, dataCache, pkg_pn = None):
 
     # Need to ensure data store is expanded
     localdata = data.createCopy(cfgData)
-    bb.data.update_data(localdata)
     bb.data.expandKeys(localdata)
 
     preferred_versions = {}

bitbake/lib/bb/runqueue.py

@@ -2010,6 +2010,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
         for tid in self.rqdata.runq_setscene_tids:
             #bb.warn("Added endpoint 2 %s" % (tid))
             for dep in self.rqdata.runtaskentries[tid].depends:
+                    if tid in sq_revdeps[dep]:
+                        sq_revdeps[dep].remove(tid)
                     if dep not in endpoints:
                         endpoints[dep] = set()
                     #bb.warn("  Added endpoint 3 %s" % (dep))
@@ -2029,12 +2031,13 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                 if point in self.rqdata.runq_setscene_tids:
                     sq_revdeps_new[point] = tasks
                     tasks = set()
+                    continue
                 for dep in self.rqdata.runtaskentries[point].depends:
                     if point in sq_revdeps[dep]:
                         sq_revdeps[dep].remove(point)
                     if tasks:
                         sq_revdeps_new[dep] |= tasks
-                    if (len(sq_revdeps[dep]) == 0 or len(sq_revdeps_new[dep]) != 0) and dep not in self.rqdata.runq_setscene_tids:
+                    if len(sq_revdeps[dep]) == 0 and dep not in self.rqdata.runq_setscene_tids:
                         newendpoints[dep] = task
             if len(newendpoints) != 0:
                 process_endpoints(newendpoints)

bitbake/lib/bb/tests/data.py

@@ -258,13 +258,11 @@ class TestConcatOverride(unittest.TestCase):
     def test_prepend(self):
         self.d.setVar("TEST", "${VAL}")
         self.d.setVar("TEST_prepend", "${FOO}:")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "foo:val")
 
     def test_append(self):
         self.d.setVar("TEST", "${VAL}")
         self.d.setVar("TEST_append", ":${BAR}")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "val:bar")
 
     def test_multiple_append(self):
@@ -272,47 +270,40 @@ class TestConcatOverride(unittest.TestCase):
         self.d.setVar("TEST_prepend", "${FOO}:")
         self.d.setVar("TEST_append", ":val2")
         self.d.setVar("TEST_append", ":${BAR}")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "foo:val:val2:bar")
 
     def test_append_unset(self):
         self.d.setVar("TEST_prepend", "${FOO}:")
         self.d.setVar("TEST_append", ":val2")
         self.d.setVar("TEST_append", ":${BAR}")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "foo::val2:bar")
 
     def test_remove(self):
         self.d.setVar("TEST", "${VAL} ${BAR}")
         self.d.setVar("TEST_remove", "val")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "bar")
 
     def test_doubleref_remove(self):
         self.d.setVar("TEST", "${VAL} ${BAR}")
         self.d.setVar("TEST_remove", "val")
         self.d.setVar("TEST_TEST", "${TEST} ${TEST}")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST_TEST"), "bar bar")
 
     def test_empty_remove(self):
         self.d.setVar("TEST", "")
         self.d.setVar("TEST_remove", "val")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "")
 
     def test_remove_expansion(self):
         self.d.setVar("BAR", "Z")
         self.d.setVar("TEST", "${BAR}/X Y")
         self.d.setVar("TEST_remove", "${BAR}/X")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "Y")
 
     def test_remove_expansion_items(self):
         self.d.setVar("TEST", "A B C D")
         self.d.setVar("BAR", "B D")
         self.d.setVar("TEST_remove", "${BAR}")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "A C")
 
 class TestOverrides(unittest.TestCase):
@@ -322,17 +313,15 @@ class TestOverrides(unittest.TestCase):
         self.d.setVar("TEST", "testvalue")
 
     def test_no_override(self):
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "testvalue")
 
     def test_one_override(self):
         self.d.setVar("TEST_bar", "testvalue2")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "testvalue2")
 
     def test_one_override_unset(self):
         self.d.setVar("TEST2_bar", "testvalue2")
-        bb.data.update_data(self.d)
+
         self.assertEqual(self.d.getVar("TEST2"), "testvalue2")
         self.assertCountEqual(list(self.d.keys()), ['TEST', 'TEST2', 'OVERRIDES', 'TEST2_bar'])
 
@@ -340,18 +329,15 @@ class TestOverrides(unittest.TestCase):
         self.d.setVar("TEST_bar", "testvalue2")
         self.d.setVar("TEST_local", "testvalue3")
         self.d.setVar("TEST_foo", "testvalue4")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "testvalue3")
         self.assertCountEqual(list(self.d.keys()), ['TEST', 'TEST_foo', 'OVERRIDES', 'TEST_bar', 'TEST_local'])
 
     def test_multiple_combined_overrides(self):
         self.d.setVar("TEST_local_foo_bar", "testvalue3")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST"), "testvalue3")
 
     def test_multiple_overrides_unset(self):
         self.d.setVar("TEST2_local_foo_bar", "testvalue3")
-        bb.data.update_data(self.d)
         self.assertEqual(self.d.getVar("TEST2"), "testvalue3")
 
     def test_keyexpansion_override(self):
@@ -359,14 +345,12 @@ class TestOverrides(unittest.TestCase):
         self.d.setVar("TEST_bar", "testvalue2")
         self.d.setVar("TEST_${LOCAL}", "testvalue3")
         self.d.setVar("TEST_foo", "testvalue4")
-        bb.data.update_data(self.d)
         bb.data.expandKeys(self.d)
         self.assertEqual(self.d.getVar("TEST"), "testvalue3")
 
     def test_rename_override(self):
         self.d.setVar("ALTERNATIVE_ncurses-tools_class-target", "a")
         self.d.setVar("OVERRIDES", "class-target")
-        bb.data.update_data(self.d)
         self.d.renameVar("ALTERNATIVE_ncurses-tools", "ALTERNATIVE_lib32-ncurses-tools")
         self.assertEqual(self.d.getVar("ALTERNATIVE_lib32-ncurses-tools"), "a")
 

bitbake/lib/bb/tinfoil.py

@@ -211,6 +211,19 @@ class Tinfoil:
 
         self.ui_module = ui_module
 
+        # Ensure the path to bitbake's bin directory is in PATH so that things like
+        # bitbake-worker can be run (usually this is the case, but it doesn't have to be)
+        path = os.getenv('PATH').split(':')
+        bitbakebinpath = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..', 'bin'))
+        for entry in path:
+            if entry.endswith(os.sep):
+                entry = entry[:-1]
+            if os.path.abspath(entry) == bitbakebinpath:
+                break
+        else:
+            path.insert(0, bitbakebinpath)
+            os.environ['PATH'] = ':'.join(path)
+
         if self.server_connection:
             _server_connections.append(self.server_connection)
             if config_only:

bitbake/lib/bb/ui/taskexp.py

@@ -78,7 +78,7 @@ class PackageReverseDepView(Gtk.TreeView):
 class DepExplorer(Gtk.Window):
     def __init__(self):
         Gtk.Window.__init__(self)
-        self.set_title("Dependency Explorer")
+        self.set_title("Task Dependency Explorer")
         self.set_default_size(500, 500)
         self.connect("delete-event", Gtk.main_quit)
 
@@ -106,30 +106,21 @@ class DepExplorer(Gtk.Window):
 
         box = Gtk.VBox(homogeneous=True, spacing=4)
 
-        # Runtime Depends
+        # Task Depends
         scrolled = Gtk.ScrolledWindow()
         scrolled.set_policy(Gtk.PolicyType.AUTOMATIC, Gtk.PolicyType.AUTOMATIC)
         scrolled.set_shadow_type(Gtk.ShadowType.IN)
-        self.rdep_treeview = PackageDepView(self.depends_model, TYPE_RDEP, "Runtime Depends")
-        self.rdep_treeview.connect("row-activated", self.on_package_activated, COL_DEP_PACKAGE)
-        scrolled.add(self.rdep_treeview)
-        box.add(scrolled)
-
-        # Build Depends
-        scrolled = Gtk.ScrolledWindow()
-        scrolled.set_policy(Gtk.PolicyType.AUTOMATIC, Gtk.PolicyType.AUTOMATIC)
-        scrolled.set_shadow_type(Gtk.ShadowType.IN)
-        self.dep_treeview = PackageDepView(self.depends_model, TYPE_DEP, "Build Depends")
+        self.dep_treeview = PackageDepView(self.depends_model, TYPE_DEP, "Dependencies")
         self.dep_treeview.connect("row-activated", self.on_package_activated, COL_DEP_PACKAGE)
         scrolled.add(self.dep_treeview)
         box.add(scrolled)
         pane.add2(box)
 
-        # Reverse Depends
+        # Reverse Task Depends
         scrolled = Gtk.ScrolledWindow()
         scrolled.set_policy(Gtk.PolicyType.AUTOMATIC, Gtk.PolicyType.AUTOMATIC)
         scrolled.set_shadow_type(Gtk.ShadowType.IN)
-        self.revdep_treeview = PackageReverseDepView(self.depends_model, "Reverse Depends")
+        self.revdep_treeview = PackageReverseDepView(self.depends_model, "Dependent Tasks")
         self.revdep_treeview.connect("row-activated", self.on_package_activated, COL_DEP_PARENT)
         scrolled.add(self.revdep_treeview)
         box.add(scrolled)
@@ -160,22 +151,15 @@ class DepExplorer(Gtk.Window):
             current_package = None
         else:
             current_package = model.get_value(it, COL_PKG_NAME)
-        self.rdep_treeview.set_current_package(current_package)
         self.dep_treeview.set_current_package(current_package)
         self.revdep_treeview.set_current_package(current_package)
 
 
     def parse(self, depgraph):
-        for package in depgraph["pn"]:
-            self.pkg_model.insert(0, (package,))
-
-        for package in depgraph["depends"]:
-            for depend in depgraph["depends"][package]:
-                self.depends_model.insert (0, (TYPE_DEP, package, depend))
-
-        for package in depgraph["rdepends-pn"]:
-            for rdepend in depgraph["rdepends-pn"][package]:
-                self.depends_model.insert (0, (TYPE_RDEP, package, rdepend))
+        for task in depgraph["tdepends"]:
+            self.pkg_model.insert(0, (task,))
+            for depend in depgraph["tdepends"][task]:
+                self.depends_model.insert (0, (TYPE_DEP, task, depend))
 
 
 class gtkthread(threading.Thread):

bitbake/lib/bb/utils.py

@@ -979,6 +979,30 @@ def contains_any(variable, checkvalues, truevalue, falsevalue, d):
         return truevalue
     return falsevalue
 
+def filter(variable, checkvalues, d):
+    """Return all words in the variable that are present in the checkvalues.
+
+    Arguments:
+
+    variable -- the variable name. This will be fetched and expanded (using
+    d.getVar(variable)) and then split into a set().
+
+    checkvalues -- if this is a string it is split on whitespace into a set(),
+    otherwise coerced directly into a set().
+
+    d -- the data store.
+    """
+
+    val = d.getVar(variable)
+    if not val:
+        return ''
+    val = set(val.split())
+    if isinstance(checkvalues, str):
+        checkvalues = set(checkvalues.split())
+    else:
+        checkvalues = set(checkvalues)
+    return ' '.join(sorted(checkvalues & val))
+
 def cpu_count():
     return multiprocessing.cpu_count()
 

bitbake/lib/toaster/orm/models.py

@@ -902,7 +902,7 @@ class Target_Image_File(models.Model):
         'ext4.gz', 'ext3', 'ext3.gz', 'hdddirect', 'hddimg', 'iso', 'jffs2',
         'jffs2.sum', 'multiubi', 'qcow2', 'squashfs', 'squashfs-lzo',
         'squashfs-xz', 'tar', 'tar.bz2', 'tar.gz', 'tar.lz4', 'tar.xz', 'ubi',
-        'ubifs', 'vdi', 'vmdk', 'wic', 'wic.bz2', 'wic.gz', 'wic.lzma'
+        'ubifs', 'vdi', 'vmdk', 'wic', 'wic.bmap', 'wic.bz2', 'wic.gz', 'wic.lzma'
     }
 
     target = models.ForeignKey(Target)

bitbake/lib/toaster/toastermain/settings.py

@@ -38,8 +38,7 @@ ADMINS = (
 
 MANAGERS = ADMINS
 
-TOASTER_SQLITE_DEFAULT_DIR = os.path.join(os.environ.get('TOASTER_DIR', ''),
-                                          'build')
+TOASTER_SQLITE_DEFAULT_DIR = os.environ.get('TOASTER_DIR')
 
 DATABASES = {
     'default': {

documentation/bsp-guide/bsp-guide.xml

@@ -130,12 +130,34 @@
         Permission is granted to copy, distribute and/or modify this document under
         the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-nc-sa/2.0/uk/">Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by Creative Commons.
       </para>
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>
-          from the Yocto Project website.
-      </note>
+            <note><title>Manual Notes</title>
+                <itemizedlist>
+                    <listitem><para>
+                        For the latest version of the Yocto Project Board
+                        Support Package (BSP) Developer's Guide associated with
+                        this Yocto Project release (version
+                        &YOCTO_DOC_VERSION;),
+                        see the Yocto Project Board Support Package (BSP)
+                        Developer's Guide from the
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                        </para></listitem>
+                    <listitem><para>
+                        This version of the manual is version
+                        &YOCTO_DOC_VERSION;.
+                        For later releases of the Yocto Project (if they exist),
+                        go to the
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                        and use the drop-down "Active Releases" button
+                        and choose the Yocto Project version for which you want
+                        the manual.
+                        </para></listitem>
+                    <listitem><para>
+                        For an in-development version of the Yocto Project
+                        Board Support Package (BSP) Developer's Guide, see
+                        <ulink url='&YOCTO_DOCS_URL;/latest/bsp-guide/bsp-guide.html'></ulink>.
+                        </para></listitem>
+                </itemizedlist>
+            </note>
     </legalnotice>
 
     </bookinfo>

documentation/dev-manual/dev-manual-common-tasks.xml

@@ -10884,10 +10884,11 @@
 
             <para>
                 Before you employ <filename>DL_DIR</filename> or the
-                archiver class, you need to decide how you choose to
-                provide source.
-                The source archiver class can generate tarballs and SRPMs
-                and can create them with various levels of compliance in mind.
+                <filename>archiver</filename> class, you need to decide how
+                you choose to provide source.
+                The source <filename>archiver</filename> class can generate
+                tarballs and SRPMs and can create them with various levels of
+                compliance in mind.
             </para>
 
             <para>
@@ -10982,9 +10983,9 @@
                     <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_DIRS'><filename>COPY_LIC_DIRS</filename></ulink>
                     and
                     <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_MANIFEST'><filename>COPY_LIC_MANIFEST</filename></ulink>
-                    add a copy of the license when the image is built but do not
-                    offer a path for adding licenses for newly installed packages
-                    to an image.
+                    add a copy of the license when the image is built but do
+                    not offer a path for adding licenses for newly installed
+                    packages to an image.
                     <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_CREATE_PACKAGE'><filename>LICENSE_CREATE_PACKAGE</filename></ulink>
                     adds a separate package and an upgrade path for adding
                     licenses to an image.</para>
@@ -10992,7 +10993,8 @@
             </para>
 
             <para>
-                As the source archiver has already archived the original
+                As the source <filename>archiver</filename> class has already
+                archived the original
                 unmodified source that contains the license files,
                 you would have already met the requirements for inclusion
                 of the license information with source as defined by the GPL
@@ -11004,7 +11006,7 @@
             <title>Providing Compilation Scripts and Source Code Modifications</title>
 
             <para>
-                At this point, we have addressed all we need to address
+                At this point, we have addressed all we need to
                 prior to generating the image.
                 The next two requirements are addressed during the final
                 packaging of the release.
@@ -11023,7 +11025,7 @@
                 and a distro layer, and those those layers are used to patch,
                 compile, package, or modify (in any way) any open source
                 software included in your released images, you
-                might be required to to release those layers under section 3 of
+                might be required to release those layers under section 3 of
                 GPLv2 or section 1 of GPLv3.
                 One way of doing that is with a clean
                 checkout of the version of the Yocto Project and layers used

documentation/dev-manual/dev-manual.xml

@@ -111,12 +111,32 @@
           Creative Commons.
       </para>
 
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_DEV_URL;'>Yocto Project Development Manual</ulink>
-          from the Yocto Project website.
-      </note>
+            <note><title>Manual Notes</title>
+                <itemizedlist>
+                    <listitem><para>
+                        For the latest version of the Yocto Project Development
+                        Manual associated with this Yocto Project release
+                        (version &YOCTO_DOC_VERSION;),
+                        see the Yocto Project Development Manual from the
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                        </para></listitem>
+                    <listitem><para>
+                        This version of the manual is version
+                        &YOCTO_DOC_VERSION;.
+                        For later releases of the Yocto Project (if they exist),
+                        go to the
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                        and use the drop-down "Active Releases" button
+                        and choose the Yocto Project version for which you want
+                        the manual.
+                        </para></listitem>
+                    <listitem><para>
+                        For an in-development version of the Yocto Project
+                        Development Manual, see
+                        <ulink url='&YOCTO_DOCS_URL;/latest/dev-manual/dev-manual.html'></ulink>.
+                        </para></listitem>
+                </itemizedlist>
+            </note>
     </legalnotice>
 
     </bookinfo>

documentation/kernel-dev/kernel-dev.xml

@@ -93,12 +93,33 @@
         Permission is granted to copy, distribute and/or modify this document under
         the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by Creative Commons.
       </para>
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;'>Yocto Project Linux Kernel Development Manual</ulink>
-          from the Yocto Project website.
-      </note>
+           <note><title>Manual Notes</title>
+               <itemizedlist>
+                   <listitem><para>
+                       For the latest version of the Yocto Project Linux
+                       Kernel Development Manual associated with this Yocto
+                       Project release (version &YOCTO_DOC_VERSION;),
+                       see the Yocto Project Linux Kernel Development
+                       Manual from the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                       </para></listitem>
+                   <listitem><para>
+                       This version of the manual is version
+                       &YOCTO_DOC_VERSION;.
+                       For later releases of the Yocto Project (if they exist),
+                       go to the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                       and use the drop-down "Active Releases" button
+                       and choose the Yocto Project version for which you want
+                       the manual.
+                       </para></listitem>
+                   <listitem><para>
+                        For an in-development version of the Yocto Project
+                        Linux Kernel Development Manual, see
+                        <ulink url='&YOCTO_DOCS_URL;/latest/kernel-dev/kernel-dev.html'></ulink>.
+                        </para></listitem>
+                </itemizedlist>
+            </note>
     </legalnotice>
 
     </bookinfo>

documentation/mega-manual/mega-manual.xml

@@ -77,12 +77,32 @@
         Permission is granted to copy, distribute and/or modify this document under
         the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by Creative Commons.
       </para>
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_MM_URL;'>Yocto Project Mega-Manual</ulink>
-          from the Yocto Project website.
-      </note>
+           <note><title>Manual Notes</title>
+               <itemizedlist>
+                   <listitem><para>
+                       For the latest version of the Yocto Project
+                       Mega-Manual associated with this Yocto Project release
+                       (version &YOCTO_DOC_VERSION;),
+                       see the Yocto Project Mega-Manual from the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                       </para></listitem>
+                   <listitem><para>
+                       This version of the manual is version
+                       &YOCTO_DOC_VERSION;.
+                       For later releases of the Yocto Project (if they exist),
+                       go to the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                       and use the drop-down "Active Releases" button
+                       and choose the Yocto Project version for which you want
+                       the manual.
+                       </para></listitem>
+                   <listitem><para>
+                       For an in-development version of the Yocto Project
+                       Mega-Manual, see
+                       <ulink url='&YOCTO_DOCS_URL;/latest/mega-manual/mega-manual.html'></ulink>.
+                       </para></listitem>
+               </itemizedlist>
+           </note>
 
     </legalnotice>
 

documentation/profile-manual/profile-manual.xml

@@ -96,12 +96,33 @@
           Creative Commons.
       </para>
 
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_PROF_URL;'>Yocto Project Profiling and Tracing Manual</ulink>
-          from the Yocto Project website.
-      </note>
+            <note><title>Manual Notes</title>
+                <itemizedlist>
+                    <listitem><para>
+                        For the latest version of the Yocto Project Profiling
+                        and Tracing Manual associated with this Yocto Project
+                        release (version &YOCTO_DOC_VERSION;),
+                        see the Yocto Project Profiling and Tracing Manual
+                        from the
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                        </para></listitem>
+                    <listitem><para>
+                        This version of the manual is version
+                        &YOCTO_DOC_VERSION;.
+                        For later releases of the Yocto Project (if they exist),
+                        go to the
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                        and use the drop-down "Active Releases" button
+                        and choose the Yocto Project version for which you want
+                        the manual.
+                        </para></listitem>
+                    <listitem><para>
+                        For an in-development version of the Yocto Project
+                        Profiling and Tracing Manual, see
+                        <ulink url='&YOCTO_DOCS_URL;/latest/profile-manual/profile-manual.html'></ulink>.
+                        </para></listitem>
+                </itemizedlist>
+            </note>
     </legalnotice>
 
     </bookinfo>

documentation/ref-manual/ref-manual.xml

@@ -124,12 +124,32 @@
         Permission is granted to copy, distribute and/or modify this document under
         the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by Creative Commons.
       </para>
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_REF_URL;'>Yocto Project Reference Manual</ulink>
-          from the Yocto Project website.
-      </note>
+           <note><title>Manual Notes</title>
+               <itemizedlist>
+                   <listitem><para>
+                       For the latest version of the Yocto Project Reference
+                       Manual associated with this Yocto Project release
+                       (version &YOCTO_DOC_VERSION;),
+                       see the Yocto Project Reference Manual from the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                       </para></listitem>
+                   <listitem><para>
+                       This version of the manual is version
+                       &YOCTO_DOC_VERSION;.
+                       For later releases of the Yocto Project (if they exist),
+                       go to the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                       and use the drop-down "Active Releases" button
+                       and choose the Yocto Project version for which you want
+                       the manual.
+                       </para></listitem>
+                   <listitem><para>
+                       For an in-development version of the Yocto Project
+                       Reference Manual, see
+                        <ulink url='&YOCTO_DOCS_URL;/latest/ref-manual/ref-manual.html'></ulink>.
+                        </para></listitem>
+                </itemizedlist>
+            </note>
     </legalnotice>
 
     </bookinfo>

documentation/ref-manual/ref-tasks.xml

@@ -853,7 +853,10 @@
         <title><filename>do_compile_kernelmodules</filename></title>
 
         <para>
-            Compiles loadable modules for the Linux kernel.
+            Runs the step that builds the kernel modules (if needed).
+            Building a kernel consists of two steps: 1) the kernel
+            (<filename>vmlinux</filename>) is built, and 2) the modules
+            are built (i.e. <filename>make modules</filename>).
         </para>
     </section>
 
@@ -861,9 +864,21 @@
         <title><filename>do_diffconfig</filename></title>
 
         <para>
-            Compares the old and new config files after running the
-            <link linkend='ref-tasks-menuconfig'><filename>do_menuconfig</filename></link>
-            task for the kernel.
+            When invoked by the user, this task creates a file containing the
+            differences between the original config as produced by
+            <link linkend='ref-tasks-kernel_configme'><filename>do_kernel_configme</filename></link>
+            task and the changes made by the user with other methods
+            (i.e. using
+            (<link linkend='ref-tasks-kernel_menuconfig'><filename>do_kernel_menuconfig</filename></link>).
+            Once the file of differences is created, it can be used to create
+            a config fragment that only contains the differences.
+            You can invoke this task from the command line as follows:
+            <literallayout class='monospaced'>
+     $ bitbake linux-yocto -c diffconfig
+            </literallayout>
+            For more information, see the
+            "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#generating-configuration-files'>Generating Configuration Files</ulink>"
+            section in the Yocto Project Linux Kernel Development Manual.
         </para>
     </section>
 
@@ -871,7 +886,12 @@
         <title><filename>do_kernel_checkout</filename></title>
 
         <para>
-            Checks out source/meta branches for a linux-yocto style kernel.
+            Converts the newly unpacked kernel source into a form with which
+            the OpenEmbedded build system can work.
+            Because the kernel source can be fetched in several different ways,
+            the <filename>do_kernel_checkout</filename> task makes sure that
+            subsequent tasks are given a clean working tree copy of the kernel
+            with the correct branches checked out.
         </para>
     </section>
 
@@ -879,7 +899,21 @@
         <title><filename>do_kernel_configcheck</filename></title>
 
         <para>
-            Validates the kernel configuration for a linux-yocto style kernel.
+            Validates the configuration produced by the
+            <link linkend='ref-tasks-kernel_menuconfig'><filename>do_kernel_menuconfig</filename></link>
+            task.
+            The <filename>do_kernel_configcheck</filename> task produces
+            warnings when a requested configuration does not appear in the
+            final <filename>.config</filename> file or when you override a
+            policy configuration in a hardware configuration fragment.
+            You can run this task explicitly and view the output by using
+            the following command:
+            <literallayout class='monospaced'>
+     $ bitbake linux-yocto -c kernel_configcheck -f
+            </literallayout>
+            For more information, see the
+            "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#generating-configuration-files'>Generating Configuration Files</ulink>"
+            section in the Yocto Project Linux Kernel Development Manual.
         </para>
     </section>
 
@@ -887,17 +921,41 @@
         <title><filename>do_kernel_configme</filename></title>
 
         <para>
-            Assembles the kernel configuration for a linux-yocto style kernel.
+            After the kernel is patched by the
+            <link linkend='ref-tasks-patch'><filename>do_patch</filename></link>
+            task, the <filename>do_kernel_configme</filename> task assembles
+            and merges all the kernel config fragments into a merged
+            configuration that can then be passed to the kernel configuration
+            phase proper.
+            This is also the time during which user-specified defconfigs
+            are applied if present, and where configuration modes such as
+            <filename>--allnoconfig</filename> are applied.
         </para>
     </section>
 
-    <section id='ref-tasks-kernel_link_vmlinux'>
-        <title><filename>do_kernel_link_vmlinux</filename></title>
+    <section id='ref-tasks-kernel_menuconfig'>
+        <title><filename>do_kernel_menuconfig</filename></title>
 
         <para>
-            Creates a symbolic link in
-            <filename>arch/$arch/boot</filename> for vmlinux kernel
-            images.
+            Invoked by the user to manipulate the
+            <filename>.config</filename> file used to build a linux-yocto
+            recipe.
+            This task starts the Linux kernel configuration tool, which you
+            then use to modify the kernel configuration.
+            <note>
+                You can also invoke this tool from the command line as
+                follows:
+                <literallayout class='monospaced'>
+     $ bitbake linux-yocto -c menuconfig
+                </literallayout>
+            </note>
+            See the
+            "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#generating-configuration-files'>Generating Configuration Files</ulink>"
+            section in the Yocto Project Linux Kernel Development Manual
+            for more information on this configuration tool.
+            You can also reference the
+            "<ulink url='&YOCTO_DOCS_DEV_URL;#using-menuconfig'>Using <filename>menuconfig</filename></ulink>"
+            section in the Yocto Project Development Manual.
         </para>
     </section>
 
@@ -905,8 +963,16 @@
         <title><filename>do_kernel_metadata</filename></title>
 
         <para>
-            Collects kernel metadata for a
-            <filename>linux-yocto</filename> style kernel.
+            Collects all the features required for a given kernel build,
+            whether the features come from
+            <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+            or from Git repositories.
+            After collection, the <filename>do_kernel_metadata</filename> task
+            processes the features into a series of config fragments and
+            patches, which can then be applied by subsequent tasks such as
+            <link linkend='ref-tasks-patch'><filename>do_patch</filename></link>
+            and
+            <link linkend='ref-tasks-kernel_configme'><filename>do_kernel_configme</filename></link>.
         </para>
     </section>
 
@@ -925,7 +991,17 @@
         <title><filename>do_savedefconfig</filename></title>
 
         <para>
-            Creates a minimal Linux kernel configuration file.
+            When invoked by the user, creates a defconfig file that can be
+            used instead of the default defconfig.
+            The saved defconfig contains the differences between the default
+            defconfig and the changes made by the user using other methods
+            (i.e. the
+            <link linkend='ref-tasks-kernel_menuconfig'><filename>do_kernel_menuconfig</filename></link>
+            task.
+            You can invoke the task using the following command:
+            <literallayout class='monospaced'>
+     $ bitbake linux-yocto -c savedefconfig
+            </literallayout>
         </para>
     </section>
 
@@ -933,7 +1009,14 @@
         <title><filename>do_shared_workdir</filename></title>
 
         <para>
-            Creates the shared working directory for the kernel.
+            After the kernel has been compiled but before the kernel modules
+            have been compiled, this task copies files required for module
+            builds and which are generated from the  kernel build into the
+            shared work directory.
+            With these copies successfully copied, the
+            <link linkend='ref-tasks-compile_kernelmodules'><filename>do_compile_kernelmodules</filename></link>
+            task can successfully build the kernel modules in the next step
+            of the build.
         </para>
     </section>
 
@@ -941,9 +1024,12 @@
         <title><filename>do_sizecheck</filename></title>
 
         <para>
-            Checks the size of the kernel image against
-            <link linkend='var-KERNEL_IMAGE_MAXSIZE'><filename>KERNEL_IMAGE_MAXSIZE</filename></link>
-            when set.
+            After the kernel has been built, this task checks the size of the
+            stripped kernel image against
+            <link linkend='var-KERNEL_IMAGE_MAXSIZE'><filename>KERNEL_IMAGE_MAXSIZE</filename></link>.
+            If that variable was set and the size of the stripped kernel
+            exceeds that size, the kernel build produces a warning to that
+            effect.
         </para>
     </section>
 
@@ -951,15 +1037,13 @@
         <title><filename>do_strip</filename></title>
 
         <para>
-            Strips unneeded sections out of the Linux kernel image.
-        </para>
-    </section>
-
-    <section id='ref-tasks-uboot_mkimage'>
-        <title><filename>do_uboot_mkimage</filename></title>
-
-        <para>
-            Creates a uImage file from the kernel for the U-Boot bootloader.
+            If
+            <filename>KERNEL_IMAGE_STRIP_EXTRA_SECTIONS</filename> is defined,
+            this task strips the sections named in that variable from
+            <filename>vmlinux</filename>.
+            This stripping is typically used to remove nonessential sections
+            such as <filename>.comment</filename> sections from a
+            size-sensitive configuration.
         </para>
     </section>
 
@@ -967,10 +1051,14 @@
         <title><filename>do_validate_branches</filename></title>
 
         <para>
-            Ensures that the source, metadata (or both) branches are on the
-            locations specified by their
-            <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
-            values for a linux-yocto style kernel.
+            After the kernel is unpacked but before it is patched, this task
+            makes sure that the machine and metadata branches as specified
+            by the <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
+            variables actually exist on the specified branches.
+            If these branches do not exist and
+            <link linkend='var-AUTOREV'><filename>AUTOREV</filename></link>
+            is not being used, the <filename>do_validate_branches</filename>
+            task fails during the build.
         </para>
     </section>
 </section>

documentation/ref-manual/ref-variables.xml

@@ -8332,7 +8332,7 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
                     <filename>local.conf</filename> file or you can attach it to
                     a specific image recipe by using the recipe name override:
                     <literallayout class='monospaced'>
-     NO_RECOMMENDATIONS_pn-<replaceable>target_image</replaceable> = "<replaceable>package_name</replaceable>"
+     NO_RECOMMENDATIONS_pn-<replaceable>target_image</replaceable> = "1"
                     </literallayout>
                 </para>
 

documentation/sdk-manual/sdk-manual.xml

@@ -58,12 +58,33 @@
         Permission is granted to copy, distribute and/or modify this document under
         the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by Creative Commons.
       </para>
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Software Development Kit (SDK) Developer's Guide</ulink>
-          from the Yocto Project website.
-      </note>
+           <note><title>Manual Notes</title>
+               <itemizedlist>
+                   <listitem><para>
+                       For the latest version of the Yocto Project Software
+                       Development Kit (SDK) Developer's Guide associated with
+                       this Yocto Project release (version &YOCTO_DOC_VERSION;),
+                       see the Yocto Project Software Development Kit (SDK)
+                       Developer's Guide from the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                       </para></listitem>
+                   <listitem><para>
+                       This version of the manual is version
+                       &YOCTO_DOC_VERSION;.
+                       For later releases of the Yocto Project (if they exist),
+                       go to the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                       and use the drop-down "Active Releases" button
+                       and choose the Yocto Project version for which you want
+                       the manual.
+                       </para></listitem>
+                   <listitem><para>
+                       For an in-development version of the Yocto Project
+                       Software Development Kit (SDK) Developer's Guide, see
+                       <ulink url='&YOCTO_DOCS_URL;/latest/sdk-manual/sdk-manual.html'></ulink>.
+                       </para></listitem>
+               </itemizedlist>
+           </note>
 
     </legalnotice>
 

documentation/toaster-manual/toaster-manual.xml

@@ -68,12 +68,32 @@
         Permission is granted to copy, distribute and/or modify this document under
         the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by Creative Commons.
       </para>
-      <note>
-          For the latest version of this manual associated with this
-          Yocto Project release, see the
-          <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>
-          from the Yocto Project website.
-      </note>
+           <note><title>Manual Notes</title>
+               <itemizedlist>
+                   <listitem><para>
+                       For the latest version of the Yocto Project Toaster
+                       User Manual associated with this Yocto Project release
+                       (version &YOCTO_DOC_VERSION;),
+                       see the Yocto Project Toaster User Manual from the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
+                       </para></listitem>
+                   <listitem><para>
+                       This version of the manual is version
+                       &YOCTO_DOC_VERSION;.
+                       For later releases of the Yocto Project (if they exist),
+                       go to the
+                       <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                       and use the drop-down "Active Releases" button
+                       and choose the Yocto Project version for which you want
+                       the manual.
+                       </para></listitem>
+                   <listitem><para>
+                       For an in-development version of the Yocto Project
+                       Toaster User Manual, see
+                       <ulink url='&YOCTO_DOCS_URL;/latest/toaster-manual/toaster-manual.html'></ulink>.
+                       </para></listitem>
+               </itemizedlist>
+           </note>
 
     </legalnotice>
 

documentation/yocto-project-qs/yocto-project-qs.xml

@@ -20,15 +20,20 @@
                 <itemizedlist>
                     <listitem><para>
                         For the latest version of the Yocto Project Quick
-                        Start associated with this Yocto Project release,
+                        Start associated with this Yocto Project release
+                        (version &YOCTO_DOC_VERSION;),
                         see the Yocto Project Quick Start from the
-                        <ulink url='&YOCTO_HOME_URL;'>Yocto Project website</ulink>.
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>.
                         </para></listitem>
                     <listitem><para>
-                        Later releases of the Yocto Project might exist and the
-                        associated Yocto Project Quick Start for any later
-                        release can be found at
-                        <ulink url='&YOCTO_HOME_URL;/documentation'></ulink>.
+                        This version of the manual is version
+                        &YOCTO_DOC_VERSION;.
+                        For later releases of the Yocto Project (if they exist),
+                        go to the
+                        <ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
+                        and use the drop-down "Active Releases" button
+                        and choose the Yocto Project version for which you want
+                        the manual.
                         </para></listitem>
                     <listitem><para>
                         For an in-development version of the Yocto Project

meta-poky/conf/distro/poky-tiny.conf

@@ -30,6 +30,7 @@
 
 require conf/distro/poky.conf
 DISTRO = "poky-tiny"
+DISTROOVERRIDES = "poky:poky-tiny"
 TCLIBC = "musl"
 # FIXME: consider adding a new "tiny" feature
 #DISTRO_FEATURES_append = " tiny"
@@ -37,7 +38,7 @@ TCLIBC = "musl"
 # Distro config is evaluated after the machine config, so we have to explicitly
 # set the kernel provider to override a machine config.
 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-tiny"
-PREFERRED_VERSION_linux-yocto-tiny ?= "4.4%"
+PREFERRED_VERSION_linux-yocto-tiny ?= "4.9%"
 
 # We can use packagegroup-core-boot, but in the future we may need a new packagegroup-core-tiny
 #POKY_DEFAULT_EXTRA_RDEPENDS += "packagegroup-core-boot"

meta-poky/conf/distro/poky.conf

@@ -21,13 +21,13 @@ POKY_DEFAULT_EXTRA_RRECOMMENDS = "kernel-module-af-packet"
 
 DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT} ${DISTRO_FEATURES_LIBC} ${POKY_DEFAULT_DISTRO_FEATURES}"
 
-PREFERRED_VERSION_linux-yocto ?= "4.8%"
-PREFERRED_VERSION_linux-yocto_qemux86 ?= "4.8%"
-PREFERRED_VERSION_linux-yocto_qemux86-64 ?= "4.8%"
-PREFERRED_VERSION_linux-yocto_qemuarm ?= "4.8%"
-PREFERRED_VERSION_linux-yocto_qemumips ?= "4.8%"
-PREFERRED_VERSION_linux-yocto_qemumips64 ?= "4.8%"
-PREFERRED_VERSION_linux-yocto_qemuppc ?= "4.8%"
+PREFERRED_VERSION_linux-yocto ?= "4.10%"
+PREFERRED_VERSION_linux-yocto_qemux86 ?= "4.10%"
+PREFERRED_VERSION_linux-yocto_qemux86-64 ?= "4.10%"
+PREFERRED_VERSION_linux-yocto_qemuarm ?= "4.10%"
+PREFERRED_VERSION_linux-yocto_qemumips ?= "4.10%"
+PREFERRED_VERSION_linux-yocto_qemumips64 ?= "4.10%"
+PREFERRED_VERSION_linux-yocto_qemuppc ?= "4.10%"
 
 SDK_NAME = "${DISTRO}-${TCLIBC}-${SDK_ARCH}-${IMAGE_BASENAME}-${TUNE_PKGARCH}"
 SDKPATH = "/opt/${DISTRO}/${SDK_VERSION}"
@@ -70,24 +70,17 @@ https://.*/.*    http://downloads.yoctoproject.org/mirror/sources/ \n"
 CONNECTIVITY_CHECK_URIS ?= "https://www.example.com/"
 
 SANITY_TESTED_DISTROS ?= " \
-            poky-1.8 \n \
-            poky-2.0 \n \
             poky-2.1 \n \
             poky-2.2 \n \
-            ubuntu-14.04 \n \
-            ubuntu-14.10 \n \
             ubuntu-15.04 \n \
-            ubuntu-15.10 \n \
             ubuntu-16.04 \n \
             ubuntu-16.10 \n \
-            fedora-22 \n \
-            fedora-23 \n \
             fedora-24 \n \
             fedora-25 \n \
             centos-7 \n \
             debian-8 \n \
-            opensuse-13.2 \n \
             opensuse-42.1 \n \
+            opensuse-42.2 \n \
             "
 #
 # OELAYOUT_ABI allows us to notify users when the format of TMPDIR changes in 

meta-selftest/lib/oeqa/runtime/cases/selftest.py

@@ -8,7 +8,7 @@ class Selftest(OERuntimeTestCase):
         """
         Summary: Check basic package installation functionality.
         Expected: 1. Before the test socat must be installed using scp.
-                  2. After the test socat must be unistalled using ssh.
+                  2. After the test socat must be uninstalled using ssh.
                      This can't be checked in this test.
         Product: oe-core
         Author: Mariano Lopez <mariano.lopez@intel.com>
@@ -18,10 +18,10 @@ class Selftest(OERuntimeTestCase):
         self.assertEqual(status, 0, msg="socat is not installed")
 
     @OETestDepends(['selftest.Selftest.test_install_package'])
-    def test_verify_unistall(self):
+    def test_verify_uninstall(self):
         """
         Summary: Check basic package installation functionality.
-        Expected: 1. test_install_package must unistall socat.
+        Expected: 1. test_install_package must uninstall socat.
                      This test is just to verify that.
         Product: oe-core
         Author: Mariano Lopez <mariano.lopez@intel.com>

meta-selftest/recipes-test/container-image/container-image-testpkg.bb

@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+    install -d ${D}${bindir}
+    touch ${D}${bindir}/theapp
+}

meta-selftest/recipes-test/container-image/container-test-image.bb

@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image

meta-selftest/recipes-test/images/wic-image-minimal.bb

@@ -6,7 +6,7 @@ IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
 
 IMAGE_FSTYPES = "wic"
 
-DEPENDS = "syslinux syslinux-native dosfstools-native mtools-native gptfdisk-native"
+WKS_FILE_DEPENDS = "syslinux syslinux-native dosfstools-native mtools-native gptfdisk-native"
 
 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 

meta-yocto-bsp/conf/machine/beaglebone.conf

@@ -23,7 +23,7 @@ do_image_wic[depends] += "mtools-native:do_populate_sysroot dosfstools-native:do
 SERIAL_CONSOLE = "115200 ttyO0"
 
 PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
-PREFERRED_VERSION_linux-yocto ?= "4.8%"
+PREFERRED_VERSION_linux-yocto ?= "4.9%"
 
 KERNEL_IMAGETYPE = "zImage"
 KERNEL_DEVICETREE = "am335x-bone.dtb am335x-boneblack.dtb"

meta-yocto-bsp/conf/machine/edgerouter.conf

@@ -11,7 +11,7 @@ KERNEL_ALT_IMAGETYPE = "vmlinux.bin"
 KERNEL_IMAGE_STRIP_EXTRA_SECTIONS  = ".comment"
 
 PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
-PREFERRED_VERSION_linux-yocto ?= "4.8%"
+PREFERRED_VERSION_linux-yocto ?= "4.9%"
 
 SERIAL_CONSOLE = "115200 ttyS0"
 USE_VT ?= "0"

meta-yocto-bsp/conf/machine/genericx86-64.conf

@@ -7,6 +7,6 @@ DEFAULTTUNE ?= "core2-64"
 require conf/machine/include/tune-core2.inc
 require conf/machine/include/genericx86-common.inc
 
-PREFERRED_VERSION_linux-yocto_genericx86-64 ?= "4.8%"
+PREFERRED_VERSION_linux-yocto_genericx86-64 ?= "4.10%"
 
 SERIAL_CONSOLES_CHECK = "ttyS0"

meta-yocto-bsp/conf/machine/genericx86.conf

@@ -9,4 +9,4 @@ require conf/machine/include/genericx86-common.inc
 
 MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "gma500-gfx-check"
 
-PREFERRED_VERSION_linux-yocto_genericx86 ?= "4.8%"
+PREFERRED_VERSION_linux-yocto_genericx86 ?= "4.10%"

meta-yocto-bsp/conf/machine/mpc8315e-rdb.conf

@@ -14,7 +14,7 @@ SERIAL_CONSOLE = "115200 ttyS0"
 
 MACHINE_FEATURES = "keyboard pci ext2 ext3 serial"
 
-PREFERRED_VERSION_linux-yocto ?= "4.8%"
+PREFERRED_VERSION_linux-yocto ?= "4.9%"
 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
 
 PREFERRED_PROVIDER_virtual/xserver ?= "xserver-xorg"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.1.bbappend

@@ -7,11 +7,11 @@ KBRANCH_mpc8315e-rdb = "standard/fsl-mpc8315e-rdb"
 KMACHINE_genericx86 ?= "common-pc"
 KMACHINE_genericx86-64 ?= "common-pc-64"
 
-SRCREV_machine_genericx86    ?= "fec49247816d7045aa8abe0047bcd4737af9a853"
-SRCREV_machine_genericx86-64 ?= "fec49247816d7045aa8abe0047bcd4737af9a853"
-SRCREV_machine_edgerouter ?= "fec49247816d7045aa8abe0047bcd4737af9a853"
-SRCREV_machine_beaglebone ?= "938cc4ac8d36f166c9e2e0517d6ffd6d278fe631"
-SRCREV_machine_mpc8315e-rdb ?= "fec49247816d7045aa8abe0047bcd4737af9a853"
+SRCREV_machine_genericx86    ?= "aadda25ee787681666f1d7961cd6c5a641da678d"
+SRCREV_machine_genericx86-64 ?= "aadda25ee787681666f1d7961cd6c5a641da678d"
+SRCREV_machine_edgerouter ?= "34753ac5e7c52bd1f0144c0a5ce9f0c6345f99d8"
+SRCREV_machine_beaglebone ?= "da5b7beb9866d5de72d61f297c682224e3e62ab5"
+SRCREV_machine_mpc8315e-rdb ?= "34753ac5e7c52bd1f0144c0a5ce9f0c6345f99d8"
 
 COMPATIBLE_MACHINE_genericx86 = "genericx86"
 COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
@@ -19,8 +19,8 @@ COMPATIBLE_MACHINE_edgerouter = "edgerouter"
 COMPATIBLE_MACHINE_beaglebone = "beaglebone"
 COMPATIBLE_MACHINE_mpc8315e-rdb = "mpc8315e-rdb"
 
-LINUX_VERSION_genericx86 = "4.1.36"
-LINUX_VERSION_genericx86-64 = "4.1.36"
-LINUX_VERSION_edgerouter = "4.1.36"
-LINUX_VERSION_beaglebone = "4.1.36"
-LINUX_VERSION_mpc8315e-rdb = "4.1.36"
+LINUX_VERSION_genericx86 = "4.1.37"
+LINUX_VERSION_genericx86-64 = "4.1.37"
+LINUX_VERSION_edgerouter = "4.1.37"
+LINUX_VERSION_beaglebone = "4.1.37"
+LINUX_VERSION_mpc8315e-rdb = "4.1.37"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.10.bbappend

@@ -0,0 +1,14 @@
+KBRANCH_genericx86  = "standard/base"
+KBRANCH_genericx86-64  = "standard/base"
+
+KMACHINE_genericx86 ?= "common-pc"
+KMACHINE_genericx86-64 ?= "common-pc-64"
+
+SRCREV_machine_genericx86    ?= "b259a5d744e8955a03405c6b59c5a61724755ea8"
+SRCREV_machine_genericx86-64 ?= "b259a5d744e8955a03405c6b59c5a61724755ea8"
+
+COMPATIBLE_MACHINE_genericx86 = "genericx86"
+COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
+
+LINUX_VERSION_genericx86 = "4.10"
+LINUX_VERSION_genericx86-64 = "4.10"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.4.bbappend

@@ -7,11 +7,11 @@ KBRANCH_edgerouter = "standard/edgerouter"
 KBRANCH_beaglebone = "standard/beaglebone"
 KBRANCH_mpc8315e-rdb = "standard/fsl-mpc8315e-rdb"
 
-SRCREV_machine_genericx86    ?= "35482df5d5ba0807eb8a7c40b554bd657e3f9987"
-SRCREV_machine_genericx86-64 ?= "35482df5d5ba0807eb8a7c40b554bd657e3f9987"
-SRCREV_machine_edgerouter ?= "35482df5d5ba0807eb8a7c40b554bd657e3f9987"
-SRCREV_machine_beaglebone ?= "35482df5d5ba0807eb8a7c40b554bd657e3f9987"
-SRCREV_machine_mpc8315e-rdb ?= "772f071dbdd4b813c921058ddf9cba207237228b"
+SRCREV_machine_genericx86    ?= "ad8b1d659ddd2699ebf7d50ef9de8940b157bfc2"
+SRCREV_machine_genericx86-64 ?= "ad8b1d659ddd2699ebf7d50ef9de8940b157bfc2"
+SRCREV_machine_edgerouter ?= "ad8b1d659ddd2699ebf7d50ef9de8940b157bfc2"
+SRCREV_machine_beaglebone ?= "ad8b1d659ddd2699ebf7d50ef9de8940b157bfc2"
+SRCREV_machine_mpc8315e-rdb ?= "b01922f8b85d96b9f6996b17ed0995c6f782fdbf"
 
 COMPATIBLE_MACHINE_genericx86 = "genericx86"
 COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
@@ -19,8 +19,8 @@ COMPATIBLE_MACHINE_edgerouter = "edgerouter"
 COMPATIBLE_MACHINE_beaglebone = "beaglebone"
 COMPATIBLE_MACHINE_mpc8315e-rdb = "mpc8315e-rdb"
 
-LINUX_VERSION_genericx86 = "4.4.36"
-LINUX_VERSION_genericx86-64 = "4.4.36"
-LINUX_VERSION_edgerouter = "4.4.36"
-LINUX_VERSION_beaglebone = "4.4.36"
-LINUX_VERSION_mpc8315e-rdb = "4.4.36"
+LINUX_VERSION_genericx86 = "4.4.41"
+LINUX_VERSION_genericx86-64 = "4.4.41"
+LINUX_VERSION_edgerouter = "4.4.41"
+LINUX_VERSION_beaglebone = "4.4.41"
+LINUX_VERSION_mpc8315e-rdb = "4.4.41"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.8.bbappend

@@ -7,11 +7,11 @@ KBRANCH_edgerouter = "standard/edgerouter"
 KBRANCH_beaglebone = "standard/beaglebone"
 KBRANCH_mpc8315e-rdb = "standard/fsl-mpc8315e-rdb"
 
-SRCREV_machine_genericx86    ?= "021b4aef55b44597587a1ce5879be642b3dca155"
-SRCREV_machine_genericx86-64 ?= "021b4aef55b44597587a1ce5879be642b3dca155"
-SRCREV_machine_edgerouter ?= "6076f16536329465b62bd2037b8582a5e18f85d1"
-SRCREV_machine_beaglebone ?= "85dc85153cd7e3b72d34f967c4c0edde590c79a8"
-SRCREV_machine_mpc8315e-rdb ?= "f73222eb3bbd07a45564397a88dec554e848da7d"
+SRCREV_machine_genericx86    ?= "9bcb4ea3fa107f1a8790c8c3408eb250e8d1d66e"
+SRCREV_machine_genericx86-64 ?= "9bcb4ea3fa107f1a8790c8c3408eb250e8d1d66e"
+SRCREV_machine_edgerouter ?= "e9912dbfdabfed45b2e15feabfc66a404a165d9b"
+SRCREV_machine_beaglebone ?= "07250a1be0e99eb54bf7d3b629125cd09129d6f5"
+SRCREV_machine_mpc8315e-rdb ?= "38e8cf7d4e6d8546bb631d109c60243d2ee01aef"
 
 COMPATIBLE_MACHINE_genericx86 = "genericx86"
 COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
@@ -19,8 +19,8 @@ COMPATIBLE_MACHINE_edgerouter = "edgerouter"
 COMPATIBLE_MACHINE_beaglebone = "beaglebone"
 COMPATIBLE_MACHINE_mpc8315e-rdb = "mpc8315e-rdb"
 
-LINUX_VERSION_genericx86 = "4.8.12"
-LINUX_VERSION_genericx86-64 = "4.8.12"
-LINUX_VERSION_edgerouter = "4.8.12"
-LINUX_VERSION_beaglebone = "4.8.12"
-LINUX_VERSION_mpc8315e-rdb = "4.8.12"
+LINUX_VERSION_genericx86 = "4.8.17"
+LINUX_VERSION_genericx86-64 = "4.8.17"
+LINUX_VERSION_edgerouter = "4.8.18"
+LINUX_VERSION_beaglebone = "4.8.18"
+LINUX_VERSION_mpc8315e-rdb = "4.8.18"

meta-yocto-bsp/recipes-kernel/linux/linux-yocto_4.9.bbappend

@@ -0,0 +1,26 @@
+KBRANCH_genericx86  = "standard/base"
+KBRANCH_genericx86-64  = "standard/base"
+
+KMACHINE_genericx86 ?= "common-pc"
+KMACHINE_genericx86-64 ?= "common-pc-64"
+KBRANCH_edgerouter = "standard/edgerouter"
+KBRANCH_beaglebone = "standard/beaglebone"
+KBRANCH_mpc8315e-rdb = "standard/fsl-mpc8315e-rdb"
+
+SRCREV_machine_genericx86    ?= "95c0a80ee83f1cf8e59d733f36e8a9dfd50a0098"
+SRCREV_machine_genericx86-64 ?= "95c0a80ee83f1cf8e59d733f36e8a9dfd50a0098"
+SRCREV_machine_edgerouter ?= "c85c54f5bf53b98afe8105e91bffcdb6c60afe8f"
+SRCREV_machine_beaglebone ?= "c85c54f5bf53b98afe8105e91bffcdb6c60afe8f"
+SRCREV_machine_mpc8315e-rdb ?= "6b67f448d63917f5ea306eb293cd9844077e4a61"
+
+COMPATIBLE_MACHINE_genericx86 = "genericx86"
+COMPATIBLE_MACHINE_genericx86-64 = "genericx86-64"
+COMPATIBLE_MACHINE_edgerouter = "edgerouter"
+COMPATIBLE_MACHINE_beaglebone = "beaglebone"
+COMPATIBLE_MACHINE_mpc8315e-rdb = "mpc8315e-rdb"
+
+LINUX_VERSION_genericx86 = "4.9.13"
+LINUX_VERSION_genericx86-64 = "4.9.13"
+LINUX_VERSION_edgerouter = "4.9.8"
+LINUX_VERSION_beaglebone = "4.9.8"
+LINUX_VERSION_mpc8315e-rdb = "4.9.8"

meta/classes/autotools.bbclass

@@ -143,7 +143,6 @@ python autotools_aclocals () {
     # Refresh variable with cache files
     d.setVar("CONFIG_SITE", siteinfo_get_files(d, aclocalcache=True))
 }
-autotools_copy_aclocals[vardepsexclude] += "MACHINE SDK_ARCH BUILD_ARCH SDK_OS BB_TASKDEPDATA"
 
 CONFIGURE_FILES = "${S}/configure.in ${S}/configure.ac ${S}/config.h.in ${S}/acinclude.m4 Makefile.am"
 

meta/classes/base.bbclass

@@ -10,7 +10,7 @@ inherit utility-tasks
 inherit metadata_scm
 inherit logging
 
-OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath"
+OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"
 OE_IMPORTS[type] = "list"
 
 def oe_import(d):
@@ -222,7 +222,6 @@ python base_eventhandler() {
 
     if isinstance(e, bb.event.BuildStarted):
         localdata = bb.data.createCopy(e.data)
-        bb.data.update_data(localdata)
         statuslines = []
         for func in oe.data.typed_value('BUILDCFG_FUNCS', localdata):
             g = globals()
@@ -355,7 +354,6 @@ def set_packagetriplet(d):
         localdata = bb.data.createCopy(d)
         overrides = localdata.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item
         localdata.setVar("OVERRIDES", overrides)
-        bb.data.update_data(localdata)
 
         archs.append(localdata.getVar("PACKAGE_ARCHS").split())
         tos.append(localdata.getVar("TARGET_OS"))

meta/classes/binconfig.bbclass

@@ -13,14 +13,14 @@ def get_binconfig_mangle(d):
         s += " -e 's:=%s${exec_prefix}/:=\\1OEEXECPREFIX/:'" % optional_quote
         s += " -e 's:-L${libdir}:-LOELIBDIR:;'"
         s += " -e 's:-I${includedir}:-IOEINCDIR:;'"
+        s += " -e 's:-L${WORKDIR}:-LOELIBDIR:'"
+        s += " -e 's:-I${WORKDIR}:-I-IOEINCDIR:'"
         s += " -e 's:OEBASELIBDIR:${STAGING_BASELIBDIR}:;'"
         s += " -e 's:OELIBDIR:${STAGING_LIBDIR}:;'"
         s += " -e 's:OEINCDIR:${STAGING_INCDIR}:;'"
         s += " -e 's:OEDATADIR:${STAGING_DATADIR}:'"
         s += " -e 's:OEPREFIX:${STAGING_DIR_HOST}${prefix}:'"
         s += " -e 's:OEEXECPREFIX:${STAGING_DIR_HOST}${exec_prefix}:'"
-        s += " -e 's:-I${WORKDIR}:-I${STAGING_INCDIR}:'"
-        s += " -e 's:-L${WORKDIR}:-L${STAGING_LIBDIR}:'"
         if d.getVar("OE_BINCONFIG_EXTRA_MANGLE", False):
             s += d.getVar("OE_BINCONFIG_EXTRA_MANGLE")
 

meta/classes/buildhistory.bbclass

@@ -414,7 +414,7 @@ buildhistory_get_installed() {
 
 	# Produce installed package sizes list
 	oe-pkgdata-util -p ${PKGDATA_DIR} read-value "PKGSIZE" -n -f $pkgcache > $1/installed-package-sizes.tmp
-	cat $1/installed-package-sizes.tmp | awk '{print $2 "\tKiB " $1}' | sort -n -r > $1/installed-package-sizes.txt
+	cat $1/installed-package-sizes.tmp | awk '{print $2 "\tKiB\t" $1}' | sort -n -r > $1/installed-package-sizes.txt
 	rm $1/installed-package-sizes.tmp
 
 	# We're now done with the cache, delete it
@@ -594,7 +594,6 @@ def buildhistory_get_build_id(d):
     if d.getVar('BB_WORKERCONTEXT') != '1':
         return ""
     localdata = bb.data.createCopy(d)
-    bb.data.update_data(localdata)
     statuslines = []
     for func in oe.data.typed_value('BUILDCFG_FUNCS', localdata):
         g = globals()

meta/classes/buildstats.bbclass

@@ -31,6 +31,11 @@ def get_process_cputime(pid):
                 i = f.readline().strip()
                 if not i:
                     break
+                if not ":" in i:
+                    # one more extra line is appended (empty or containing "0")
+                    # most probably due to race condition in kernel while
+                    # updating IO stats
+                    break
                 i = i.split(": ")
                 iostats[i[0]] = i[1]
     resources = resource.getrusage(resource.RUSAGE_SELF)

meta/classes/copyleft_filter.bbclass

@@ -6,7 +6,7 @@
 #
 # vi:sts=4:sw=4:et
 
-COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL*'
+COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL* AGPL*'
 COPYLEFT_LICENSE_INCLUDE[type] = 'list'
 COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which include licenses'
 

meta/classes/cve-check.bbclass

@@ -62,7 +62,7 @@ python do_cve_check () {
 }
 
 addtask cve_check after do_unpack before do_build
-do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
+do_cve_check[depends] = "cve-check-tool-native:do_populate_sysroot cve-check-tool-native:do_populate_cve_db"
 do_cve_check[nostamp] = "1"
 
 python cve_check_cleanup () {

meta/classes/distrodata.bbclass

@@ -38,32 +38,27 @@ python do_distrodata_np() {
             pnstripped = pn.split("-native")
             bb.note("Native Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pn.find("-cross") != -1:
             pnstripped = pn.split("-cross")
             bb.note("cross Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pn.find("-crosssdk") != -1:
             pnstripped = pn.split("-crosssdk")
             bb.note("cross Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pn.startswith("nativesdk-"):
             pnstripped = pn.replace("nativesdk-", "")
             bb.note("NativeSDK Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
 
         if pn.find("-initial") != -1:
             pnstripped = pn.split("-initial")
             bb.note("initial Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         """generate package information from .bb file"""
         pname = localdata.getVar('PN')
@@ -127,31 +122,26 @@ python do_distrodata() {
             pnstripped = pn.split("-native")
             bb.note("Native Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pn.startswith("nativesdk-"):
             pnstripped = pn.replace("nativesdk-", "")
             bb.note("NativeSDK Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pn.find("-cross") != -1:
             pnstripped = pn.split("-cross")
             bb.note("cross Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pn.find("-crosssdk") != -1:
             pnstripped = pn.split("-crosssdk")
             bb.note("cross Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pn.find("-initial") != -1:
             pnstripped = pn.split("-initial")
             bb.note("initial Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         """generate package information from .bb file"""
         pname = localdata.getVar('PN')
@@ -292,7 +282,6 @@ python do_checkpkg() {
             pnstripped = pname.split("-native")
             bb.note("Native Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pname.startswith("nativesdk-"):
             if d.getVar('BBCLASSEXTEND'):
@@ -300,19 +289,16 @@ python do_checkpkg() {
             pnstripped = pname.replace("nativesdk-", "")
             bb.note("NativeSDK Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pname.find("-cross") != -1:
             pnstripped = pname.split("-cross")
             bb.note("cross Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         if pname.find("-initial") != -1:
             pnstripped = pname.split("-initial")
             bb.note("initial Split: %s" % pnstripped)
             localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES'))
-            bb.data.update_data(localdata)
 
         pdesc = localdata.getVar('DESCRIPTION')
         pgrp = localdata.getVar('SECTION')
@@ -401,7 +387,6 @@ python do_distro_check() {
         return
 
     localdata = bb.data.createCopy(d)
-    bb.data.update_data(localdata)
     tmpdir = d.getVar('TMPDIR')
     distro_check_dir = os.path.join(tmpdir, "distro_check")
     logpath = d.getVar('LOG_DIR')

meta/classes/grub-efi.bbclass

@@ -133,7 +133,6 @@ python build_efi_cfg() {
 
         for btype in btypes:
             localdata.setVar('OVERRIDES', label + ':' + overrides)
-            bb.data.update_data(localdata)
 
             cfgfile.write('\nmenuentry \'%s%s\'{\n' % (label, btype[0]))
             lb = label

meta/classes/image-container.bbclass

@@ -0,0 +1,21 @@
+ROOTFS_BOOTSTRAP_INSTALL = ""
+IMAGE_TYPES_MASKED += "container"
+IMAGE_TYPEDEP_container = "tar.bz2"
+
+python __anonymous() {
+    if "container" in d.getVar("IMAGE_FSTYPES") and \
+       d.getVar("IMAGE_CONTAINER_NO_DUMMY") != "1" and \
+       "linux-dummy" not in d.getVar("PREFERRED_PROVIDER_virtual/kernel"):
+        msg = '"container" is in IMAGE_FSTYPES, but ' \
+              'PREFERRED_PROVIDER_virtual/kernel is not "linux-dummy". ' \
+              'Unless a particular kernel is needed, using linux-dummy will ' \
+              'prevent a kernel from being built, which can reduce ' \
+              'build times. If you don\'t want to use "linux-dummy", set ' \
+              '"IMAGE_CONTAINER_NO_DUMMY" to "1".'
+
+        # Raising skip recipe was Paul's clever idea. It causes the error to
+        # only be shown for the recipes actually requested to build, rather
+        # than bb.fatal which would appear for all recipes inheriting the
+        # class.
+        raise bb.parse.SkipRecipe(msg)
+}

meta/classes/image.bbclass

@@ -1,8 +1,8 @@
 inherit rootfs_${IMAGE_PKGTYPE}
 
-# Only Linux SDKs support populate_sdk_ext, fall back to populate_sdk
+# Only Linux SDKs support populate_sdk_ext, fall back to populate_sdk_base
 # in the non-Linux SDK_OS case, such as mingw32
-SDKEXTCLASS ?= "${@['populate_sdk', 'populate_sdk_ext']['linux' in d.getVar("SDK_OS")]}"
+SDKEXTCLASS ?= "${@['populate_sdk_base', 'populate_sdk_ext']['linux' in d.getVar("SDK_OS")]}"
 inherit ${SDKEXTCLASS}
 
 TOOLCHAIN_TARGET_TASK += "${PACKAGE_INSTALL}"
@@ -142,6 +142,9 @@ inherit ${IMAGE_TYPE_live}
 IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", "qcow2", "hdddirect"], "image-vm", "", d)}'
 inherit ${IMAGE_TYPE_vm}
 
+IMAGE_TYPE_container = '${@bb.utils.contains("IMAGE_FSTYPES", "container", "image-container", "", d)}'
+inherit ${IMAGE_TYPE_container}
+
 def build_uboot(d):
     if 'u-boot' in (d.getVar('IMAGE_FSTYPES') or ''):
         return "image_types_uboot"
@@ -151,6 +154,9 @@ def build_uboot(d):
 IMAGE_TYPE_uboot = "${@build_uboot(d)}"
 inherit ${IMAGE_TYPE_uboot}
 
+IMAGE_TYPE_wic = "image_types_wic"
+inherit ${IMAGE_TYPE_wic}
+
 python () {
     deps = " " + imagetypes_getdepends(d)
     d.appendVarFlag('do_rootfs', 'depends', deps)
@@ -187,8 +193,6 @@ python () {
 IMAGE_CLASSES += "image_types"
 inherit ${IMAGE_CLASSES}
 
-inherit image-wic
-
 IMAGE_POSTPROCESS_COMMAND ?= ""
 
 # some default locales
@@ -427,7 +431,6 @@ python () {
             debug = "setup_debugfs "
             realt = t[8:]
         localdata.setVar('OVERRIDES', '%s:%s' % (realt, old_overrides))
-        bb.data.update_data(localdata)
         localdata.setVar('type', realt)
         # Delete DATETIME so we don't expand any references to it now
         # This means the task's hash can be stable rather than having hardcoded

meta/classes/image_types.bbclass

@@ -33,8 +33,8 @@ def imagetypes_getdepends(d):
 
         for typedepends in (d.getVar("IMAGE_TYPEDEP_%s" % basetype) or "").split():
             base, rest = split_types(typedepends)
+            adddep(d.getVar('IMAGE_DEPENDS_%s' % base) , deps)
             resttypes += rest
-            adddep(d.getVar('IMAGE_DEPENDS_%s' % typedepends) , deps)
 
         for ctype in resttypes:
             adddep(d.getVar("CONVERSION_DEPENDS_%s" % ctype), deps)
@@ -242,6 +242,7 @@ IMAGE_TYPES = " \
     hdddirect \
     elf \
     wic wic.gz wic.bz2 wic.lzma \
+    container \
 "
 
 # Compression is a special case of conversion. The old variable

meta/classes/image_types_wic.bbclass

@@ -8,7 +8,7 @@ WICVARS ?= "\
 WKS_FILE ??= "${IMAGE_BASENAME}.${MACHINE}.wks"
 WKS_FILES ?= "${WKS_FILE} ${IMAGE_BASENAME}.wks"
 WKS_SEARCH_PATH ?= "${THISDIR}:${@':'.join('%s/wic' % p for p in '${BBPATH}'.split(':'))}:${@':'.join('%s/scripts/lib/wic/canned-wks' % l for l in '${BBPATH}:${COREBASE}'.split(':'))}"
-WKS_FULL_PATH = "${@wks_search('${WKS_FILES}'.split(), '${WKS_SEARCH_PATH}') or ''}"
+WKS_FULL_PATH = "${@wks_search(d.getVar('WKS_FILES').split(), d.getVar('WKS_SEARCH_PATH')) or ''}"
 
 def wks_search(files, search_path):
     for f in files:
@@ -40,11 +40,8 @@ USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 do_image_wic[depends] += "wic-tools:do_build"
-
-python () {
-    if d.getVar('USING_WIC') and 'do_bootimg' in d:
-        bb.build.addtask('do_image_wic', '', 'do_bootimg', d)
-}
+WKS_FILE_DEPENDS ??= ''
+DEPENDS += "${@ '${WKS_FILE_DEPENDS}' if d.getVar('USING_WIC') else '' }"
 
 python do_write_wks_template () {
     """Write out expanded template contents to WKS_FULL_PATH."""

meta/classes/insane.bbclass

@@ -398,7 +398,7 @@ def package_qa_check_dbg(path, name, d, elf, messages):
 
     if not "-dbg" in name and not "-ptest" in name:
         if '.debug' in path.split(os.path.sep):
-            messages("debug-files", "non debug package contains .debug directory: %s path %s" % \
+            package_qa_add_message(messages, "debug-files", "non debug package contains .debug directory: %s path %s" % \
                      (name, package_qa_clean_path(path,d)))
 
 QAPATHTEST[perms] = "package_qa_check_perm"
@@ -910,7 +910,6 @@ def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d):
     if not "-dbg" in pkg and not "packagegroup-" in pkg and not "-image" in pkg:
         localdata = bb.data.createCopy(d)
         localdata.setVar('OVERRIDES', pkg)
-        bb.data.update_data(localdata)
 
         # Now check the RDEPENDS
         rdepends = bb.utils.explode_deps(localdata.getVar('RDEPENDS') or "")
@@ -1017,7 +1016,6 @@ def package_qa_check_deps(pkg, pkgdest, skip, d):
 
     localdata = bb.data.createCopy(d)
     localdata.setVar('OVERRIDES', pkg)
-    bb.data.update_data(localdata)
 
     def check_valid_deps(var):
         try:
@@ -1223,6 +1221,9 @@ python do_package_qa () {
     bb.note("DONE with PACKAGE QA")
 }
 
+# binutils is used for most checks, so need to set as dependency
+# POPULATESYSROOTDEPS is defined in staging class.
+do_package_qa[depends] += "${POPULATESYSROOTDEPS}"
 do_package_qa[vardepsexclude] = "BB_TASKDEPDATA"
 do_package_qa[rdeptask] = "do_packagedata"
 addtask do_package_qa after do_packagedata do_package before do_build

meta/classes/kernel-module-split.bbclass

@@ -33,6 +33,7 @@ PACKAGESPLITFUNCS_prepend = "split_kernel_module_packages "
 KERNEL_MODULES_META_PACKAGE ?= "kernel-modules"
 
 KERNEL_MODULE_PACKAGE_PREFIX ?= ""
+KERNEL_MODULE_PACKAGE_SUFFIX ?= "-${KERNEL_VERSION}"
 KERNEL_MODULE_PROVIDE_VIRTUAL ?= "1"
 
 python split_kernel_module_packages () {
@@ -131,7 +132,8 @@ python split_kernel_module_packages () {
     module_regex = '^(.*)\.k?o$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
-    module_pattern = module_pattern_prefix + 'kernel-module-%s-' + d.getVar("KERNEL_VERSION")
+    module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
+    module_pattern = module_pattern_prefix + 'kernel-module-%s' + module_pattern_suffix
 
     postinst = d.getVar('pkg_postinst_modules')
     postrm = d.getVar('pkg_postrm_modules')

meta/classes/kernel-yocto.bbclass

@@ -148,7 +148,8 @@ do_kernel_metadata() {
 	# run1: pull all the configuration fragments, no matter where they come from
 	elements="`echo -n ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES}`"
 	if [ -n "${elements}" ]; then
-		scc --force -o ${S}/${meta_dir}:cfg,meta ${includes} ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES}
+		echo "${bsp_definition}" > ${S}/${meta_dir}/bsp_definition
+		scc --force -o ${S}/${meta_dir}:cfg,merge,meta ${includes} ${bsp_definition} ${sccs} ${patches} ${KERNEL_FEATURES}
 		if [ $? -ne 0 ]; then
 			bbfatal_log "Could not generate configuration queue for ${KMACHINE}."
 		fi
@@ -165,6 +166,7 @@ do_kernel_metadata() {
 }
 
 do_patch() {
+	set +e
 	cd ${S}
 
 	check_git_config
@@ -177,6 +179,19 @@ do_patch() {
 			bbfatal_log "Patch failures can be resolved in the linux source directory ${S})"
 		fi
 	fi
+
+	if [ -f "${meta_dir}/merge.queue" ]; then
+		# we need to merge all these branches
+		for b in $(cat ${meta_dir}/merge.queue); do
+			git show-ref --verify --quiet refs/heads/${b}
+			if [ $? -eq 0 ]; then
+				bbnote "Merging branch ${b}"
+				git merge -q --no-ff -m "Merge branch ${b}" ${b}
+			else
+				bbfatal "branch ${b} does not exist, cannot merge"
+			fi
+		done
+	fi
 }
 
 do_kernel_checkout() {
@@ -350,6 +365,10 @@ do_validate_branches() {
 			current_branch=`git rev-parse --abbrev-ref HEAD`
 			git branch "$current_branch-orig"
 			git reset --hard ${force_srcrev}
+			# We've checked out HEAD, make sure we cleanup kgit-s2q fence post check
+			# so the patches are applied as expected otherwise no patching
+			# would be done in some corner cases.
+			kgit-s2q --clean
 		fi
 	fi
 }

meta/classes/kernel.bbclass

@@ -4,6 +4,8 @@ PROVIDES += "virtual/kernel"
 DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native lzop-native"
 PACKAGE_WRITE_DEPS += "depmodwrapper-cross virtual/update-alternatives-native"
 
+do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot"
+
 S = "${STAGING_KERNEL_DIR}"
 B = "${WORKDIR}/build"
 KBUILD_OUTPUT = "${B}"
@@ -325,6 +327,10 @@ do_install[prefuncs] += "package_get_auto_pr"
 
 # Must be ran no earlier than after do_kernel_checkout or else Makefile won't be in ${S}/Makefile
 do_kernel_version_sanity_check() {
+	if [ "x${KERNEL_VERSION_SANITY_SKIP}" = "x1" ]; then
+		exit 0
+	fi
+
 	# The Makefile determines the kernel version shown at runtime
 	# Don't use KERNEL_VERSION because the headers it grabs the version from aren't generated until do_compile
 	VERSION=$(grep "^VERSION =" ${S}/Makefile | sed s/.*=\ *//)
@@ -348,7 +354,7 @@ do_kernel_version_sanity_check() {
 	reg="${reg}${EXTRAVERSION}"
 
 	if [ -z `echo ${PV} | grep -E "${reg}"` ]; then
-		bbfatal "Package Version (${PV}) does not match of kernel being built (${vers}). Please update the PV variable to match the kernel source."
+		bbfatal "Package Version (${PV}) does not match of kernel being built (${vers}). Please update the PV variable to match the kernel source or set KERNEL_VERSION_SANITY_SKIP=\"1\" in your recipe."
 	fi
 	exit 0
 }
@@ -528,6 +534,9 @@ do_kernel_link_images() {
 	if [ -f ../../../vmlinuz ]; then
 		ln -sf ../../../vmlinuz
 	fi
+	if [ -f ../../../vmlinuz.bin ]; then
+		ln -sf ../../../vmlinuz.bin
+	fi
 }
 
 do_strip() {
@@ -608,8 +617,6 @@ kernel_do_deploy() {
 		ln -sf ${base_name}.bin ${DEPLOYDIR}/${type}
 	done
 
-	cp ${COREBASE}/meta/files/deploydir_readme.txt ${DEPLOYDIR}/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
-
 	cd ${B}
 	# Update deploy directory
 	for type in ${KERNEL_IMAGETYPES} ; do

meta/classes/license.bbclass

@@ -131,6 +131,10 @@ def write_license_files(d, license_manifest, pkg_dic):
                 bb.utils.mkdirhier(pkg_rootfs_license_dir)
                 pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'),
                                             pkg_dic[pkg]["PN"]) 
+
+                pkg_manifest_licenses = [canonical_license(d, lic) \
+                        for lic in pkg_dic[pkg]["LICENSES"]]
+
                 licenses = os.listdir(pkg_license_dir)
                 for lic in licenses:
                     rootfs_license = os.path.join(rootfs_license_dir, lic)
@@ -138,9 +142,18 @@ def write_license_files(d, license_manifest, pkg_dic):
                     pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic)
 
                     if re.match("^generic_.*$", lic):
-                        generic_lic = re.search("^generic_(.*)$", lic).group(1)
-                        if oe.license.license_ok(canonical_license(d,
-                            generic_lic), bad_licenses) == False:
+                        generic_lic = canonical_license(d,
+                                re.search("^generic_(.*)$", lic).group(1))
+
+                        # Do not copy generic license into package if isn't
+                        # declared into LICENSES of the package.
+                        if not re.sub('\+$', '', generic_lic) in \
+                                [re.sub('\+', '', lic) for lic in \
+                                 pkg_manifest_licenses]:
+                            continue
+
+                        if oe.license.license_ok(generic_lic,
+                                bad_licenses) == False:
                             continue
 
                         if not os.path.exists(rootfs_license):
@@ -279,7 +292,7 @@ def get_deployed_files(man_file):
     """
 
     dep_files = []
-    excluded_files = ["README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt"]
+    excluded_files = []
     with open(man_file, "r") as manifest:
         all_files = manifest.read()
     for f in all_files.splitlines():
@@ -345,7 +358,7 @@ def copy_license_files(lic_files_paths, destdir):
     import errno
 
     bb.utils.mkdirhier(destdir)
-    for (basename, path) in lic_files_paths:
+    for (basename, path, beginline, endline) in lic_files_paths:
         try:
             src = path
             dst = os.path.join(destdir, basename)
@@ -353,7 +366,7 @@ def copy_license_files(lic_files_paths, destdir):
                 os.remove(dst)
             if os.path.islink(src):
                 src = os.path.realpath(src)
-            canlink = os.access(src, os.W_OK) and (os.stat(src).st_dev == os.stat(destdir).st_dev)
+            canlink = os.access(src, os.W_OK) and (os.stat(src).st_dev == os.stat(destdir).st_dev) and beginline is None and endline is None
             if canlink:
                 try:
                     os.link(src, dst)
@@ -364,20 +377,19 @@ def copy_license_files(lic_files_paths, destdir):
                         canlink = False
                     else:
                         raise
-                try:
-                    if canlink:
-                        os.chown(dst,0,0)
-                except OSError as err:
-                    if err.errno in (errno.EPERM, errno.EINVAL):
-                        # Suppress "Operation not permitted" error, as
-                        # sometimes this function is not executed under pseudo.
-                        # Also ignore "Invalid argument" errors that happen in
-                        # some (unprivileged) container environments (no root).
-                        pass
-                    else:
-                        raise
+                # Only chown if we did hardling, and, we're running under pseudo
+                if canlink and os.environ.get('PSEUDO_DISABLED') == '0':
+                    os.chown(dst,0,0)
             if not canlink:
-                shutil.copyfile(src, dst)
+                begin_idx = int(beginline)-1 if beginline is not None else None
+                end_idx = int(endline) if endline is not None else None
+                if begin_idx is None and end_idx is None:
+                    shutil.copyfile(src, dst)
+                else:
+                    with open(src, 'r') as src_f:
+                        with open(dst, 'w') as dst_f:
+                            dst_f.write(''.join(src_f.readlines()[begin_idx:end_idx]))
+
         except Exception as e:
             bb.warn("Could not copy license file %s to %s: %s" % (src, dst, e))
 
@@ -398,6 +410,8 @@ def find_license_files(d):
     generic_directory = d.getVar('COMMON_LICENSE_DIR')
     # List of basename, path tuples
     lic_files_paths = []
+    # hash for keep track generic lics mappings
+    non_generic_lics = {}
     # Entries from LIC_FILES_CHKSUM
     lic_chksums = {}
     license_source_dirs = []
@@ -447,7 +461,8 @@ def find_license_files(d):
             # we really should copy to generic_ + spdx_generic, however, that ends up messing the manifest
             # audit up. This should be fixed in emit_pkgdata (or, we actually got and fix all the recipes)
 
-            lic_files_paths.append(("generic_" + license_type, os.path.join(license_source, spdx_generic)))
+            lic_files_paths.append(("generic_" + license_type, os.path.join(license_source, spdx_generic),
+                                    None, None))
 
             # The user may attempt to use NO_GENERIC_LICENSE for a generic license which doesn't make sense
             # and should not be allowed, warn the user in this case.
@@ -458,7 +473,8 @@ def find_license_files(d):
             # if NO_GENERIC_LICENSE is set, we copy the license files from the fetched source
             # of the package rather than the license_source_dirs.
             lic_files_paths.append(("generic_" + license_type,
-                                    os.path.join(srcdir, non_generic_lic)))
+                                    os.path.join(srcdir, non_generic_lic), None, None))
+            non_generic_lics[non_generic_lic] = license_type
         else:
             # Add explicity avoid of CLOSED license because this isn't generic
             if license_type != 'CLOSED':
@@ -476,7 +492,9 @@ def find_license_files(d):
             bb.fatal("%s: LIC_FILES_CHKSUM contains an invalid URL:  %s" % (d.getVar('PF'), url))
         # We want the license filename and path
         chksum = parm['md5'] if 'md5' in parm else parm['sha256']
-        lic_chksums[path] = chksum
+        beginline = parm.get('beginline')
+        endline = parm.get('endline')
+        lic_chksums[path] = (chksum, beginline, endline)
 
     v = FindVisitor()
     try:
@@ -485,19 +503,22 @@ def find_license_files(d):
         bb.fatal('%s: %s' % (d.getVar('PF'), exc))
     except SyntaxError:
         bb.warn("%s: Failed to parse it's LICENSE field." % (d.getVar('PF')))
-
     # Add files from LIC_FILES_CHKSUM to list of license files
     lic_chksum_paths = defaultdict(OrderedDict)
-    for path, chksum in lic_chksums.items():
-        lic_chksum_paths[os.path.basename(path)][chksum] = os.path.join(srcdir, path)
+    for path, data in lic_chksums.items():
+        lic_chksum_paths[os.path.basename(path)][data] = (os.path.join(srcdir, path), data[1], data[2])
     for basename, files in lic_chksum_paths.items():
         if len(files) == 1:
-            lic_files_paths.append((basename, list(files.values())[0]))
+            # Don't copy again a LICENSE already handled as non-generic
+            if basename in non_generic_lics:
+                continue
+            data = list(files.values())[0]
+            lic_files_paths.append(tuple([basename] + list(data)))
         else:
             # If there are multiple different license files with identical
             # basenames we rename them to <file>.0, <file>.1, ...
-            for i, path in enumerate(files.values()):
-                lic_files_paths.append(("%s.%d" % (basename, i), path))
+            for i, data in enumerate(files.values()):
+                lic_files_paths.append(tuple(["%s.%d" % (basename, i)] + list(data)))
 
     return lic_files_paths
 

meta/classes/multilib.bbclass

@@ -36,7 +36,6 @@ python multilib_virtclass_handler () {
         e.data.setVar("MLPREFIX", variant + "-")
         override = ":virtclass-multilib-" + variant
         e.data.setVar("OVERRIDES", e.data.getVar("OVERRIDES", False) + override)
-        bb.data.update_data(e.data)
         return
 
     if bb.data.inherits_class('native', e.data):

meta/classes/multilib_global.bbclass

@@ -29,7 +29,6 @@ def preferred_ml_updates(d):
                 localdata = bb.data.createCopy(d)
                 override = ":virtclass-multilib-" + p
                 localdata.setVar("OVERRIDES", localdata.getVar("OVERRIDES", False) + override)
-                bb.data.update_data(localdata)
                 if "-canadian-" in pkg:
                     newname = localdata.expand(v)
                 else:
@@ -57,7 +56,6 @@ def preferred_ml_updates(d):
                 localdata = bb.data.createCopy(d)
                 override = ":virtclass-multilib-" + p
                 localdata.setVar("OVERRIDES", localdata.getVar("OVERRIDES", False) + override)
-                bb.data.update_data(localdata)
                 newname = localdata.expand(prov)
                 if newname != prov:
                     newval = localdata.expand(val)
@@ -80,7 +78,6 @@ def preferred_ml_updates(d):
             localdata = bb.data.createCopy(d)
             override = ":virtclass-multilib-" + p
             localdata.setVar("OVERRIDES", localdata.getVar("OVERRIDES", False) + override)
-            bb.data.update_data(localdata)
             newname = localdata.expand(prov)
             if newname != prov and not d.getVar(newname, False):
                 d.setVar(newname, localdata.expand(newval))

meta/classes/native.bbclass

@@ -174,6 +174,11 @@ python native_virtclass_handler () {
 addhandler native_virtclass_handler
 native_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
 
+python do_addto_recipe_sysroot () {
+    bb.build.exec_func("extend_recipe_sysroot", d)
+}
+addtask addto_recipe_sysroot after do_populate_sysroot
+
 inherit nopackages
 
 do_packagedata[stamp-extra-info] = ""

meta/classes/npm.bbclass

@@ -32,6 +32,9 @@ npm_do_compile() {
 }
 
 npm_do_install() {
+	# changing the home directory to the working directory, the .npmrc will
+	# be created in this directory
+	export HOME=${WORKDIR}
 	mkdir -p ${NPM_INSTALLDIR}/
 	npm install --prefix ${D}${prefix} -g --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry
 	if [ -d ${D}${prefix}/etc ] ; then

meta/classes/package.bbclass

@@ -1613,15 +1613,31 @@ python package_do_shlibs() {
                 if name and name not in needed[pkg]:
                      needed[pkg].append((name, file, []))
 
+    def mingw_dll(file, needed, sonames, renames, pkgver):
+        if not os.path.exists(file):
+            return
+
+        if file.endswith(".dll"):
+            # assume all dlls are shared objects provided by the package
+            sonames.append((os.path.basename(file), os.path.dirname(file).replace(pkgdest + "/" + pkg, ''), pkgver))
+
+        if (file.endswith(".dll") or file.endswith(".exe")):
+            # use objdump to search for "DLL Name: .*\.dll"
+            p = sub.Popen([d.expand("${HOST_PREFIX}objdump"), "-p", file], stdout = sub.PIPE, stderr= sub.PIPE)
+            out, err = p.communicate()
+            # process the output, grabbing all .dll names
+            if p.returncode == 0:
+                for m in re.finditer("DLL Name: (.*?\.dll)$", out.decode(), re.MULTILINE | re.IGNORECASE):
+                    dllname = m.group(1)
+                    if dllname:
+                        needed[pkg].append((dllname, file, []))
+
     if d.getVar('PACKAGE_SNAP_LIB_SYMLINKS') == "1":
         snap_symlinks = True
     else:
         snap_symlinks = False
 
-    if (d.getVar('USE_LDCONFIG') or "1") == "1":
-        use_ldconfig = True
-    else:
-        use_ldconfig = False
+    use_ldconfig = bb.utils.contains('DISTRO_FEATURES', 'ldconfig', True, False, d)
 
     needed = {}
     shlib_provider = oe.package.read_shlib_providers(d)
@@ -1647,6 +1663,8 @@ python package_do_shlibs() {
                     continue
                 if targetos == "darwin" or targetos == "darwin8":
                     darwin_so(file, needed, sonames, renames, pkgver)
+                elif targetos.startswith("mingw"):
+                    mingw_dll(file, needed, sonames, renames, pkgver)
                 elif os.access(file, os.X_OK) or lib_re.match(file):
                     ldconfig = linux_so(file, needed, sonames, renames, pkgver)
                     needs_ldconfig = needs_ldconfig or ldconfig

meta/classes/package_deb.bbclass

@@ -106,7 +106,6 @@ python do_package_deb () {
 
         localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
 
-        bb.data.update_data(localdata)
         basedir = os.path.join(os.path.dirname(root))
 
         pkgoutdir = os.path.join(outdir, localdata.getVar('PACKAGE_ARCH'))
@@ -237,6 +236,10 @@ python do_package_deb () {
         debian_cmp_remap(rsuggests)
         # Deliberately drop version information here, not wanted/supported by deb
         rprovides = dict.fromkeys(bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES") or ""), [])
+        # Remove file paths if any from rprovides, debian does not support custom providers
+        for key in list(rprovides.keys()):
+            if key.startswith('/'):
+                del rprovides[key]
         rprovides = collections.OrderedDict(sorted(rprovides.items(), key=lambda x: x[0]))
         debian_cmp_remap(rprovides)
         rreplaces = bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES") or "")

meta/classes/package_ipk.bbclass

@@ -63,7 +63,6 @@ python do_package_ipk () {
 
         localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
 
-        bb.data.update_data(localdata)
         basedir = os.path.join(os.path.dirname(root))
         arch = localdata.getVar('PACKAGE_ARCH')
 

meta/classes/package_rpm.bbclass

@@ -335,8 +335,6 @@ python write_specfile () {
 
         localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
 
-        bb.data.update_data(localdata)
-
         conffiles = get_conffiles(pkg, d)
         dirfiles = localdata.getVar('DIRFILES')
         if dirfiles is not None:

meta/classes/package_tar.bbclass

@@ -38,7 +38,6 @@ python do_package_tar () {
 
         overrides = localdata.getVar('OVERRIDES', False)
         localdata.setVar('OVERRIDES', '%s:%s' % (overrides, pkg))
-        bb.data.update_data(localdata)
 
         bb.utils.mkdirhier(root)
         basedir = os.path.dirname(root)

meta/classes/patch.bbclass

@@ -12,25 +12,7 @@ inherit terminal
 
 python () {
     if d.getVar('PATCHTOOL') == 'git' and d.getVar('PATCH_COMMIT_FUNCTIONS') == '1':
-        tasks = list(filter(lambda k: d.getVarFlag(k, "task"), d.keys()))
-        extratasks = []
-        def follow_chain(task, endtask, chain=None):
-            if not chain:
-                chain = []
-            chain.append(task)
-            for othertask in tasks:
-                if othertask == task:
-                    continue
-                if task == endtask:
-                    for ctask in chain:
-                        if ctask not in extratasks:
-                            extratasks.append(ctask)
-                else:
-                    deps = d.getVarFlag(othertask, 'deps', False)
-                    if task in deps:
-                        follow_chain(othertask, endtask, chain)
-            chain.pop()
-        follow_chain('do_unpack', 'do_patch')
+        extratasks = bb.build.tasksbetween('do_unpack', 'do_patch', d)
         try:
             extratasks.remove('do_unpack')
         except ValueError:

meta/classes/populate_sdk_ext.bbclass

@@ -11,8 +11,6 @@ TOOLCHAIN_HOST_TASK_task-populate-sdk-ext = " \
 
 TOOLCHAIN_TARGET_TASK_task-populate-sdk-ext = ""
 
-SDK_RDEPENDS_append_task-populate-sdk-ext = " ${SDK_TARGETS}"
-
 SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 
 SDK_EXT = ""
@@ -47,8 +45,10 @@ def get_sdk_install_targets(d, images_only=False):
         sdk_install_targets = d.getVar('SDK_TARGETS')
 
         depd = d.getVar('BB_TASKDEPDATA', False)
+        tasklist = bb.build.tasksbetween('do_image_complete', 'do_build', d)
+        tasklist.remove('do_build')
         for v in depd.values():
-            if v[1] == 'do_image_complete':
+            if v[1] in tasklist:
                 if v[0] not in sdk_install_targets:
                     sdk_install_targets += ' {}'.format(v[0])
 
@@ -253,6 +253,8 @@ python copy_buildsystem () {
 
     # Create local.conf
     builddir = d.getVar('TOPDIR')
+    if derivative and os.path.exists(builddir + '/conf/auto.conf'):
+        shutil.copyfile(builddir + '/conf/auto.conf', baseoutpath + '/conf/auto.conf')
     if derivative:
         shutil.copyfile(builddir + '/conf/local.conf', baseoutpath + '/conf/local.conf')
     else:
@@ -267,8 +269,12 @@ python copy_buildsystem () {
                     env_whitelist_values[varname] = origvalue
                 return origvalue, op, 0, True
         varlist = ['[^#=+ ]*']
+        oldlines = []
+        if os.path.exists(builddir + '/conf/auto.conf'):
+            with open(builddir + '/conf/auto.conf', 'r') as f:
+                oldlines += f.readlines()
         with open(builddir + '/conf/local.conf', 'r') as f:
-            oldlines = f.readlines()
+            oldlines += f.readlines()
         (updated, newlines) = bb.utils.edit_metadata(oldlines, varlist, handle_var)
 
         with open(baseoutpath + '/conf/local.conf', 'w') as f:
@@ -312,7 +318,7 @@ python copy_buildsystem () {
             f.write('BUILDCFG_HEADER = ""\n\n')
 
             # Map gcc-dependent uninative sstate cache for installer usage
-            f.write('SSTATE_MIRRORS = "file://universal/(.*) file://universal-4.9/\\1\\nfile://universal-4.9/(.*) file://universal-4.8/\\1"\n\n')
+            f.write('SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\\1 file://universal-4.9/(.*) file://universal-4.8/\\1"\n\n')
 
             # Allow additional config through sdk-extra.conf
             fn = bb.cookerdata.findConfigFile('sdk-extra.conf', d)
@@ -332,22 +338,6 @@ python copy_buildsystem () {
             f.write('require conf/locked-sigs.inc\n')
             f.write('require conf/unlocked-sigs.inc\n')
 
-    if os.path.exists(builddir + '/conf/auto.conf'):
-        if derivative:
-            shutil.copyfile(builddir + '/conf/auto.conf', baseoutpath + '/conf/auto.conf')
-        else:
-            with open(builddir + '/conf/auto.conf', 'r') as f:
-                oldlines = f.readlines()
-            (updated, newlines) = bb.utils.edit_metadata(oldlines, varlist, handle_var)
-            with open(baseoutpath + '/conf/auto.conf', 'w') as f:
-                f.write('# WARNING: this configuration has been automatically generated and in\n')
-                f.write('# most cases should not be edited. If you need more flexibility than\n')
-                f.write('# this configuration provides, it is strongly suggested that you set\n')
-                f.write('# up a proper instance of the full build system and use that instead.\n\n')
-                for line in newlines:
-                    if line.strip() and not line.startswith('#'):
-                        f.write(line)
-
     # Write a templateconf.cfg
     with open(baseoutpath + '/conf/templateconf.cfg', 'w') as f:
         f.write('meta/conf\n')
@@ -632,7 +622,9 @@ def get_ext_sdk_depends(d):
     deps = d.getVarFlag('do_image_complete', 'deps', False)
     pn = d.getVar('PN')
     deplist = ['%s:%s' % (pn, dep) for dep in deps]
-    for task in ['do_image_complete', 'do_rootfs', 'do_build']:
+    tasklist = bb.build.tasksbetween('do_image_complete', 'do_build', d)
+    tasklist.append('do_rootfs')
+    for task in tasklist:
         deplist.extend((d.getVarFlag(task, 'depends') or '').split())
     return ' '.join(deplist)
 
@@ -647,7 +639,7 @@ python do_sdk_depends() {
 addtask sdk_depends
 
 do_sdk_depends[dirs] = "${WORKDIR}"
-do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)}"
+do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)} meta-extsdk-toolchain:do_populate_sysroot"
 do_sdk_depends[recrdeptask] = "${@d.getVarFlag('do_populate_sdk', 'recrdeptask', False)}"
 do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa do_populate_sysroot do_deploy ${SDK_RECRDEP_TASKS}"
 do_sdk_depends[rdepends] = "${@get_sdk_ext_rdepends(d)}"
@@ -655,7 +647,6 @@ do_sdk_depends[rdepends] = "${@get_sdk_ext_rdepends(d)}"
 def get_sdk_ext_rdepends(d):
     localdata = d.createCopy()
     localdata.appendVar('OVERRIDES', ':task-populate-sdk-ext')
-    bb.data.update_data(localdata)
     return localdata.getVarFlag('do_populate_sdk', 'rdepends')
 
 do_populate_sdk_ext[dirs] = "${@d.getVarFlag('do_populate_sdk', 'dirs', False)}"

meta/classes/ptest.bbclass

@@ -2,7 +2,7 @@ SUMMARY_${PN}-ptest ?= "${SUMMARY} - Package test files"
 DESCRIPTION_${PN}-ptest ?= "${DESCRIPTION}  \
 This package contains a test directory ${PTEST_PATH} for package test purposes."
 
-PTEST_PATH ?= "${libdir}/${PN}/ptest"
+PTEST_PATH ?= "${libdir}/${BPN}/ptest"
 FILES_${PN}-ptest = "${PTEST_PATH}"
 SECTION_${PN}-ptest = "devel"
 ALLOW_EMPTY_${PN}-ptest = "1"

meta/classes/relative_symlinks.bbclass

@@ -0,0 +1,5 @@
+do_install[postfuncs] += "install_relative_symlinks"
+
+python install_relative_symlinks () {
+    oe.path.replace_absolute_symlinks(d.getVar('D'), d)
+}

meta/classes/sanity.bbclass

@@ -3,7 +3,7 @@
 #
 
 SANITY_REQUIRED_UTILITIES ?= "patch diffstat makeinfo git bzip2 tar \
-    gzip gawk chrpath wget cpio perl file"
+    gzip gawk chrpath wget cpio perl file which"
 
 def bblayers_conf_file(d):
     return os.path.join(d.getVar('TOPDIR'), 'conf/bblayers.conf')
@@ -226,7 +226,6 @@ def check_toolchain_tune(data, tune, multilib):
         # Apply the overrides so we can look at the details.
         overrides = localdata.getVar("OVERRIDES", False) + ":virtclass-multilib-" + multilib
         localdata.setVar("OVERRIDES", overrides)
-    bb.data.update_data(localdata)
     bb.debug(2, "Sanity-checking tuning '%s' (%s) features:" % (tune, multilib))
     features = (localdata.getVar("TUNE_FEATURES_tune-%s" % tune) or "").split()
     if not features:
@@ -877,18 +876,16 @@ def check_sanity_everybuild(status, d):
                  'git', 'gitsm', 'hg', 'osc', 'p4', 'svn', \
                  'bzr', 'cvs', 'npm', 'sftp', 'ssh']
     for mirror_var in mirror_vars:
-        mirrors = (d.getVar(mirror_var) or '').replace('\\n', '\n').split('\n')
-        for mirror_entry in mirrors:
-            mirror_entry = mirror_entry.strip()
-            if not mirror_entry:
-                # ignore blank lines
-                continue
+        mirrors = (d.getVar(mirror_var) or '').replace('\\n', ' ').split()
 
-            try:
-                pattern, mirror = mirror_entry.split()
-            except ValueError:
-                bb.warn('Invalid %s: %s, should be 2 members.' % (mirror_var, mirror_entry.strip()))
-                continue
+        # Split into pairs
+        if len(mirrors) % 2 != 0:
+            bb.warn('Invalid mirror variable value for %s: %s, should contain paired members.' % (mirror_var, mirrors.strip()))
+            continue
+        mirrors = list(zip(*[iter(mirrors)]*2))
+
+        for mirror_entry in mirrors:
+            pattern, mirror = mirror_entry
 
             decoded = bb.fetch2.decodeurl(pattern)
             try:

meta/classes/sstate.bbclass

@@ -30,8 +30,6 @@ SSTATE_DUPWHITELIST = "${DEPLOY_DIR_IMAGE}/ ${DEPLOY_DIR}/licenses/ ${DEPLOY_DIR
 SSTATE_DUPWHITELIST += "${STAGING_ETCDIR_NATIVE}/sgml ${STAGING_DATADIR_NATIVE}/sgml"
 # Archive the sources for many architectures in one deploy folder
 SSTATE_DUPWHITELIST += "${DEPLOY_DIR_SRC}"
-# Ignore overlapping README
-SSTATE_DUPWHITELIST += "${DEPLOY_DIR}/sdk/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt"
 
 SSTATE_SCAN_FILES ?= "*.la *-config *_config"
 SSTATE_SCAN_CMD ??= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES").split())}" \) -type f'
@@ -583,29 +581,6 @@ python sstate_hardcode_path () {
 def sstate_package(ss, d):
     import oe.path
 
-    def make_relative_symlink(path, outputpath, d):
-        # Replace out absolute TMPDIR paths in symlinks with relative ones
-        if not os.path.islink(path):
-            return
-        link = os.readlink(path)
-        if not os.path.isabs(link):
-            return
-        if not link.startswith(tmpdir):
-            return
-
-        #base = os.path.relpath(link, os.path.dirname(path))
-
-        depth = outputpath.rpartition(tmpdir)[2].count('/')
-        base = link.partition(tmpdir)[2].strip()
-        while depth > 1:
-            base = "/.." + base
-            depth -= 1
-        base = "." + base
-
-        bb.debug(2, "Replacing absolute path %s with relative path %s for %s" % (link, base, outputpath))
-        os.remove(path)
-        os.symlink(base, path)
-
     tmpdir = d.getVar('TMPDIR')
 
     sstatebuild = d.expand("${WORKDIR}/sstate-build-%s/" % ss['task'])
@@ -619,15 +594,20 @@ def sstate_package(ss, d):
         if d.getVar('SSTATE_SKIP_CREATION') == '1':
             continue
         srcbase = state[0].rstrip("/").rsplit('/', 1)[0]
+        # Find and error for absolute symlinks. We could attempt to relocate but its not
+        # clear where the symlink is relative to in this context. We could add that markup
+        # to sstate tasks but there aren't many of these so better just avoid them entirely.
         for walkroot, dirs, files in os.walk(state[1]):
-            for file in files:
+            for file in files + dirs:
                 srcpath = os.path.join(walkroot, file)
-                dstpath = srcpath.replace(state[1], state[2])
-                make_relative_symlink(srcpath, dstpath, d)
-            for dir in dirs:
-                srcpath = os.path.join(walkroot, dir)
-                dstpath = srcpath.replace(state[1], state[2])
-                make_relative_symlink(srcpath, dstpath, d)
+                if not os.path.islink(srcpath):
+                    continue
+                link = os.readlink(srcpath)
+                if not os.path.isabs(link):
+                    continue
+                if not link.startswith(tmpdir):
+                    continue
+                bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
         os.rename(state[1], sstatebuild + state[0])
 
@@ -663,7 +643,6 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 
     # Copy the data object and override DL_DIR and SRC_URI
     localdata = bb.data.createCopy(d)
-    bb.data.update_data(localdata)
 
     dldir = localdata.expand("${SSTATE_DIR}")
     bb.utils.mkdirhier(dldir)
@@ -820,7 +799,6 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
     if mirrors:
         # Copy the data object and override DL_DIR and SRC_URI
         localdata = bb.data.createCopy(d)
-        bb.data.update_data(localdata)
 
         dldir = localdata.expand("${SSTATE_DIR}")
         localdata.delVar('MIRRORS')