Commit Graph

5307 Commits

Author SHA1 Message Date
Patrick Ohly
5df6796d1c bitbake: ConfHandler.py: allow require or include with multiple parameters
"inherit" already allows inheriting more than one class in a single
statement. The same also makes sense for "include" and "require",
because then one can generate a list of files to be included
dynamically also for the case that more than one file needs to be
included.

(Bitbake rev: 8d0a76f5a595dddf16b7268bae2c00ef5f568316)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 10:47:09 +01:00
Patrick Ohly
6dafbaeff4 bitbake: ConfHandler.py: allow require or include without parameter
Writing .bbappends that only have an effect when some configuration
variable like DISTRO_FEATURES is changed becomes easier when allowing
"include" or "require" without a parameter. The same was already
allowed for "inherit".

Then one can write in a .bbappend:

  require ${@bb.utils.contains('DISTRO_FEATURES', 'foo', 'bar.inc', '', d)}

(Bitbake rev: 8b39c6361758b96fce50a53a6dba8008cd7e6433)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 10:47:09 +01:00
Richard Purdie
7ac2d8dce2 bitbake: cookerdata: Implement LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES
Currently, someone trying a master branch of any given layer may
or may not find it works with the version of OE-Core that they
have picked.

This patchset introduces LAYERSERIES_CORENAMES and
LAYERSERIES_COMPAT_<layername> with the intention of allowing layers
to indicate which versions of the core project they're compatible with.

In general the master branch will use the next release codename as
the value of LAYERSERIES_CORENAMES, meaning that the waterfall model
of layer releases is supported with layers updating their
LAYERSERIES_COMPAT_ values as they branch for release.

Both variables support multiple strings and a given layer is accepted
if there is overlap of any one value. This means a layer can be compatible
with multiple core versions.

Setting LAYERSERIES_COMPAT_ will be required by the Yocto Project
Compatible v2 standard but the system will not error if its not set at this
point.

This should make it clear when a given layer is unmaintained and untested
with new releaes of OE-Core, a common user pain point.

(Bitbake rev: 92c49b9b1a16dfd35444db8143bd4cae4cda70cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Ming Liu
f3e06c4749 bitbake: bitbake-user-manual-metadata.xml: fix a typo
'buildCompleted' should be 'BuildCompleted'.

(Bitbake rev: 9bd96cf938f746ace8d157f6b1b6c2550798085f)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
55c4781cde bitbake: knotty: prevent extra logger from being enabled for tinfoil
tinfoil sets up its own logger by default, but if and when we initialise
the UI (by default knotty) will also set one up, leading to duplicated
messages specifically from tasks. To avoid this, rather than adding some
kind of parameter, just check if there is already a logger outputting to
stdout/stderr and if so, skip adding our own.

Part of the fix for [YOCTO #11275].

(Bitbake rev: 66d866745f35468d1540a793d07e3a401298b84b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
d44d5a2e79 bitbake: tinfoil: fix duplication of log messages
Adding an additional logger in setup_bitbake() interacts poorly with the
logger we have added by default in tinfoil's constructor, with the
result that messages may be doubled or even tripled in tinfoil-using
scripts. Disable adding this one when calling setup_bitbake() from
tinfoil to avoid this problem.

Part of the fix for [YOCTO #11275].

(Bitbake rev: 8a5bae76f91f2411187c638a42fa3c762052cf11)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
833520d2fb bitbake: command / cooker: drop Hob support commands and functions
Drop a number of the commands and support functions that were created
solely to support functionality in the now-removed Hob UI. In most cases
we now have this functionality elsewhere e.g. to modify config files and
recipes we have bb.utils.edit_metadata_file() and friends, and in OE we
have oe.recipeutils.patch_recipe_file() and friends which build on top of
the former. Additionally, some of it represented pretty egregious
incursion of OE metadata-specific references into BitBake code.

For now I have left in the find*File functions and commands as they are
relatively generic and possibly still useful, but they might be removed
in future.

(Bitbake rev: a322f13183c66a28d93cc4bc3d839d95fc1d90f4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
3d4c64fb9f bitbake: cooker: add BB_CMDLINE to enable access to UI command line with memres
In OpenEmbedded's buildhistory class we want access to the bitbake
command line that launched the build, and up to now we were simply using
sys.argv from within the event handler to get that. Unfortunately that
doesn't work in memory resident mode, since the event handler is
naturally executing within the server and thus will give you the command
that launched the bitbake server which is much less interesting. Add a
dynamic variable BB_CMDLINE to provide access to this, set from sys.argv
within the UI process in updateToServer().

(Note that BB_CMDLINE isn't currently passed through to the worker, so
this is only really readable from event handlers plus any explicit
getVariable calls - in theory an observe-only UI could read it for
example.)

Part of the fix for [YOCTO #11634].

(Bitbake rev: 8ad31a3eff5cfcb7b1c462578a582dafcbc426d4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
7520e22648 bitbake: cooker: set config valid flags in initConfigurationData()
If we set these flags here then we can not only tidy up some of the code
that calls initConfigurationData() (and reset() which simply calls the
former), we also avoid calling initConfigurationData() an extra time
unnecessarily during startup (since setting baseconfig_valid = False in
updateConfigOpts() also resulted in initConfigurationData() being
called from updateCacheSync() at the end of the command.)

(Bitbake rev: 74d2ef99a920aa9e70470ba8e600ea973a3c672a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
67f676f704 bitbake: cooker: fix file changes not triggering task re-execution with memres
If you build a recipe, modify one of the files referred to in SRC_URI,
then build it again, it should re-run do_fetch and the tasks that depend
upon it to incorporate the modified file. With memory resident mode this
was not working unless you restarted the server, because the mtime of
the file was cached and thus we never checked the actual file. Clear out
the mtime caches in buildTargets() to ensure this isn't an issue. I did
a minor refactoring to break this out to its own function since we were
already clearing out the build mtime cache here.

(The test I used for this was . ./oe-init-build-env-memres,
bitbake -c fetch mdadm, modify the comments at the top of one of the
patches referred to in the recipe, and then bitbake -c fetch mdadm again
and it should re-execute the fetch task ("... 0 didn't need to be
re-run".)

Fixes [YOCTO #10732].

(Bitbake rev: fe4f3487368bc35043a1ba464e1c69d2c9cbefc6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
453835679b bitbake: cookerdata: fix variable history not showing in bitbake -e with memres
CookerConfiguration sets the "tracking" option to True when -e is
specified in order to have history tracking enabled in the datastore
so that we can show variable history (which isn't enabled by default for
performance reasons), however in memory resident mode this wasn't doing
anything because it was happening on the UI side only. We do have a
mechanism for updating the cooker configuration in the server, but the
tracking option wasn't being included in the list of options we updated,
so we just need to add this option to fix the issue.

Fixes [YOCTO #10730].

(Bitbake rev: 57d4977555cf892b15dd0302dfe261fe37d49327)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:38 +01:00
Patrick Ohly
337bea6573 bitbake: cookerdata: Add support for BBFILES_DYNAMIC
BBFILES_DYNAMIC can be used to activate content only when some other
layers are present. The other layers are identified by the collections
that they define.

The main use case is to avoid .bbappends without the corresponding .bb
file in layers that want to modify other layers via .bbappends without
introducing a hard dependency on those other layers. .bb files could
also be handled via BBFILES_DYNAMIC.

Entries in BBFILES_DYNAMIC must have the form <collection
name>:<filename pattern>. Example usage:
 BBFILES_DYNAMIC += " \
     clang-layer:${LAYERDIR}/bbappends/meta-clang/*/*/*.bbappend \
     core:${LAYERDIR}/bbappends/openembedded-core/meta/*/*/*.bbappend \
 "

Parsing is aborted when invalid entries are found with an error
message like this:

 ERROR: BBFILES_DYNAMIC entries must be of the form <collection name>:<filename pattern>, not:
     /work/my-layer/bbappends/meta-security-isafw/*/*/*.bbappend
     /work/my-layer/bbappends/openembedded-core/meta/*/*/*.bbappend

Based on a patch by Richard Purdie.

(Bitbake rev: 04f8bd50aa04b12cf91dd6a3154527ad2c24695c)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-09 17:13:45 +01:00
Chang Rebecca Swee Fun
576821ea0a bitbake: bitbake-layers: check layer dependencies before adding
In the original implementation, "bitbake-layers add-layers <layer>"
succeeded without error checking. This will further introduce
failures in recipe parsing only when "bitbake" command is executed.
Adding a meta layer without its dependency layer(s) should failed
and exit the process gracefully.

Added extra argument "-F" to force add a layer without checking
layer dependency.

[YOCTO #10913]

(Bitbake rev: 705ab252e631903e6d2e46202b419a9e8adcd861)

Signed-off-by: Phoong Stanley Cheong Kwan <stanley.cheong.kwan.phoong@intel.com>
Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:52:51 +01:00
Andy Voltz
644fbb9aca bitbake: tinfoil: allow extra features to be passed to prepare
(Bitbake rev: 2e35de1f19dc73a61a18a3eb186efede078d597d)

Signed-off-by: Andy Voltz <andy.voltz@timesys.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Andy Voltz
44b3eb65d9 bitbake: cooker: fix undefined dep in generatePkgDepTreeData
The loop for populating the depends of the tree data should use dep
instead of item.

(Bitbake rev: 37c298b27e3f72b21513124237cfeffb22c0e2f0)

Signed-off-by: Andy Voltz <andy.voltz@timesys.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Mikko Rapeli
bc9c714e8b bitbake: bitbake: Fix return value checks from subprocess.call()'s
Python function subprocess.call() returns the return value of the
executed process. If return values are not checked, errors may
go unnoticed and bad things can happen.

Change all callers of subprocess.call() which do not check for
the return value to use subprocess.check_call() which raises
CalledProcessError if the subprocess returns with non-zero value.

https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module

All users of the function were found with:

$ git grep "subprocess\.call" | \
  egrep -v 'if.*subprocess\.call|=\ +subprocess\.call|return.*subprocess\.call'

Tested similar patch on top of yocto jethro. Only compile tested
core-image-minimal on poky master branch.

(Bitbake rev: d2cf67bcaf001acb6be8fc5884fb450649849847)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Ola x Nilsson
ca68d71ddc bitbake: ConfHandler: Require whitespace between export and variable name
(Bitbake rev: 22bb7c9270f02ddae72e13d849375feee5f4a98b)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Richard Purdie
5f711bc63e bitbake: siggen: Make calc_taskhash match get_taskhash for file checksums
The code in these two functions is meant to be equivlanet in behaviour
but isn't. Add in code to ensure files that don't exist are handled
consistently by both functions. Users did report being able to generate
tracebacks otherwise.

(Bitbake rev: c11b4832879354b387a8c9c6a3096f2d621afeac)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Christopher Larson
35ecff3cf0 bitbake: fetch/git: add support for removing arbitrary revs for shallow
In certain cases, it's valuable to be able to exert more control over what
history is removed, beyond srcrev+depth. As one example, you can remove most
of the upstream kernel history from a kernel repository, keeping predominently
the non-publically-accessible content. If the repository is private, the
history in that repo couldn't be restored via `git fetch --unshallow`, but
upstream history could be.

Example usage:

    # Remove only these revs, not at a particular depth
    BB_GIT_SHALLOW_DEPTH_pn-linux-foo = "0"
    BB_GIT_SHALLOW_REVS_pn-linux-foo = "v4.1"

(Bitbake rev: 97f856f0455d014ea34c28b1c25f09e13cdc851b)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
30485b2b1a bitbake: fetch/gitannex: add support for shallow mirror tarballs
When we're building from a shallow mirror tarball, we don't want to do
anything with ud.clonedir, as it's not being used when we unpack. As such,
disable updating annex in that case. Also include annex files in the shallow
tarball.

(Bitbake rev: ca0dd3c95502b22c369fbf37f915f45e02c06887)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
f5308b8cc1 bitbake: fetch/gitsm: add support for shallow mirror tarballs
When we're building from a shallow mirror tarball, we don't want to do
anything with ud.clonedir, as it's not being used when we unpack. As such,
disable updating the submodules in that case. Also include the repositories in
.git/modules in the shallow tarball. It does not actually make the submodule
repositories shallow at this time.

(Bitbake rev: 6c0613f1f2f9d4f009545f82a9173e80396f9d34)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
8144f6f408 bitbake: fetch/git: add support for keeping extra refs for shallow
By default, all unused refs (branches & tags) are removed from the repository,
as shallow processing scales with the number of refs it has to process. Add
the ability to explicitly specify additional refs to keep. This is
particularly useful for recipes with custom checkout processes, or whose
git-based versioning requires a tag be available (i.e. for `git describe
--tags`). The new `BB_GIT_SHALLOW_EXTRA_REFS` variable is a space-separated
list of refs, fully specified, and support wildcards.

Example usages:

    BB_GIT_SHALLOW_EXTRA_REFS = "refs/tags/v1.0"
    BB_GIT_SHALLOW_EXTRA_REFS += "refs/heads/*"

(Bitbake rev: 1771934cd9f8b5847c6fcae0a906fb99d6b0db16)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
bf87c5cd19 bitbake: fetch/git: support per-branch/per-url depths for shallow
Allow the user to explicitly adjust the depth for named urls/branches. The
un-suffixed BB_GIT_SHALLOW_DEPTH is used as the default.

Example usage:

    BB_GIT_SHALLOW_DEPTH = "1"
    BB_GIT_SHALLOW_DEPTH_doc = "0"
    BB_GIT_SHALLOW_DEPTH_meta = "0"

(Bitbake rev: 9dfc517e5bcc6dd203a0ad685cc884676d2984c4)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
27d56982c7 bitbake: fetch/git: add support for shallow mirror tarballs
This adds support to the git fetcher for fetching, using, and generating
mirror tarballs of shallow git repositories. The external git-make-shallow
script is used for shallow mirror tarball creation.

This implements support for shallow mirror tarballs, not shallow clones.
Supporting shallow clones directly is not really doable for us, as we'd need
to hardcode the depth between branch HEAD and the SRCREV, and that depth would
change as the branch is updated.

When BB_GIT_SHALLOW is enabled, we will always attempt to fetch a shallow
mirror tarball. If the shallow mirror tarball cannot be fetched, it will try
to fetch the full mirror tarball and use that. If a shallow tarball is to be
used, it will be unpacked directly at `do_unpack` time, rather than extracting
it to DL_DIR at `do_fetch` time and cloning from there, to keep things simple.
There's no value in keeping a shallow repository in DL_DIR, and dealing with
the state for when to convert the clonedir to/from shallow is not worthwhile.

To clarify when shallow is used vs a real repository, a current clone is
preferred to either tarball, a shallow tarball is preferred to an out of date
clone, and a missing clone will use either tarball (attempting the shallow one
first).

All referenced branches are truncated to SRCREV (that is, commits *after*
SRCREV but before HEAD are removed) to further shrink the repository. By
default, the shallow construction process removes all unused refs
(branches/tags) from the repository, other than those referenced by the URL.

Example usage:

    BB_GIT_SHALLOW ?= "1"

    # Keep only the top commit
    BB_GIT_SHALLOW_DEPTH ?= "1"

    # This defaults to enabled if both BB_GIT_SHALLOW and
    # BB_GENERATE_MIRROR_TARBALLS are enabled
    BB_GENERATE_SHALLOW_TARBALLS ?= "1"

(Bitbake rev: 5ed7d85fda7c671be10ec24d7981b87a7d0d3366)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
2a60c40637 bitbake: git-make-shallow: add script to make a git repo shallow
This script will be used by the git fetcher to create shallow mirror tarballs.

    usage: git-make-shallow [-h] [--ref REF] [--shrink] REVISION [REVISION ...]

    Remove the history of the specified revisions, then optionally filter the
    available refs to those specified.

    positional arguments:
      REVISION           a git revision/commit

    optional arguments:
      -h, --help         show this help message and exit
      --ref REF, -r REF  remove all but the specified refs (cumulative)
      --shrink, -s       shrink the git repository by repacking and pruning

While git does provide the ability to clone at a specific depth, and fetch all
remote refs at a particular depth, the depth is across all branches/tags, and
doesn't provide the flexibility we need, hence this script.

Refs (branches+tags) can be filtered, as the process of history removal scales
up rapidly with the number of refs. Even the existing `git fetch --depth=` is
extremely slow on an upstream kernel repository with all the branches and tags
kept.

This uses the same underlying mechanism to implement the history removal which
git itself uses (.git/shallow), and the results, when configured similarly, are
in line with the results git itself produces with `fetch --depth`.

(Bitbake rev: 0254020f0e1911c0eaf99111b91828d2a74a4ee1)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
ab4e578b86 bitbake: fetch: support multiple mirror tarball filenames
Remove ud.mirrortarball in favor of ud.mirrortarballs. Each tarball will be
attempted, in order, and the first available will be used. This is needed for
git shallow mirror tarball support, as we want to be able to use either
a shallow or full mirror tarball.

(Bitbake rev: 02eebee6709e57b523862257f75929e64f16d6b0)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Scott Rifenbark
31ec7fa01d bitbake: bitbake-user-manual: Updated INHERIT descriptions
Fixes [YOCTO #11498]

Enhanced the glossary description for the INHERIT variable to give more
detail and to cross-reference the reader back into the Chapter 3
section describing its use.

(Bitbake rev: 362f6044fcaafe51ab4377af8f2606165b112717)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-23 17:45:37 +01:00
Richard Purdie
8238165be5 bitbake: data_smart: Don't cache/process capitalised overrides
Bitbake now only processes overrides which are lowercase since
this allows variables like SRC_URI not to pollute the cache.

There was a corner case where XXX_append_SomeThing was still being
processed (yet XXX_append_SomeThing_SomeOtherThing would not be).

This patch ensures we're consistent and only process lowercase
_append/_prepend and _remove operators too.

(Bitbake rev: 6eb56624e6d8dc1944e559b4f6584bfe66f566ba)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-18 13:19:55 +01:00
Paul Eggleton
3dea90e8b7 bitbake: lib/bb/fetch2: fix typo in conflicting revision message
spcify -> specify

(Bitbake rev: c46f183b3ea94789dd83cb280a1bf5735189c9bc)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-15 16:21:21 +01:00
Paul Eggleton
5cba1620e4 bitbake: lib/bb/COW.py: fix typo in Tim Ansell's name
See AUTHORS for reference.

(Bitbake rev: a6345899ffeb595ca287b295945713eb95032124)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-15 16:21:20 +01:00
David Reyna
c519dc1cd4 bitbake: toaster: move release lookup from morty to pyro
Move the release name lookup into the layer index
from 'morty' to the 'pyro'.

Move the bitbake branch from 1.32 to 1.34.

[YOCTO #11377]

(Bitbake rev: 21d963149b5d97452420230a252101115b708d85)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-20 07:59:40 +01:00
Paul Eggleton
d45f5be977 bitbake: lib/bb/command: fix overrides in remote datastores where there is no default
If you have, for example, no value set for a variable VARIABLE and a
then VARIABLE_pn-something = "value" and then you parse something.bb,
you expect getVar('VARIABLE') on the resulting datastore to return
"value", but the code here assumed that if the variable wasn't set
without overrides then we didn't need to return the overridedata and
thus we didn't get the overridden value.

In OE this affected the ability to get RECIPE_MAINTAINER for a recipe
in a script using tinfoil (since this is only set from an inc file with
_pn- overrides for each recipe, and no default is set).

(Bitbake rev: b3d2c9917c5fd8278878328794daa107ddf79b64)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-20 07:59:40 +01:00
Richard Purdie
1423508b29 bitbake: bitbake: Update version to 1.34.0 for stable release
(Bitbake rev: b4da94a368c6c44c6e5b6e6e9a1e041ed84b4554)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19 10:42:12 +01:00
Juro Bystricky
00d49a05a6 bitbake: command.py: multiconfig support for findBestProvider
In a multiconfig environment, a tinfoil call such as

    tinfoil.parse_recipe("multiconfig:arduino-101-sss:gcc")

can fail with an error such as:

  File "/data/master/poky/bitbake/lib/bb/tinfoil.py", line 373, in get_recipe_file
    raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
bb.providers.NoProvider: Unable to find any recipe file matching "multiconfig:arduino-101-sss:gcc"

The culprit is findBestProvider, which does not handle multiconfig.
This patch fixes the error and in the case mentioned above the tinfoil call returns:

  "multiconfig:arduino-101-sss:/data/master/poky/meta/recipes-devtools/gcc/gcc_6.3.bb"

[YOCTO#11210]

(Bitbake rev: e9c03fbfd7b057b28645affa263cb4aebfa24b04)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19 10:20:24 +01:00
Zhixiong Chi
bcab5e12c1 bitbake: bitbake-layers: add signal hander to avoid exception
Fixed:
bitbake-layers show-recipes | less
press "q" to exit

There will be a Broken pipe error output as follows:
"BrokenPipeError: [Errno 32] Broken pipe"

(Bitbake rev: 4fca9a07f2d6b0544977112672b786982d7bb8f2)

Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:37 +01:00
Nathan Rossi
bf0fc2da66 bitbake: lib/bb/{data, siggen}: Don't warn on SkipRecipe exceptions
Recipes/variables that raise a SkipRecipe exception are intentionally
skipped, and should not generate warnings.

[YOCTO #11319]

(Bitbake rev: edf6e6094a9f7ad4b2ba06eef8fd34756edbedce)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:16:05 +01:00
Richard Purdie
000b7d3d1c bitbake: data_smart: Ensure _remove operations on newly set variables are cleared
We clear append/prepend on newly set variables, we should also clear
remove operations. If we don't do this, there is no way we can actually
delete a remove operation. Bitbake internally uses parsing=True to avoid
these side effects when making its own internal calls.

Also add a testcase to bitbake-selftest to ensure we remain consistent going
forward from here.

(Bitbake rev: 3a319f079d699c870d8531e051ab65e6278d1fa5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:58 +01:00
David Reyna
b06f7cbb94 bitbake: toaster: fix SDK artifact capture
Use the TaskArtifacts event to scan the SDK and ESDK manifests
to cleanly collect the respective artifact files.

The previous method was broken when the SDK file deployment moved
from the do_populate_sdk[_ext] tasks to their sstate tasks. That
method is disabled (but not yet removed) in preparation for the
rest of refactor work for the parent #10283 work.

[YOCTO #10850]

(Bitbake rev: 1360d7b847cc01031edb2f4b289fac9560d72fa7)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:05:09 +01:00
Paul Eggleton
f187553b6d bitbake: bitbake-diffsigs: fix traceback with no arguments
In the move over to argparse we've made the two signature file arguments
optional and thus if -t is not in use we need to explicitly check if at
least one signature file has been specified - and if not, show an error
and the command-line help.

(Bitbake rev: 9011366689b26305281fcb2d412dcacece917e18)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:05:09 +01:00
Markus Lehtonen
fc6370626e bitbake: cache: prevent division-by-zero in load_cahefile()
[YOCTO 11315]

(Bitbake rev: 227c5acd4b40154bc61202e7bb67a13818a7d727)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:05:09 +01:00
David Reyna
2a40a0587e bitbake: toaster: fix add layer button for Machines
Fix a missing close quote in the layer add button, and
use the layer index for the xhr_layer call.

[YOCTO #11213]

(Bitbake rev: 54d61e7c64c97799de85b3563119f0f5d051675e)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:05:09 +01:00
David Reyna
d8be835e28 bitbake: toaster: Toaster filters are broken
A syntax fix inadvertantly broke the Toaster filter feature.

[YOCTO #11317]

(Bitbake rev: 29c9fa31d0f03553e24391568d2a5cb588595420)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:05:09 +01:00
Paul Eggleton
b372e85d9b bitbake: bitbake-diffsigs: colourise output
If the output is a TTY, add colour to the output in order to make it
easier to read. At the moment this is fairly basic, just add colour to
the "titles" of each change and to the diff output.

I tried to introduce this without changing the code too much - rather
than moving everything over to the new python formatting style, I've
introduced a color_format() function which takes care of the colour
formatting, either accepting additional format arguments or
alternatively leaving the caller to use the old-style formatting (%) to
insert values.

(Bitbake rev: 04a023c8fdea1e1812fcdcaf00345aab59f9abe1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
5f7bf1f66d bitbake: lib/bb/siggen: show word-diff for single-line values containing spaces
If a variable value has changed and either the new or old value contains
spaces, a word diff should be appropriate and may be a bit more readable.
Import the "simplediff" module and use it to show a word diff (in the
style of GNU wdiff and git diff --word-diff).

Also use a similar style diff to show changes in the runtaskhashes list.
I didn't use an actual word-diff here since it's a little different - we
can be sure that the list is a list and not simply a free-format string.

(Bitbake rev: 20db6b6553c80e18afc4f43dc2495435f7477822)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
5d8b89fc0b bitbake: lib/bb/siggen: add collapsed mode to compare_sigfiles()
If we just want to drill down to the actual differences then we don't
need to see certain things in the output, e.g. basehash changing or the
signature of dependent tasks. This will be used for comparing signatures
within buildhistory-diff in OE-Core; the default mode as used by
bitbake-diffsigs and bitbake -S printdiff remains unchanged for the
moment.

(Bitbake rev: 6543a59b1ebd3194a7c6421cffc66ebe31a67c62)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
f79a969c3a bitbake: bitbake-diffsigs: add an option to find and compare specific signatures
With the -t option which recurses to find the ultimate cause of a
signature change, it was hardcoded to take the last two executions of
the specified task. On the other hand, if you have two specific task
hashes (say from bitbake output, or some other tool) then you'll want to
pick those, so provide an option to specify those as well. (Note, the
new -s option needs to be specified alongside -t rather than instead of
it.)

(Bitbake rev: d9813b1a4223cf8dc80cab90e467ddf4bf8d1078)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
e0eda80371 bitbake: bitbake-diffsigs: change to use argparse
Argparse is a bit easier to deal with than optparse, and since we're
about to add some options, migrate this script over.

(Bitbake rev: 7f130e0b5ce6cfc6b35176465f260092cd3b3d64)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
8c015e45ff bitbake: lib/bb/siggen: don't show unchanged runtaskdeps list
If the runtaskdeps list hasn't actually changed (but the signatures of
some of the tasks did) then it doesn't make sense to print out the old
and new lists as they are both the same and may be very long, e.g. for
do_rootfs in OE.

(Bitbake rev: cb170543605288b3e8badfac3a54c588f4c95413)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
445e622b23 bitbake: lib/bb/siggen: show a diff when dumping changes to multi-line values
When dumping changes to signatures e.g. output of bitbake -s printdiff,
if for example a function has changed, it's much more readable to see a
unified diff of the changes rather than just printing the old function
followed by the new function, so use difflib to do that.

Note: I elected to keep to one item in the returned list per change,
rather than one line per line of output, so that the caller can still
look at changes individually if needed. Thus I've added some handling to
bitbake-diffsigs to split the change into lines so that each line is
displayed indented.

(Bitbake rev: 4d254ae63a35231c98e3f73f669b040ed1144042)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
d88a4ade7e bitbake: bitbake-diffsigs: drop naive logic for removing duplicate files
This logic doesn't work in practice, certainly not with current versions
where sigdata files are preserved in the stamps directory and therefore
there will often be multiple sigdata files - you can now easily get
files for the same signature from sstate and the stamps directory with the
result that bitbake-diffsigs reports nothing has changed. Instead, let's
change the find_siginfo function in OE-Core to simply not return
duplicates so we don't have to filter them out here.

(Bitbake rev: f0d7ab259d8ef95643e7229474b7850608aa4426)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00