Commit Graph

5033 Commits

Author SHA1 Message Date
Richard Purdie
4434777705 bitbake: tests/fetch: Update gnome.org urls after upstream changes
(Bitbake rev: 43ff74d675baed13fc17e1d12c0e013b16ba249b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-29 10:43:23 +01:00
Paulo Neves
a8368651ff bitbake: fetcher: Fixed remote removal not throwing exception.
Before this fix it is assumed that the removal of the
remote can only fail because there is not remote to remove. This
is a false assumption. Example error which would be ignored:

git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output:
Note: A branch outside the refs/remotes/ hierarchy was not removed;
to delete it, use:
  git branch -d master
error: could not lock config file config
error: Could not remove config section 'remote.origin'

Due to the masking of this error a stranger error will be
presented to the user, because this time we do not mask the
exception:

git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output:
fatal: remote origin already exists.

The most likely reason that the remote cannot be removed nor
modified is that the DL_DIR/git2 does not have permissions
compatible with the user running bitbake.

This commit fixes:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728

(Bitbake rev: 9c86c582a10c9b23abad7d34b6cbf12f7086294d)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
David Reyna
4c41fc3f6e bitbake: toaster: enable project import and merged Toaster settings
Enable the ability to select an existing build directory into Toaster.
This  opens to the user the backend features of 12823, for command line
user compatibility.

Enable the ability to select saving Toaster settings in the regular
"bblayers.conf" and "local.conf" instead of the default
"toaster_bblayers.conf" and "toaster.conf". This opens to the user the
backend features of 12821, for command line user compatibility.

[YOCTO #12902]

(Bitbake rev: 8ce51fbd92ab42365a38e0c1c260bb4979377a89)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Awais Belal
1035d9decb bitbake: toaster/layerdetails.js: don't hide local layer info
The local layer info (provided through custom fixtures) should
not be hidden. It is better to handle it in the same manner
as an imported layer, otherwise the layer path and dependency
info is not shown. The layer editing fields are handled in the
html side of things appropriately so this does not harm that
implementation.

[YOCTO #12891]

(Bitbake rev: aa79967920b2617ce007f66c89f3343f1a7f34b4)

Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Awais Belal
876450442b bitbake: toaster/models.py: allow local paths for custom recipe's base
In a case where the layer source is local only and the recipe
is not yet built, we can search for the path with layer's
local_source_dir, and if available that should be used rather
than just skipping the scenario.

[YOCTO #12891]

(Bitbake rev: 59f3e04122fca43835603779ac2d201464fbaebf)

Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Awais Belal
3b7ec34371 bitbake: toaster/checksettings: allow CUSTOM_XML_ONLY setting through env
This change allows the CUSTOM_XML_ONLY toaster setting to be
provided through the environment so the user can do this without
mingling with the settings.xml, for scenarios where modifying
settings.xml is not achievable.

[YOCTO #12891]

(Bitbake rev: 55333f1c3ded1c53120e6cb32b440cc707521e2b)

Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Awais Belal
bc344a4285 bitbake: toastergui/newproject.html: fix release divs
The release drop down divs are not being closed
appropriately, which showed adverse reactions on
the UI that aligned the "Create project" button
with the left edge of the screen without any
margins. This fixes these divs which in turn
aligns the button appropriately.

[YOCTO #12891]

(Bitbake rev: a6e20bddddd083fbecb5ed8d5f4824da640fdd44)

Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Awais Belal
215c187712 bitbake: toaster/widgets.py: avoid divide by zero issues
There can be cases where the variables being used
to divide in build percentage expressions can be
zero. For example, a setup consisting of only local
repos will have repos_to_clone=0 and will generate
a divide by zero scenario.
Fix this by checking the divisor in such cases.

[YOCTO #12891]

(Bitbake rev: 679c70aa32d23e9247f8a68efcb579ad733af84b)

Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Awais Belal
af52215712 bitbake: toaster: use a more flexible way to find bitbake
The current mechanism for finding the bitbake binary
assumes a directory structure which is identical to
poky, where oe-core's meta and bitbake directories are
at the same level. There can be a case where bitbake
is used from elsewhere and in such cases the above
mentioned assumption fails to hold, whereas this is
totally allowed by the oe-init-build-env script which
can take bitbakedir as an argument.
The better approach is to allow bitbake to be derived
from PATH, while keeping the older mechanism in place so
it can be removed after tests are done in various
environments. This makes more sense as toaster has
also been launched from the same bitbake instance
that is the one in PATH.

[YOCTO #12891]

(Bitbake rev: 365d8d94ae3e4e0f95e0806dbcb7c77c20a55d2d)

Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Awais Belal
95e37ba260 bitbake: toaster: allow pokydirname to be evaluated when all layers are local
Toaster depends on pokydirname for identifying the location of
the oe-init-build-env script (and there might be other purposes
in the future). The problem with current approach is that it
only checks/sets the variable with git based repos, whereas
toaster provides mechanisms to allow having layers that are all
locally available. The evaluation of the variable fails in such
scenarios, so use a more flexible mechanism in this case and
try to locate poky in the local layers as well, if not already
set.

[YOCTO #12891]

(Bitbake rev: 6c3c196b28603591371ec7e62871fbb4296f2c71)

Signed-off-by: Awais Belal <awais.belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Paul Eggleton
87d3a9685d bitbake: parse/ast: ensure saved event handlers really do get restored
In finalize() we save event handlers, register the ones relevant to the
recipe being finalised, trigger events, and then restore the handlers so
that one recipe's custom handlers (actually implemented within a class
inherited by the recipe) do not affect other recipes. However, if an
exception occurs during parsing, the saved handlers were not being
restored. Use a try...finally block to ensure that the handlers are
always restored.

This issue became apparent since in OpenEmbedded-Core we have recently
introduced a find_intercepts() handler for the
bb.event.RecipePreFinalise event in image-postinst-intercepts.bbclass
that images and old-style SDK recipes will end up inheriting. So far it
doesn't seem that the the error has manifested itself in normal builds,
but when parsing OE-Core recipes in the OE layer index it has:
core-image-rt-* image recipes were parsed which in the default
configuration raise SkipRecipe. The next non-image recipe that is parsed
will trigger a real exception, because the find_intercepts() handler is
still registered and gets fired, but in the context of the new recipe
the POSTINST_INTERCEPTS_PATHS variable is not set, and the code in
find_intercepts() is written with the reasonable assumption that that
isn't possible given that the class itself sets a default, and thus it
fails.

(Bitbake rev: e5f1f8fa201774e0c3c554d59b277baa2128708f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-24 07:53:14 +01:00
Richard Purdie
d2ef693c9a bitbake: bitbake: Bump version 1.39.0 -> 1.39.1
(Bitbake rev: 49c3fd2489867c09dec6919a25b53d935a8204bb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23 09:02:54 +01:00
Joshua Watt
cbe2d3cb0a bitbake: daemonize: Flush stdio on exit
In spite of a comment suggesting otherwise, os._exit() does not flush
buffered output from file descriptors before exiting the process like
os.exit() does. This means that any un-flushed output is lost in the
daemon process, in particular the traceback from any thrown exceptions,
making debugging exceptions in the daemon quite difficult.

The solution is to flush stdout and stderr before exiting.

(Bitbake rev: fcc8e1ff53696f78dd64b4ee32f3c433b7a47df0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23 08:50:23 +01:00
Karsten Strand
b6bc5f068a bitbake: toaster: Fix comparison in recipe template
Use == instead of = when comparing task outcome to OUTCOME_FAILED.

Prior to this fix the recipe template would cause a TemplateSyntaxError
exception.

(Bitbake rev: a53ffec4ed3d0f9221bca398e20e8f480fb2b325)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23 08:50:23 +01:00
David Reyna
08fbdc02e3 bitbake: Toaster: Implement the project-specific feature and releated enhancements and defects.
Here is the primary driving enhancement:

*  Bug 12785 - Support Project Specific configuration for external
   tools (e.g. ISS, Eclipse)

  -  Isolated project-specific configuration page (full Toaster context
     hidden)
  -  Support for new project, reconfigure existing project, and import
     existing command line project
  -  Ability to define variables (e.g. image recipe) and pass them back
     to external GUI
  -  Ability to execute the cloning phase, so that external GUI receive
     a buildable project
  -  Ability to call back to the external GUI when updates are completed
     and ready
  -  Compatibility of above projects with the normal full Toaster interface
  -  Ability to pass to a 'complete' or 'cancel' web page so that the
     external GUI can immediately stop that Toaster instance, and not
     leave dangling servers nor edit sessions open

Here are the supporting enhancements, where at least the
back end is implemented:

*  Bug 12821 - Make Toaster conf changes compatible with command line usage
*  Bug 12822 - Support importing user changes to conf files into Toaster
*  Bug 12823 - Support importing user build directories into Toaster
*  Bug 12824 - Scan imported layers for content so that they are
               immediately available
*  Bug 12825 - show layer clone item in progress bar

Here are defects fixed:

*  Bug 12817 - builddelete.py requires explicit 'add_arguments'
*  Bug 12818 - Remove orphaned imported layers when project is deleted
*  Bug 12826 - fix imported layer management
*  Bug 12819 - build using selected bitbake env, not Toaster's env
*  Bug 12820 - Toaster randomizes the layer order in toaster_bblayers.conf

[YOCTO #12785]

(Bitbake rev: 985d6cec290bdd80998a63483561a73c75d82d65)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-20 10:20:51 +01:00
Rasmus Villemoes
d4011ce0a3 bitbake: bb/utils.py: add break_hardlinks helper
bb.utils.copyfile is called in a few places with identical src and dst
in order to create an st_nlinks==1 version of the file. That that even
works relies on an implementation detail of copyfile (namely, that it
creates a temporary file and then does a rename). Moreover, it's a waste
of time if the file already has st_nlinks==1.

So create a helper that optimizes away the copy in the st_nlinks==1
case. Of course, this helper relies on the same implementation detail,
but that's now contained within bb.utils itself.

To test that we do at least sometimes hit the no-copy path, I tested
locally with

     if sstat[stat.ST_NLINK] == 1:
+        bb.note("Woohoo, 2*%d bytes I/O avoided" % sstat[stat.ST_SIZE])
         return True

(and the obvious places in oe-core patched), and the do_package log files
are indeed filled with woohoo notes.

(Bitbake rev: 7ae93cf40ab91965147055100432961436bce46c)

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-16 09:50:42 +01:00
Andreas Müller
331463011f bitbake: runqueue: Move decision if a task can be started to one common place
(Bitbake rev: 99d9e4389e1f1d78d17a23ee078fe3f4a12cb31d)

Signed-off-by: Andreas Müller <schnitzeltony@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15 21:45:58 +01:00
Ross Burton
beb93fff65 bitbake: utils/md5_file: don't iterate line-by-line
Opening a file in binary mode and iterating it seems like the simple solution
but will still break on newlines, which for binary files isn't really useful as
the size of the chunks could be huge or tiny.

Instead, let's be a bit more clever: we'll be MD5ing lots of files, but we don't
want to fill up memory: use mmap() to open the file and read the file in 8k
blocks.

(Bitbake rev: a0ac8d67f1471a0c611d691b856fede67efb53f6)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15 09:45:02 +01:00
Ross Burton
63a4ff7cf5 bitbake: checksum: sanity check path when recursively checksumming
In case something goes tragically wrong, catch a request to checksum / and
refuse.

(Bitbake rev: e7cd4c86ef8a2c2bbf068e84c83fdc9e052b6e3d)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-14 16:32:48 +01:00
Kai Kang
5581b374b9 bitbake: event.py: fix typo
(Bitbake rev: d49483eb733ee27325349246fa37a84140c0941d)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-14 16:32:48 +01:00
Richard Purdie
dbda297fd9 bitbake: runqueue: Allow the hash validate function to have an idea of build completion
There is an oversight in the current hash validation API in that the
function can't know how many setscene tasks already completed. Rather
than trying to add additional parameters to the function, causing
incompatibilities, store the value in the datastore.

This is useful to allow build status reporting to the user for
figures on sstate reusage and build completion.

(Bitbake rev: ec037d3e49264037b81212f498d98e292ae7c334)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-08 10:52:00 +01:00
Mark Hatle
da24071e92 bitbake: toaster/orm/management/commands/lsupdates.py: Use new layerindexlib module
Change lsupdates.py to use the new layerindexlib module to load the data from
the public layer index.  It still does all of the manual parsing.  This
is intended to be a stop gap until the toaster can use the module itself to
manage the data.

Everything else is functionally equivalent to the prior version.

(Bitbake rev: 8e482342c652e298b5f5ea58eda72c5eb14ce2bd)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-02 10:18:27 +01:00
Mark Hatle
cbf41cbe55 bitbake: bitbake-layers: disable parsing for layerindex commands
These don't need to access recipe information, so let's not waste the
user's time parsing all recipes.

(Bitbake rev: 6a242a399ae93393c3fc60eda541c1f8f77fed57)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-02 10:18:27 +01:00
Mark Hatle
ddfcda7ec3 bitbake: bblayers/layerindex.py: Switch to use the new layerindexlib class
Display changes:
The output will now include references to the layers that the user already
has on their system.  It does this by querying the cooker derived index.

The code that enables this behavior is labeled as 'TODO' currently.  As
part of the work we need to make a final determination if this is the
desired output.

Also changed the default branch to no longer define itself as 'master'.

When the user does NOT set a branch, the default is now the
'LAYERSERIES_CORENAMES', and if that doesn't exist 'master'.  This is
subtly different in behavior, but more consistent with user expectations.

(Bitbake rev: 478c4b5489943f756ae03d6b6d3b5d665cedbce0)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-02 10:18:27 +01:00
Mark Hatle
1ac19d1bf1 bitbake: layerindexlib: Initial layer index processing module implementation
The layer index module is expected to be used by various parts of the system
in order to access a layerindex-web (such as layers.openembedded.org) and
perform basic processing on the information, such as dependency scanning.

Along with the layerindex implementation are associated tests.  The tests
properly honor BB_SKIP_NETTESTS='yes' to prevent test failures.

Tests Implemented:
   - Branch, LayerItem, LayerBranch, LayerDependency, Recipe, Machine and
      Distro objects
   - LayerIndex setup using the layers.openembedded.org restapi
   - LayerIndex storing and retrieving from a file
   - LayerIndex verify dependency resolution ordering
   - LayerIndex setup using simulated cooker data

(Bitbake rev: fd0ee6c10dbb5592731e56f4c592fe687682a3e6)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-02 10:18:27 +01:00
Mark Hatle
0dea950931 bitbake: bblayers/layerindex.py: Fix addition of layers
When a layer is added it needs to be in a list, otherwise the system will
error such as:

    Specified layer directory / doesn't contain a conf/layer.conf file

Additionally, instead of calling the add layer function over and over, it
is better to add all of the new content in one command.  Otherwise the
order is important as the system now checks if the layer can be added.  For
instance, trying to add meta-python:

   Layer                Required by          Git repository                                          Subdirectory
   ===================================================================================================================
   meta-python          -                    git://git.openembedded.org/meta-openembedded            meta-python
   meta-oe              meta-python          git://git.openembedded.org/meta-openembedded            meta-oe
   openembedded-core    meta-python          git://git.openembedded.org/openembedded-core            meta
   Adding layer "meta-python" (.../oe-core/meta-openembedded/meta-python) to conf/bblayers.conf
   ERROR: Layer 'meta-python' depends on layer 'openembedded-layer', but this layer is not enabled in your configuration

The system would try to add meta-python before the dependent meta-oe.  Adding
them both at the same time resolves this issue.

(Bitbake rev: 8aeaabf13db645f33495e00b82117327e153d70a)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-02 10:18:27 +01:00
Richard Purdie
9571101105 bitbake: bblayers: Don't parse all recipes when calling add-layer
When adding multiple layers in a chain of commands, reparsing all recipes
each time can be painfully slow. Instead just parse the base configuration
which gives some confidence things worked out correctly without as much
overhead.

(Bitbake rev: cfd1302031f3cca96300a0e445a47b1614ecd00c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-01 10:07:22 +01:00
Alejandro Enedino Hernandez Samaniego
b8730f68d2 bitbake: bitbake: Add support for multiconfig dependencies
This patch adds the capability for tasks from different
multiconfigs to depend on one another.

These dependencies can be enabled using the following format:

task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"

For the sake of simplicity consider the following example:

Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.

Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"

Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.

This patch makes modifications to:
- cooker: To glue both multiconfigs in one place and make sure
  the dependencies can be provided.
- taskdata: To parse and add a new kind of dependency (mcdepends) to
  the taskdata object.
- runqueue: To differentiate tasks from different multiconfigs,
  add the specified dependencies to the corresponding tasks, and
  create a working runqueue that contains tasks from both multiconfigs.
- siggen: To avoid looking for tasks from different multiconfigs on
  objects where they dont belong.

The taskdata objects are still not aware of the concept of multiconfig,
so each object doesnt know which multiconfig its building, hence why
the mcdepends are added to all taskdata objects equally (we really
dont expect many of these), but the actual dependencies are added only
to the required tasks by the runqueue.

(Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-01 10:07:22 +01:00
Martin Jansa
9c419271b5 bitbake: codeparser: Track oe.utils.conditional usage
* base_conditional was already removed from oe-core:
  http://git.openembedded.org/openembedded-core/commit/?id=0391fcad9103abca0796a068f957d0df63ab4776
  after the usage was migrated to oe.utils.conditional:
  http://git.openembedded.org/openembedded-core/commit/?id=c97acbd034532895ce57c6717ed1b3ccc7900b0d
  so we can handle just ".conditional" version

* add 1st parameter to variable dependencies, that way when you use
  FOO = "${@oe.utils.conditional('VAR', 'VALUE', 'true', 'false')"
  FOO variable will have dependency on VAR variable and you don't need
  to add FOO[vardeps] += "VAR" manually every time you use
  oe.utils.conditional

* this is similar to contains tracking from:
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=3890
  http://git.openembedded.org/bitbake/commit/?id=ed2d0a22a80299de0cfd377999950cf4b26c512e
  http://git.openembedded.org/bitbake/commit/?id=0b9d117631ce909312d53b93289e61defc6be01c
  but conditional is simpler, we don't need to handle the first
  parameter as a set

(Bitbake rev: 5156b4bb6876dac636be9726df22c8ee792714dd)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-30 12:45:15 +01:00
Paul Eggleton
6ed781e071 bitbake: utils: add optional callback to edit_bblayers_conf()
Add a callback that lets you modify or remove items in addition to the
current scheme where you can only add or remove. This enables you to for
example replace a layer with a temporary copy (which is what we will use
this for first in OE's oe-selftest).

(Bitbake rev: bfedb4e85a84e817dbe5d8694b8f8fcdd6f2f22a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-30 12:45:15 +01:00
Richard Purdie
d854f1b6dc bitbake: server/process: Always place the server logfile in the build directory
Currently the bitbake-cookerdaemon.log is placed into cwd. This seems like a
bad idea, we can place it in the build directory alongside the lockfile that
represents the server instead.

(Bitbake rev: 1620dbc48ffb2a882371cf9174a7b12648befc8a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-25 16:48:27 +01:00
Joshua Watt
9b034e9a83 bitbake: runqueue: Stop on first unsatisfied dependency
(Bitbake rev: 9876b5da1c65bf09a790542cb4057f2d93868cf7)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Joshua Watt
53d1027625 bitbake: runqueue: Remove unused variable
(Bitbake rev: 91e5540f53aca93e3489255cfd95feaa0afd0498)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Joshua Watt
6b66f22686 bitbake: siggen: Delete trailing whitespace
(Bitbake rev: b2fd783fa23403e8f08d998951b05bcabb458af1)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Joshua Watt
714f2f5bae bitbake: runqueue: Remove trailing whitespace
(Bitbake rev: 9b9aecbbb1a3fa67f7d3b1669186c9f4ced3a590)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Ming Liu
7cb125d7ee bitbake: runqueue.py: Fix a virtual class extension stamps issue
The file_name parameter passed to bb.parse.siggen.invalidate_task
should be a virtual file name instead of a real file name, or else you
will encounter a following error, for instance, when you run:

$ bitbake nativesdk-lzip -c unpack -f

the error arise:
| ERROR: An uncaught exception occurred in runqueue
| if file_name:
|    > taintfn = d.stamp[file_name] + '.' + task + '.taint'
| else:
| KeyError: 'virtual:nativesdk:/opt/poky/meta/recipes-extended/lzip/lzip_1.19.bb'

when multilib builds are used on OE.

(Bitbake rev: da37bdad46e11e7ce93ba7a59d58757b769dc16b)

Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-02 21:38:06 +01:00
Nikolai Merinov
070a06ec53 bitbake: fetch2: Use correct check in parameter handling in uri_replace()
If uri_find contain parameters then original URI parameters should
be checked against parameters from uri_find instead of parameters
from uri_replace.

(Bitbake rev: 8efa7826a61501589afa33eb698c0ab3a622bf2e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21 09:34:40 +01:00
Mark Hatle
a0cb1021c0 bitbake: runqueue.py: Initial implementation of per task process limits
On high core machines, in do_fetch, it is possible to DDoS your own machine.
A method to limit any arbitrary task type to a certain number of simultaneous
threads is needed.  (Similar to how BB_NUMBER_THREADS works in the general
case.)  The format of this new limitation is:

    do_fetch[number_threads] = "2"

This should be set globally.  If it is set in individual recipes it could
result in unpredictable behavior.

Note: a value for number_threads > BB_NUMBER_THREADS will have no effect.

(Bitbake rev: 055865047c63b9c3b213b47a1884924ce0adeda0)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-15 17:57:30 +01:00
Mark Hatle
27101e647f bitbake: runqueue.py: Minor cleanup for RunQueueStats and users
The RunQueueStats:taskCompleted and RunQueueStats:taskSkipped can take
multiple arguments.  However, nowehere in bitbake are multiple arguments used.
Change this to match the behavior of the other APIs where it needs to be
called once for each task.

Additionally, these two functions were usually called in tandem, however in
the wrong order.  It really doesn't matter as there is no specific preemption
point between the calls.  But the taskSkipped should be called first to
increment the 'active' count, and then taskCompleted called to decrement it.

(Bitbake rev: 26d5ea9bb892bd6a2e1fd29a9023e0b0644edc16)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-15 17:57:30 +01:00
Andre McCurdy
7838294271 bitbake: fetch2: unify the way fetchers determine DL_DIR and FETCHCMD
Currently there is quite some variation between the fetchers in terms
of how they determine the subdirectory within DL_DIR and the base
fetch command to run. Some rely on variables being set externally
(e.g. from bitbake.conf in oe-core), some respect these external
variables but provide fallback defaults and some use only hardcoded
internal values. Try to unify the approach used across the various
fetchers.

(Bitbake rev: efd5e35af4b08501c67e8b30f30d9457f6fdf610)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-15 17:57:30 +01:00
Matt Hoosier
8ed0441be0 bitbake: fetch/gitsm: avoid live submodule fetching during unpack()
Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

[YOCTO #12739]

(Bitbake rev: 11b6a5d5c1b1bb0ce0c5bb3983610d13a3e8f84a)

Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-07 08:53:19 +01:00
Richard Purdie
1c7ad49bfd bitbake: bitbake: Update version to post release 1.39
(Bitbake rev: a6a4dd35e3fd112b9fac6fcefe61253a61b8aa2c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-06 13:35:15 +01:00
Tzu Hsiang Lin
719d068bde bitbake: fetch2: fix import error for Python 3.6.5
When running bitbake command with Python 3.6.5 always result in
import error causing by the change of distutils module.
This patch replaces the method to search executable in PATH by
"/usr/bin/env <command>".

(Bitbake rev: bd9a1b063633af2936ba1dd87b19202424900151)

Signed-off-by: Tzu Hsiang Lin <t9360341@ntut.org.tw>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-29 21:07:46 +01:00
Jakub Dębski
ef7222c6d1 bitbake: fetch2/__init__.py: Fixed handling of uris with empty path
For mirrors or premirrors defined like: "http://.*/.* http://somewhere.org"
fetching ends with errors because function fetch2/__init__.py:encodeurl()
creates url like "http://somewhere.orgsomefile.tar.gz".

It happens because function fetch2/__init__.py:decodeurl()
for url "http://somewhere.org" returns
['http', 'somewhere.org', '', '', '', {}]
and then in function fetch2/__init__.py:uri_replace()
variable result_decode will be
['http', 'somewhere.org', 'somefile.tar.gz', '', '', {}]
(because of line: result_decoded[loc] = os.path.join(result_decoded[loc], basename))
for which encodeurl returns "http://somewhere.orgsomefile.tar.gz".

In addition for mirror "http://.*/.* http://somewhere.org/"
everything works fine.

(Bitbake rev: d822ae24ef5485e550804cbd9130ebd73b2aa48e)

Signed-off-by: Jakub Dębski <jdebski@enigma.com.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-22 13:13:19 +01:00
Andre McCurdy
b68ed05be6 bitbake: fetch/git: make fewer calls to _contains_ref() from download()
Updating a local git repo clone currently results in multiple calls
to self._contains_ref(), some of which appear to be redundant and can
be eliminated by minor tweaks to the logic in download().

Also drop redundant calls to os.path.exists(ud.clonedir) before
self.need_update(), since need_update() includes its own built-in
check for the existance of ud.clonedir.

(Bitbake rev: 61b0df5523afc8f805043f3adc9c106690e6f133)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-22 13:13:19 +01:00
David Reyna
b6bc5b2840 bitbake: toaster: add 'Sumo' to release selection
Add Sumo (YP-2.5) to the release selection for new projects.

[YOCTO #12713]

(Bitbake rev: 76b17ffcea5c7275c2f9735a058256ba909b1a75)

Signed-off-by: David Reyna <david.reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-26 11:43:26 +01:00
Mattias Hansson
c4338a9420 bitbake: lib/bb/utils: prevent movefile from changing dest dir permissions
Prevent movefile from falsely setting the source file's owner and
permissions on the destination directory instead of the destination
file when moving between devices.

This bug caused the last file moved into a directory to dictate the
directory's owner and permissions.

(Bitbake rev: 82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40)

Signed-off-by: Mattias Hansson <mattias.hansson@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-23 17:26:05 +01:00
Richard Purdie
a0f22fd971 bitbake: cookerdata: Issue warning if layer doesn't set LAYERSERIES_COMPAT_x
We'd like layers to set this variable so that we know which layers are compatible
with which others, even if the branch is a generic un-updated "master" branch.

Start printing a warning to highlight this issue.

(Bitbake rev: cca81e33b58c390dcf5cc3a31555a43b79177166)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-06 11:39:49 +01:00
Paul Eggleton
ba54521c67 lib/bb/shell.py: drop
This was removed in the bitbake repository back in 2010 (bitbake
revision 24857e2ceb405916b0b0b3e75c6c2375a909b9ba) - it's not clear
to me what happened, but for some reason the file remained in the poky
repository. However it does not appear that it has been used since then
and the reasons for removing even more valid now than they were then -
the code has moved on even further, we have memory resident bitbake
server, and if we were to re-implement this we would do it in a
different way. Drop the file and bring us back in sync with the bitbake
repo.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-28 12:59:37 +01:00
David Reyna
2f110a55dc bitbake: toaster: do not fail on optional 'custom.xml' file
Explicitly capture and ignore errors when trying to load the optional
'custom.xml' fixture file.

[YOCTO #12554]

(Bitbake rev: 132458939d3987ebc58685397714af3d6d5cd8fd)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-28 12:52:34 +01:00