Commit Graph

69 Commits

Author SHA1 Message Date
Yoann Congal
159a49f35a bitbake: tinfoil: Track data during finalizeData
finalizeData changes the value of variables, so, it needs to have data
tracking enabled to get a full and accurate variable value history.

Fixes [YOCTO #15652]

(Bitbake rev: 228979b506f7d0e24c643440bcc7becf684da283)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-18 11:03:17 +00:00
Yoann Congal
e4761312f7 bitbake: tinfoil: Refactor temporary data tracking in a context manager
A new context manager Tinfoil._data_tracked_if_enabled() is introduced to
replace the following structure:
  if self.tracking:
      self.run_command('enableDataTracking')
  # Code that need data tracking
  if self.tracking:
      self.run_command('disableDataTracking')

(Bitbake rev: 0fea4555d2143c6b23a79d3d5cf791103a68141b)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-18 11:03:17 +00:00
Chris Laplante
9d1b31d254 bitbake: cooker: Make cooker 'skiplist' per-multiconfig/mc
Previously, the cooker skiplist was shared across multiconfigs
(including default ''). If you had a recipe that was incompatible with
several multiconfigs for different reasons, then the displayed reason
(i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might
vary across invocations of bitbake. This was caused by the random order
in which recipes are parsed under different multiconfig contexts, with
each skip reason overwriting the previously assigned reason.

I hit this specificially when using COMPATIBLE_MACHINE, but
COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too.

(Bitbake rev: c51f01a35ed9a928402eab0899598b5c59602eef)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-17 11:48:25 +00:00
Yoann Congal
30a3ac4a2e bitbake: tinfoil: add new "finalizeData" API
Create a new tinfoil API "finalizeData" which does key expantion and
runs the anonymous functions to allow bitbake-getvar to have the same
output as "bitbake -e".

(Bitbake rev: 63db64ca98564420a378f14c337074ae06da8c63)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-18 13:41:38 +00:00
Peter Kjellerstedt
c7f99d688c bitbake: tinfoil: Do not fail when logging is disabled and full config is used
If Tinfoil is initialized with setup_logging = False and
Tinfoil.prepare() is called with config_only = False, then it fails
because self.localhandlers is only initialized when
setup_logging = True.

This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`:

  Traceback (most recent call last):
    File ".../bitbake/bin/bitbake-getvar", line 41, in <module>
      tinfoil.prepare(quiet=2)
    File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare
      for handler in self.localhandlers:
  AttributeError: 'Tinfoil' object has no attribute 'localhandlers'.
  Did you mean: 'oldhandlers'?

(Bitbake rev: 616101ddb630e2c9975022068b52a87c4cf647f6)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-26 10:37:17 +01:00
Richard Purdie
37c31a5adc bitbake: lib: Drop inotify support and replace with mtime checks
With the flush in serverlog() removed and a memory resident bitbake with a
60s timeout, the following could fail in strange ways:

rm bitbake-cookerdaemon.log
bitbake-layers add-layer ../meta-virtualization/
bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake -m

specifically that it might error adding meta-oe with an error related to meta-virt.

This clearly shows that whilst bblayers.conf was modified, bitbake was not
recognising that. This would fit with the random autobuilder issues seen when
the serverlog flush() call was removed.

The issue appears to be that you have no way to "sync()" the inotify events with
the command stream coming over the socket. There is no way to know if there are
changes in the IO queue which bitbake needs to wait for before proceeding with
the next command.

I did experiment with os.sync() and fsync on the inotify fd, however nothing
addressed the issue. Since it is extremely important we have accurate cache data,
the only realistic thing to do is to switch to stat() calls and check mtime.

For bitbake commands, this is straightforward since we can revalidate the cache
upon new connections/commands. For tinfoil this is problematic and we need to
introduce and explict command "revalidateCaches" that the code can use to force
bitbake to re-check it's cache validity. I've exposed this through tinfoil with
a new "modified_files" function.

So, this patch:

a) drops inotify support within bitbake's cooker/server and switch to using mtime
b) requires a new function call in tinfoil when metadata has been modified

(Bitbake rev: da3ec3801bdb80180b3f1ac24edb27a698415ff7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-18 11:35:05 +01:00
Richard Purdie
92c0dba456 bitbake: tinfoil: Don't wait for events indefinitely
If for some reason the bitbake server dies, we shouldn't loop indefinitely
waiting for events within tinfoil. Add a ping test and exit if things
have somehow failed.

(Bitbake rev: 32da768ae67cfe0b2d154ce4667a7cfdfa5bf4b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 23:14:48 +00:00
Richard Purdie
4dc7967cf9 bitbake: tinfoil: Ensure CommandExit is handled
By inspection, tinfoil handles two of the three command exit cases but
one is missing. Add the CommandExit case in case this is the cause of
one of our recipetool/devtool hangs. Regardless, the fix is necessary.

(Bitbake rev: eadddd94835b6b6a8517dfed3d29e6dbb2d35988)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-29 12:49:54 +00:00
Richard Purdie
a10ac380b7 bitbake: tinfoil: Allow run_command not to wait on events
There are some commands where we want to see the events returned so allow
the caller to request this. This also allows us to fix an infamous bug in
the tinfoil testsuite in OE-Core.

(Bitbake rev: 0e8421c41d97d5d50a553d70c8f775d521f1a199)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-26 09:27:43 +00:00
Zygmunt Krynicki
aacc917646 bitbake: tinfoil: Fix typo "receive" and "something"
(Bitbake rev: 2e0407f6d8fbea22a4bfde124f7665a912482401)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:25 +00:00
Richard Purdie
d08d362cf9 bitbake: tinfoil/data_smart: Allow variable history emit() to function remotely
We can't access the emit() function of varhistory currently as the datastore parameter
isn't handled correctly, nor is the output stream. Add a custom wrapper for this
function which handles the two details correctly.

(Bitbake rev: ba0fa084ccd2b1ade96425d158fd31e49e42f286)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Manuel Leonhardt
e6ec2b4dcc bitbake: tinfoil: Honor quiet when parsing recipes
When using parse_recipes, honor quiet so that scripts and custom plugins
for recipetool are able to mute progress bars from bitbake that would
otherwise print to STDOUT.

(Bitbake rev: ab9d26c4847a062cadaae5fb8caac0ead5f958db)

Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Richard Purdie
ebaa3e3f51 bitbake: tinfoil: When sending commands we need to process events
The server may be displaying useful information for the user through log
messages so we should display anything that has been sent. Its either this
or expecting every UI to implement this code around every command call
which isn't good API.

[YOCTO #14054]

(Bitbake rev: 64ae9d7e2fad804dd9e12706c6d76b4b22f9586b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-10 13:37:56 +01:00
Richard Purdie
c4daf38f47 bitbake: tinfoil: Ensure sockets don't leak even when exceptions occur
We're seeing leaking open socket connections when errors occur and
tinfoil is in use. Improve the exception handling so the sockets
are closed even if exceptions occur, allowing more robust behaviour
when things go wrong.

(Bitbake rev: cefbec9ff47ca973a74ec7300cd736f3e0f0bce0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-12 16:02:11 +01:00
Frazer Clews
abc6f864b9 bitbake: lib: fix most undefined code picked up by pylint
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.

(Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5)

Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
b9bbb5c7b7 bitbake: main/server/process: Drop configuration object passing
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.

This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.

The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.

(Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Joshua Watt
b9fdb6a426 bitbake: bitbake: cooker: Split file collections per multiconfig
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.

(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Hongxu Jia
03826c61fd bitbake: tinfoil: fix config_data mess up insane check while parsing multiple recipes
Since commit [tinfoil: Simplify remote datastore connections][1] and
[tinfoil: Add back ability to parse on top of a datastore][2] applied,
bitbake run command parseRecipeFile with param config_data.dsindex rather
than config_data.

While calling tinfoil.parse_recipe_file() with one config_data (with the same
config_data.dsindex) to parse multiple recipes, it will mess up insane check.

It broke update_layer.py on layerindex, here are the simplified steps:
[snip]
t= bb.tinfoil.Tinfoil()
t.prepare()
data = bb.data.createCopy(t.config_data)
fn = "path_to/oe-core/meta/recipes-graphics/images/core-image-clutter.bb"
t.parse_recipe_file(fn, appends=False, config_data=data)
fn = "path_to/oe-core/meta/recipes-graphics/packagegroups/packagegroup-core-x11-base.bb"
t.parse_recipe_file(fn, appends=False, config_data=data)

|  File "path_to/oe-core/meta/classes/insane.bbclass", line 1303,
in __anon_1304__path_to_oe_core_meta_classes_insane_bbclass
|    bb.fatal("Fatal QA errors found, failing task.")
[snip]

In above failure, RDEPENDS is assigned `${PACKAGE_INSTALL} ${LINGUAS_INSTALL}
${IMAGE_INSTALL_DEBUGFS}' in core-image-clutter.bb, but it broke insane check
on packagegroup-core-x11-base.bb

>From commit [remotedata: enable transporting datastore from the client to
the server][3], it create a new DataSmart to save receive_datastore's remote_data
Similarly, make a copy of config_data(with different config_data.dsindex) could
fix the issue.

[1] http://git.openembedded.org/bitbake/commit/?id=85e03a64dd0a4ebe71009ec4bdf4192c04a9786e
[2] http://git.openembedded.org/bitbake/commit/?id=4618da2094189e4d814b7d65672cb65c86c0626a
[3] http://git.openembedded.org/bitbake/commit/?id=784d2f1a024efe632fc9049ce5b78692d419d938

(Bitbake rev: a3074807974536e370289c25fddcb9ad93cbc137)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24 14:31:42 +01:00
Richard Purdie
70ee928f8f bitbake: tinfoil: Add back ability to parse on top of a datastore
This option was removed recently as we didn't realise the layerindex
relies upon it. Add back the API which it turns out can be supported
as long as we assume the datastore passed in is a remote datastore
which it usually would be unless created locally.

(Bitbake rev: 4618da2094189e4d814b7d65672cb65c86c0626a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:45:34 +01:00
Richard Purdie
25f1f4644d bitbake: tinfoil: Add iterator support for DataStoreConnector
Some usages need to iterate the datastore. This is slow and not recommended
but support this.

(Bitbake rev: 38b4e330c6c2007e8ead878e1d9e9580d3b19c7b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Richard Purdie
d53958766e bitbake: tinfoil: Simplify remote datastore connections
The current approach to remote datastores used in tinfoil is breaking. For
example, adding a devupstream extension to a recipe with a git upstream,
making it the preferred version and then running "devtool modify" on it
causes get_srcrev() circular dependency issues. The problem is the override
handling in the datastore is broken.

This gets broken since remotedata:recieve_datastore() sets d.dict but doesn't
update d.overridedata (or d.inchistory or d.varhistory). We could play
whack-a-mole but the current implementation seems to be flawed to me. It
also doesn't cover, or only partially covers some datastore operations and
each needs new dedicated command API.

Instead, step back and reimplement the way the datastore connector works.

With this change, the datastore is either remote or local but the data is not
spread on two sides of the connection. All the API is proxied over the connection
by a single function for the datastore (and two to support variable history
and include history).

This code does not support using the datastore as a parameter to any data store
functions. We did have one case of that but its just bad code and can be
replaced.

The result is something which is much simpler and less invasive to the datastore
code itself, meaning its behaviour should be much more consistent. The existing
tests for the remote data no longer make any sense and are removed.

The one bug this code would have is if key/value pairs are returned over the IPC
and those values contained a DataSmart object since we don't recurse into return
values to find such things. Nothing appears to do that currently so lets worry
about it if its ever an issue. This change should simplfy a ton of other issues
and avoid a ton of other bugs so is a huge net gain.

Tested with bitbake's and OE's selftests.

(Bitbake rev: 85e03a64dd0a4ebe71009ec4bdf4192c04a9786e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Richard Purdie
c0e79ee2e9 bitbake: tinfoil: Drop parse_recipe_file custom datastore support
This parameter is unused except for a single selftest and is problematic
with regard to fixing some other bugs. Remove it for now, if really needed
we could re-implement it in some other way in the future. Experience
tells us we likely don't want to support this kind of change to the metadata
anyway as its not as useful as it first sounds/appears.

(Bitbake rev: f2341dc46b6abfc08a4d01c7abdd3d4630577999)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Richard Purdie
ca5b3b4d1c bitbake: tinfoil: Update to match recent knotty console changes
This updates tinfoil to match recent changes to the logging code
in knotty.

(Bitbake rev: e67dfa4a4d0d63e4752655f25367582e5a95f1da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
300fd2a659 bitbake: knotty: Remove dependency on format variable
Passing around the log formatter variable was unnecessary since the log
levels of interest can be accesses as class members of
bb.msg.BBLogFormatter. Switching to do this will make using the
structured python logging much easier, since it can be difficult to
extract out the formatter for a specific handler.

(Bitbake rev: c1c867df24b4ef204027d485acac7c75c63f2bc0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
ecb1d9f317 bitbake: tinfoil: Add multiconfig support
Adds support for the Tinfoil cache adaptor to be bound to a specific
multiconfig and invoke the appropriate commands for that multiconfig
instead of the default. The cooker adapter now creates a cache adapter
for each multiconfig specified in BBMULTICONFIG so that each multiconfig
is present.

(Bitbake rev: d515481681dca4a0bc733c6ad8a8498a82e3d9b9)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-07 16:07:00 +00:00
Frazer Clews
0ac5174c7d bitbake: lib: remove unused imports
removed unused imports which made the code harder to read, and slightly
but less efficient

(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Paul Eggleton
41c868d076 bitbake: tinfoil: ensure get_recipe_info() returns Null if recipe not found
If a matching recipe is not found then return Null instead of raising
KeyError because we were blindly using None as a key for pkg_fn.

(Bitbake rev: 431e89e322850a2497157c3c0843da9df6bc9a3e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-21 13:06:45 +00:00
Richard Purdie
a87cae5bb1 bitbake: tinfoil: Ensure we clean up loggers
This is primarily paranoid but ensure we remove any loggers we setup
either directly or indirectly so the initial state is restored after
we exit.

(Bitbake rev: af7d63b1f76fd3f7fa92ed15ae61ca47d9e13472)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-09 12:24:40 +00:00
Paul Eggleton
8f716abaa4 bitbake: tinfoil: ensure variable history tracking works when parsing a recipe
If you set tracking=True when creating the tinfoil object, that ensures
history is collected for the main datastore, but at the end of parsing
the configuration, history tracking gets turned off to save time with
the result that we don't collect history for any recipes we parse.
Enable tracking when we parse a recipe (and disable it afterwards if we
enabled it) in order to fix this.

This fixes functionality in OE's devtool that relies upon variable
history (such as devtool upgrade updating PV when it's set within a
recipe).

(Bitbake rev: cc8b4c81bb589fb70774a0151f87a8d277f40f06)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:04 +01:00
Paul Eggleton
a3971620dc bitbake: tinfoil: ensure log lines get printed when tasks fail
If a task fails during build_targets(), we need to print out the log
lines as knotty does or the user will be missing information about the
failure.

(This should get some deeper refactoring, but now isn't the time for
that.)

(Bitbake rev: 24879df071d4803db3d39ae1d5cad852daa92f28)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Paul Eggleton
6ef0a56770 bitbake: tinfoil: fix log message doubling when config_only=False
With config_only=False we launch the UI and it sets up a logger, whereas
when config_only=True we don't, with the result that with True we are
seeing log messages from both our logger and the one set up by the UI.
Suppress our loggers with config_only=True to avoid this.

Fixes [YOCTO #11275] (again).

(Bitbake rev: b5e3b28b7c982dd8a3991d727f25710dbf58bb80)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Paul Eggleton
46e2251625 bitbake: Revert "tinfoil: fix duplication of log messages"
In combination with the recent server reworking, this change actually
prevents messages sent from tasks from being logged properly. This will
of course give us the duplicated messages back, and I really hate to do
that effectively a second time, but that's better than seeing no error
at all in the case of a failure - we'll have to find the proper way of
avoiding the duplication that doesn't result in some messages going
missing.

This reverts commit 8a5bae76f91f2411187c638a42fa3c762052cf11.

(Bitbake rev: 645c8dd15762516ae5ab64a1df47fadb95d072d1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Paul Eggleton
2b5c3617c2 bitbake: tinfoil: add more doc comments
We want this API to be easier to use, so add missing function
documentation to help with that.

(Bitbake rev: 3e0e002d6497caa987f327cd83ad4db82cca6810)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
388ae704bb bitbake: tinfoil: add simple API for getting cached recipe information
A common task for tinfoil-using scripts is to iterate over all recipes.
This isn't too difficult with the current API, but the pkg_* variables
are a little awkward and are really designed for bitbake's internal
usage - and it gets a bit more difficult when you want to access some of
the other information such as packages and rprovides. To resolve this,
create a new recipe info class and add an all_recipes() function to
generate this for all recipes. Also add a get_recipe_info() function to
get the information for a specific recipe (by PN).

(It might perhaps be suggested that we already have a structure similar
to this in the cache, however the one we add here is designed for
external use and allows the internal structures to change if needed
without affecting the API).

(Bitbake rev: 308994028e59735ca726c5d2c1f0f85baccfe89d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
9e8bbe7c6f bitbake: tinfoil: enable access to additional cached items
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.

(Bitbake rev: 3df9b7c615174a6557581f3cd157842a28f6bb26)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
7eb654cb4d bitbake: tinfoil: add functionality for running full builds
Up to this point, if you wanted to run build tasks in the normal way
they get run from a python script, there was no other way than to shell
out to bitbake. Worse than that, you couldn't have tinfoil active during
that because only one bitbake instance could be running at once. As long
as we're prepared to handle the events produced, we can create a wrapper
around calling the buildTargets command. Borrow code from knotty to do
this in such a way that we get the expected running task display
(courtesy of TermFilter) and Ctrl+C handling.

(Bitbake rev: 43761eee756be52a1021be53a40dc591a6c35fa7)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
eb7401d470 bitbake: tinfoil: set a flag when recipes have been parsed
Make it easy to determine if recipes are parsed (and thus information
about available recipes is in memory).

(Bitbake rev: 7efde2df2ff25063d36ac015146f1975284a69ff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
b0f0eaf63c bitbake: tinfoil: add a parse_recipes() function
Python style recommends underscore based naming rather than camelCase,
and thus the former has been used for most of tinfoil's functions. Add
an underscored version of parseRecipes() for consistency and change the
one place we call it to use the new version.

(Bitbake rev: 821f6c41d850752d2bcc2ccd4f8e75b2897a0a3e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Paul Eggleton
192b2e1536 bitbake: tinfoil: add internal mode to build_file() function
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.

(Bitbake rev: ba53e067a2d448dd63b4ca252557ce98aa8e6321)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
4602408c69 bitbake: server: Rework the server API so process and xmlrpc servers coexist
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.

Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.

These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.

Changes:

 * The foreground option for bitbake-server was dropped. Just tail
   the log if you really want this, the codepaths were complicated enough
   without adding one for this.
 * BBSERVER="autodetect" was dropped. The server will autostart and
   autoconnect in process mode. You have to specify an xmlrpc server
   address since that can't be autodetected. I can't see a use case
   for autodetect now.
 * The transport/servetype option to bitbake was dropped.
 * A BB_SERVER_TIMEOUT variable is added which allows the server
   to stay resident for a period of time after the last client
   disconnects before unloading. This is used if the -T/--idle-timeout
   option is not passed to bitbake.

This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.

(Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Paul Eggleton
d44d5a2e79 bitbake: tinfoil: fix duplication of log messages
Adding an additional logger in setup_bitbake() interacts poorly with the
logger we have added by default in tinfoil's constructor, with the
result that messages may be doubled or even tripled in tinfoil-using
scripts. Disable adding this one when calling setup_bitbake() from
tinfoil to avoid this problem.

Part of the fix for [YOCTO #11275].

(Bitbake rev: 8a5bae76f91f2411187c638a42fa3c762052cf11)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Andy Voltz
644fbb9aca bitbake: tinfoil: allow extra features to be passed to prepare
(Bitbake rev: 2e35de1f19dc73a61a18a3eb186efede078d597d)

Signed-off-by: Andy Voltz <andy.voltz@timesys.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Paul Eggleton
03b4226242 bitbake: tinfoil: improve get_recipe_file() exception text
* Turn reasons from a list into a string (usually there will be only one
  reason, but the interface provides for more than one) and state up
  front that the recipe is unavailable for clarity
* Use quotes around invalid recipe name

(Bitbake rev: 8922f1d23400049d2894a97915a533769a24ca07)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-31 10:08:35 +01:00
Paul Eggleton
c2ac78c4d8 bitbake: tinfoil: fix get_recipe_file() to return an error on invalid recipe
This function calls cooker.findBestProvider() but didn't handle the fact
that that function returns a tuple (None, None, None, None) when there
is no matching recipe. (This fixes devtool in OpenEmbedded showing a
traceback when an invalid recipe is specified instead of a proper error
message.)

(Bitbake rev: 54a4757ca706afc6e98c7692f960592e80cab12b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-31 10:08:35 +01:00
Paul Eggleton
542eb4731c bitbake: tinfoil: enable client-side logger handling by default
If you had a script that uses tinfoil and it failed to connect to the
BitBake server, you did't see any of the expected messages - this was
because client-side logging wasn't being handled at all. Since you'll
almost always want this when using tinfoil, have it use the new
bb.msg.logger_create() function to enable client-side logging by
default.

Relates to [YOCTO #11185].

(Bitbake rev: 824e73e0f3eaa96b4d84da7e31f9a17ce5c5d7ee)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-27 20:23:19 +01:00
Paul Eggleton
5b1807021e bitbake: data_smart: implement missing remote datastore operations
Enable the following operations from a remote datastore to affect the
other end:

* setVarFlag()
* delVar()
* delVarFlag()
* renameVar()

In practice I don't expect these to be used much, but they should be
present so that the implementation is at least reasonably filled out
and that the tests pass.

Also add tests for the interface, mostly by subclassing the existing
local test classes so that they are using a remote datastore. (These
don't actually test remote usage via tinfoil, just that the
datastore's interface can be used.)

(Bitbake rev: 282dc0719d22a39df746eea762ebe05c66aa8f8a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:22 +00:00
Paul Eggleton
99414bdb1c bitbake: tinfoil: fix override handling in remote datastores
There was a huge gap in the remote datastore code introduced in the
tinfoil2 rework - we weren't handling overrides at all, since these are
stored separately from the actual data in the DataSmart object. Thus,
when a datastore actually represents a remote datastore we need to go
back to that remote datastore to get the override data as well, so
introduce code to do that.

To avoid a second round-trip I had to modify the _findVar() function to
return the override data as well. This will increase the overhead a
little when that data is superfluous, but without making the function
even uglier I don't think there's a way to avoid that.

(Bitbake rev: 4f9d6f060ed247fb6fa2f45668a892a1788d3f91)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:22 +00:00
Paul Eggleton
157d9d92a0 bitbake: tinfoil: ensure PATH includes bitbake's bin directory
The runqueue expects to be able to run 'bitbake-worker' from PATH, but
for example in the OE extensible SDK, tinfoil is used within devtool
where this isn't the case (we're not exposing bitbake to the user, thus
bitbake's bin directory isn't in PATH).

This fixes devtool modify usage within the extensible SDK which has
apparently been broken since the tinfoil2 changes went in.

Fixes [YOCTO #11034].

(Bitbake rev: 52d87e4a58a52174e8e0b6297abaac8fd93ffcc3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00