Commit Graph

120 Commits

Author SHA1 Message Date
Richard Purdie
3bdabde2df bitbake: command: Avoid time intensive distractions for ping
We noticed some users were seeing very slow ping response times which
caused 'server timeout' issues. There were some function calls in runCommand
which could be slow such as the inotify callback processing.

Mark up the ping command such that it doesn't need configuration information,
it is allowed on a readonly server and specifically skip the inofity processing
too since ping would never need that.

This will hopefully resolve various ping timeout issues that were being reported.

(Bitbake rev: 0fc821a22f2b49cbd336d9658d92942c0d733be1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-02 07:46:40 +01:00
Richard Purdie
695998f921 bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a recipe
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.

Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.

The idea is that this would make layer overrides possible:

OVERRIDES .= ":layer-${FILE_LAYERNAME}"

which then opens possibilities like:

WARN_QA:append:layer-core = " patch-fuzz"

as an example where OE-Core could enable specific QA tests only for that
specific layer.

(Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25 13:16:24 +01:00
Richard Purdie
165e8b563d bitbake: process/cooker/command: Fix currentAsyncCommand locking/races
currentAsyncCommand currently doesn't have any locking and we have
a conflict in "idle" conditions since the idle functions count needs
to be zero *and* there needs to be no active command.

Move the changes/checks of currentAsyncCommand to within the lock
and then we can add it to the condition for idle, simplifying some
of the code.

(Bitbake rev: b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 10:58:36 +00:00
Richard Purdie
4e4f040a73 bitbake: cooker/command: Drop async command handler indirection via cooker
Indirecting the async command handler via cooker is confusing and no
longer needed. Drop it to make things slightly clearer.

(Bitbake rev: 4a41a7d0594e6a84a67b9de70a505858aebcd84a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 10:58:36 +00:00
Richard Purdie
82e77db0ce bitbake: command: Ensure that failure cases call finishAsyncComand
There are a couple of failure cases that runCommands needs to better
handle, ensuring finishAsyncCommand is called. This doesn't matter if the
server is about to shut down but for memory resident bitbake and with
threading enabled, correctness is more important this could could in
theory lead to problems with the recent code changes.

By using the idleFinish class, it ensures the current async command
is terminated correctly and the various state pieces around commands
finishing are called. This also makes the code more uniform matching
the other exception handling code.

(Bitbake rev: 367a83ed46c7fbcdd06579b6cc3a6e48a89ca7fb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-02 17:32:57 +00:00
Richard Purdie
4c57c6eeec bitbake: server/process: Run idle commands in a separate idle thread
When bitbake is off running heavier "idle" commands, it doesn't service it's
command socket which means stopping/interrupting it is hard. It also means we
can't "ping" from the UI to know if it is still alive.

For those reasons, split idle command execution into it's own thread.

The commands are generally already self containted so this is easier than
expected. We do have to be careful to only handle inotify poll() from a single
thread at a time. It also means we always have to use a thread lock when sending
events since both the idle thread and the command thread may generate log messages
(and hence events). The patch depends on  previous fixes to the builtins locking
in event.py and the heartbeat enable/disable changes as well as other locking
additions.

We use a condition to signal from the idle thread when other sections of code
can continue, thanks to Joshua Watt for the review and tweaks squashed into this
patch. We do have some sync points where we need to ensure any currently executing
commands have finished before we can start a new async command for example.

(Bitbake rev: 67dd9a5e84811df8869a82da6a37a41ee8fe94e2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
c4ecfc4dc5 bitbake: server/process: Improve idle loop exit code
When idle handlers want to exit, returning "False" isn't very clear
and also causes challenges with the ordering of the removing the idle
handler and marking that no async command is running.

Use a specific class to signal the exit condition allowing clearer code
and allowing the async command to be cleared after the handler has been
removed, reducing any opportunity for races.

(Bitbake rev: 102e8d0d4c5c0dd8c7ba09ad26589deec77e4308)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
4895e78234 bitbake: command: Tweak finishAsyncCommand ordering to avoid races
(Bitbake rev: 89435442946767cfe58eedde363802add8f1ab29)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
26f255da09 bitbake: command: Add ping command
Add a simple ping command so the UI can check the server is still there.

(Bitbake rev: fd3359de0b9f18fac187a629df203be0b2c87545)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-21 14:15:26 +00:00
Chen Qi
a5f1b44310 bitbake: command.py: cleanup bb.cache.parse_recipe
The parse_recipe has been removed from bb.cache, replace it with
the databuilder._parse_recipe.

(Bitbake rev: d386fa81848247a3d407debf889db8cbcce03359)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-27 23:48:15 +00:00
Richard Purdie
3a4aeda0fa bitbake: cache/cookerdata: Move recipe parsing functions from cache to databuilder
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).

(Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20 08:31:28 +00:00
Richard Purdie
2446bdf59a bitbake: cooker/command: Add a dummy event for tinfoil testing
We need a command genetating an event to test through the tinfoil API. The
current test has IO load issues so add a dummy version which won't have
the IO constraints.

(Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-26 16:35:00 +01:00
Richard Purdie
5a4501abeb bitbake: command: Ensure we catch/handle exceptions
If an exception occurs in early setup, bitbake could just hang. Return
the exception rather than doing that.

[YOCTO #14408]

(Bitbake rev: c8a4107132ce51f84ae84bf1ceb1c3fd90f156d3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-06 06:34:58 +01:00
Richard Purdie
d08d362cf9 bitbake: tinfoil/data_smart: Allow variable history emit() to function remotely
We can't access the emit() function of varhistory currently as the datastore parameter
isn't handled correctly, nor is the output stream. Add a custom wrapper for this
function which handles the two details correctly.

(Bitbake rev: ba0fa084ccd2b1ade96425d158fd31e49e42f286)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Richard Purdie
20397f8d4b bitbake: command: Ensure exceptions inheriting from BBHandledException are visible
Previous changes allowed BBHandledException to be detected but not exceptions
which inherit from it. Fix this. The code really needs totally reworking
to preserve the exceptions.

[YOCTO #14054]

(Bitbake rev: ef762d92df6c2554c6248e80212f984d9ec4c651)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-10 13:37:56 +01:00
Richard Purdie
79adf16931 bitbake: cooker/command: Fix disconnection handling
After the recent init changes, if a client disconnects before issuing a
command, the cooker can break in the reset handlers. Add some guards
in the code to prevent this.

(Bitbake rev: 12605e30e4c4e1ae6a67c97363b892ebf0b9566c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-12 15:29:56 +01:00
Richard Purdie
0bcc00ac51 bitbake: cooker: Defer configuration init to after UI connection
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.

The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.

(Bitbake rev: 3caa43b665604475d2c87ba505efb0b9fca9c2e9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
a3448ad15e bitbake: server/process: Simplfy idle callback handler function
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.

(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-13 08:07:00 +01:00
Joshua Watt
6c2bb4a4ad bitbake: bitbake: command: Handle multiconfig in findSigInfo
Changes the findSigInfo command to accept a recipe specified with the
multiconfig prefix

(Bitbake rev: 379951b6417eacbafc92ac1413ae1358bafdddfb)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-25 15:11:48 +01:00
Richard Purdie
0aff94023f bitbake: cooker: Fix unmatched files handling leading to misleading warnings
Currently if all recipes in a layer are skipped, there are warnings that the
BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this
for skipped recipes.

The current code is hard to understand, not least as it passes variables
which functions modify by reference rather than giving a return value.

Update calc_bbfile_priority() to return values rather than modifying them.
Refactor the code to try and make it clearer what its doing and fix the
skipped recipe issue by passing in the list of parsed files.

The function is complicated by the need to not rerun regex matching more
than we ever have to which complicates the flow, it would be easier if we
just reran operations multiple times.

(Bitbake rev: 969cb27b4d978551817612ff4558bec81cfb655c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-21 16:57:43 +01:00
Joshua Watt
0ecab7a461 bitbake: bitbake: command: Move split_mc_pn to runqueue
All of the other multiconfig splitting functions are located in runqueue
so move the function to split a pn/fn there also so that its easier to
see them all together. Fixes a case where the findBestProvider() command
wasn't working for multiconfig because it was looking for a prefix of
"multiconfig:" instead of the newer "mc:"

(Bitbake rev: 325827af66434affc2da460cc8b9a5c460e38056)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Joshua Watt
b9fdb6a426 bitbake: bitbake: cooker: Split file collections per multiconfig
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.

(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Richard Purdie
70ee928f8f bitbake: tinfoil: Add back ability to parse on top of a datastore
This option was removed recently as we didn't realise the layerindex
relies upon it. Add back the API which it turns out can be supported
as long as we assume the datastore passed in is a remote datastore
which it usually would be unless created locally.

(Bitbake rev: 4618da2094189e4d814b7d65672cb65c86c0626a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:45:34 +01:00
Richard Purdie
d53958766e bitbake: tinfoil: Simplify remote datastore connections
The current approach to remote datastores used in tinfoil is breaking. For
example, adding a devupstream extension to a recipe with a git upstream,
making it the preferred version and then running "devtool modify" on it
causes get_srcrev() circular dependency issues. The problem is the override
handling in the datastore is broken.

This gets broken since remotedata:recieve_datastore() sets d.dict but doesn't
update d.overridedata (or d.inchistory or d.varhistory). We could play
whack-a-mole but the current implementation seems to be flawed to me. It
also doesn't cover, or only partially covers some datastore operations and
each needs new dedicated command API.

Instead, step back and reimplement the way the datastore connector works.

With this change, the datastore is either remote or local but the data is not
spread on two sides of the connection. All the API is proxied over the connection
by a single function for the datastore (and two to support variable history
and include history).

This code does not support using the datastore as a parameter to any data store
functions. We did have one case of that but its just bad code and can be
replaced.

The result is something which is much simpler and less invasive to the datastore
code itself, meaning its behaviour should be much more consistent. The existing
tests for the remote data no longer make any sense and are removed.

The one bug this code would have is if key/value pairs are returned over the IPC
and those values contained a DataSmart object since we don't recurse into return
values to find such things. Nothing appears to do that currently so lets worry
about it if its ever an issue. This change should simplfy a ton of other issues
and avoid a ton of other bugs so is a huge net gain.

Tested with bitbake's and OE's selftests.

(Bitbake rev: 85e03a64dd0a4ebe71009ec4bdf4192c04a9786e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Joshua Watt
86aa524667 bitbake: command: Add mc parameter to findProviders command
Adds a multiconfig selection parameter to the findProviders command.
This allows a client to find the providers for a specific multiconfig
instead of the base configuration.

(Bitbake rev: 95bb446137734865f3fe04fe97681d90250deb11)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-07 16:07:00 +00:00
Frazer Clews
fa5524890e bitbake: lib: amend code to use proper singleton comparisons where possible
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.

(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Paul Eggleton
c697226168 bitbake: cooker: clear extra config settings and remote datastores on client disconnect
When the UI disconnects, we can throw away any server-side remote
datastores we created in response to calls from the UI, and we *must*
drop everything in extraconfigdata or it will taint any future
operations.

Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors
when running devtool.DevtoolTests.test_devtool_update_recipe_local_files
within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded.

(Bitbake rev: 1ca2eec459424892391f060442ef38cf28d6a54a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-05 15:00:49 +01:00
Richard Purdie
92903ced3a bitbake: command: Fix some bugs identified by BB_SERVER_TIMEOUT
The 'needconfig' flag was meant to be the default as most commands do need
it and the "False" cases were the exception. The code was written backwards
with a default False value. Invert this to match the intent, resulting in
the config being reparsed if metadata has changed.

Also ensure the second level configuration is parsed for the getLayerPriorities
command as otherwise it can return stale info.

With these changes:

BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_add_remove

passes instead of fails.

(Bitbake rev: af3c8928a69f204d5ced02c947485990ac04a776)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
019f518287 bitbake: cooker: Ensure we handle inotify before running the next command
The inotify watch events are handled "at idle" which could in theory mean
a command could run before any preceeding inotify events have been processed.
This leads to a theoretical race window where those events may have a
signficicant effect on the command.

Add a mechanism to allow us to ensure all pending events are processed before
running commands.

(Bitbake rev: bf76cd7e5881adf264b8ba64e27a5b6ca9df4fde)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Paul Eggleton
d4c3ace097 bitbake: bitbake-diffsigs: fix regression after recent server changes
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.

(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)

Fixes [YOCTO #11844].

(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 15:14:20 +01:00
Paul Eggleton
9e8bbe7c6f bitbake: tinfoil: enable access to additional cached items
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.

(Bitbake rev: 3df9b7c615174a6557581f3cd157842a28f6bb26)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
21bb330f46 bitbake: lib/bb/event: refactor printing events
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.

(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
192b2e1536 bitbake: tinfoil: add internal mode to build_file() function
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.

(Bitbake rev: ba53e067a2d448dd63b4ca252557ce98aa8e6321)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
dd71707d5a bitbake: event/command: Allow UI to request the UI eventhander ID
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.

Use the new command in the process server backend.

(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Paul Eggleton
833520d2fb bitbake: command / cooker: drop Hob support commands and functions
Drop a number of the commands and support functions that were created
solely to support functionality in the now-removed Hob UI. In most cases
we now have this functionality elsewhere e.g. to modify config files and
recipes we have bb.utils.edit_metadata_file() and friends, and in OE we
have oe.recipeutils.patch_recipe_file() and friends which build on top of
the former. Additionally, some of it represented pretty egregious
incursion of OE metadata-specific references into BitBake code.

For now I have left in the find*File functions and commands as they are
relatively generic and possibly still useful, but they might be removed
in future.

(Bitbake rev: a322f13183c66a28d93cc4bc3d839d95fc1d90f4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
3d4c64fb9f bitbake: cooker: add BB_CMDLINE to enable access to UI command line with memres
In OpenEmbedded's buildhistory class we want access to the bitbake
command line that launched the build, and up to now we were simply using
sys.argv from within the event handler to get that. Unfortunately that
doesn't work in memory resident mode, since the event handler is
naturally executing within the server and thus will give you the command
that launched the bitbake server which is much less interesting. Add a
dynamic variable BB_CMDLINE to provide access to this, set from sys.argv
within the UI process in updateToServer().

(Note that BB_CMDLINE isn't currently passed through to the worker, so
this is only really readable from event handlers plus any explicit
getVariable calls - in theory an observe-only UI could read it for
example.)

Part of the fix for [YOCTO #11634].

(Bitbake rev: 8ad31a3eff5cfcb7b1c462578a582dafcbc426d4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
d45f5be977 bitbake: lib/bb/command: fix overrides in remote datastores where there is no default
If you have, for example, no value set for a variable VARIABLE and a
then VARIABLE_pn-something = "value" and then you parse something.bb,
you expect getVar('VARIABLE') on the resulting datastore to return
"value", but the code here assumed that if the variable wasn't set
without overrides then we didn't need to return the overridedata and
thus we didn't get the overridden value.

In OE this affected the ability to get RECIPE_MAINTAINER for a recipe
in a script using tinfoil (since this is only set from an inc file with
_pn- overrides for each recipe, and no default is set).

(Bitbake rev: b3d2c9917c5fd8278878328794daa107ddf79b64)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-20 07:59:40 +01:00
Juro Bystricky
00d49a05a6 bitbake: command.py: multiconfig support for findBestProvider
In a multiconfig environment, a tinfoil call such as

    tinfoil.parse_recipe("multiconfig:arduino-101-sss:gcc")

can fail with an error such as:

  File "/data/master/poky/bitbake/lib/bb/tinfoil.py", line 373, in get_recipe_file
    raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
bb.providers.NoProvider: Unable to find any recipe file matching "multiconfig:arduino-101-sss:gcc"

The culprit is findBestProvider, which does not handle multiconfig.
This patch fixes the error and in the case mentioned above the tinfoil call returns:

  "multiconfig:arduino-101-sss:/data/master/poky/meta/recipes-devtools/gcc/gcc_6.3.bb"

[YOCTO#11210]

(Bitbake rev: e9c03fbfd7b057b28645affa263cb4aebfa24b04)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19 10:20:24 +01:00
Paul Eggleton
5b1807021e bitbake: data_smart: implement missing remote datastore operations
Enable the following operations from a remote datastore to affect the
other end:

* setVarFlag()
* delVar()
* delVarFlag()
* renameVar()

In practice I don't expect these to be used much, but they should be
present so that the implementation is at least reasonably filled out
and that the tests pass.

Also add tests for the interface, mostly by subclassing the existing
local test classes so that they are using a remote datastore. (These
don't actually test remote usage via tinfoil, just that the
datastore's interface can be used.)

(Bitbake rev: 282dc0719d22a39df746eea762ebe05c66aa8f8a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:22 +00:00
Paul Eggleton
99414bdb1c bitbake: tinfoil: fix override handling in remote datastores
There was a huge gap in the remote datastore code introduced in the
tinfoil2 rework - we weren't handling overrides at all, since these are
stored separately from the actual data in the DataSmart object. Thus,
when a datastore actually represents a remote datastore we need to go
back to that remote datastore to get the override data as well, so
introduce code to do that.

To avoid a second round-trip I had to modify the _findVar() function to
return the override data as well. This will increase the overhead a
little when that data is superfluous, but without making the function
even uglier I don't think there's a way to avoid that.

(Bitbake rev: 4f9d6f060ed247fb6fa2f45668a892a1788d3f91)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:22 +00:00
Richard Purdie
2c4e366721 bitbake: cooker/command: Drop expanded_data
Some of our metadata assumes that BuildStarted and BuildCompleted events
see the same data store. This is the case for buildTarget but not for
buildFile and recent changes mean this is now a problem.

The update_data() call is now an empty operation and there is no difference
between the expanded_data and data so we can simply remove the expanded_data
and its references and use data everywhere. This has been inteded for a while
but the above issue makes this more pressing to finally clean up.

(Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:24 +00:00
Paul Eggleton
8d315820aa bitbake: runqueue: enable setVariable command to affect task execution
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.

(Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
797a8ee040 bitbake: cooker: allow buildFile warning to be hidden programmatically
If we want to use this function/command internally, we don't want this
warning shown.

(Bitbake rev: 5cfbb60833e7b12d698c1c2970c17ccf2a4971bf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
8f8a9ef669 bitbake: tinfoil: pass datastore to server when expanding python references
If you're expanding a value that refers to the value of a variable in
python code, we need to ensure that the datastore that gets used to get
the value of that variable is the client-side datastore and not just the
part of it that's on the server side. For example, suppose you are in
client code doing the following:

d.setVar('HELLO', 'there')
result = d.expand('${@d.getVar("HELLO", True)}')

result should be "there" but if the client part wasn't taken into
account, it would be whatever value HELLO had in the server portion of
the datastore (if any).

(Bitbake rev: cbc22a0a9aadc8606b927dbac0f1407ec2736b35)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
f1f3a112a0 bitbake: tinfoil: implement server-side recipe parsing
It's not really practical for us to parse recipes on the client side, we
need to do it on the server because that's where we have the full python
environment (including any "pure" python functions defined in classes).
Thus, add some functions to tinfoil do this including a few shortcut
functions.

(Bitbake rev: 8f635815d191c9d848a92d51fdbf5e9fd3da1727)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
7d5c9860de bitbake: tinfoil: rewrite as a wrapper around the UI
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:

* It now works when bitbake is memory-resident instead of failing with
  "ERROR: Only one copy of bitbake should be run against a build
  directory".

* We can now connect an actual UI, thus you get things like the recipe
  parsing / cache loading progress bar and parse error handling for free

* We can now handle events generated by the server if we wish to do so

* We can potentially extend this to do more stuff, e.g. actually running
  build operations - this needs to be made more practical before we can
  use it though (since you effectively have to become the UI yourself
  for this at the moment.)

The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.

Part of the fix for [YOCTO #5470].

(Bitbake rev: 3bbf8d611c859f74d563778115677a04f5c4ab43)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
e271d7dc60 bitbake: command: provide a means to shut down from the client in memres mode
In memory resident mode we don't really want to actually shut down since
it's only the client going away.

(Bitbake rev: 74db369c46043116359101cab70486afd82372c0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Richard Purdie
0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00