Commit Graph

351 Commits

Author SHA1 Message Date
Paul Eggleton
76a281c870 bitbake: taskdata: add the ability to access world targets list
In certain circumstances it can be useful to get access to the world
targets list from a recipe in order to add dependencies on some or all
of the items in it. If a special function, 'calculate_extra_depends' is
defined in the recipe, and the recipe is to be built, then call it at
the right point before we calculate which tasks should be run. The
function can append items to the "deps" list in order to add
dependencies. This is not as tidy a solution as I would have liked, but
it does at least do the job.

As part of this change, the buildWorldTargets function was moved to
bb.providers to make it possible to call from taskdata.

Part of the implementation of [YOCTO #8600].

(Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Ed Bartosh
f2b7252190 bitbake: cooker: add providermap to dep_tree
Added providermap information to the result of buildDependTree API.

This will be used by Toaster to map virtual dependencies to recipes.

(Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ross Burton
bfa7859ffa bitbake: cooker: fix findFilesMatchingInDir documentation
The documentation for findFilesMatchingInDir() was inconsistant with the
implementation: the regex was escaped before searching so effectively it's a
pure textual substring, and the machine example was broken.

(Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton
3d4273716d bitbake: cooker: use in instead of count
No point counting all instances when we just want to know if there's any or not.

(Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ed Bartosh
01419d5565 bitbake: cooker: add state.get_name method
Added method to convert state code into the human readable name.
It will be used in logging and error reporting.

(Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie
2c9431135d bitbake: cooker: Drop useless parsing exception
The SyntaxError exception simply shows the recipe that failed to parse
which is pretty useless without the actual exception. We could make it
print more info, however we can just use one of the more generic handlers
instead and remove this one.

For a python indentation error, this leads to a much more readable error
message.

(Bitbake rev: 9241eb10847634e34c5ff8767ed8c114f66ff6cf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie
3ebf7617d6 bitbake: cooker: Ensure BB_CONSOLE remains correct over server resets
The console log data is written to is created at console initialisation
time and does not change over reset events. This ensures the
BB_CONSOLELOG value is correct over such resets by preserving it.

(Bitbake rev: 335eb2db228f7543a49de71f063ac72b865c947a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:52 +00:00
Ed Bartosh
4ab7202c31 bitbake: cooker: preserve pre and post configs
Additional config files passed to bitbake server with --read and
--postread options are rewritten by client bitbake even if
it doesn't use those options.

This is a show stopper for toaster as toaster command line
builds are based on the assumption that server is aware of
toster configs, provided by --postread option.

This behaviour is fixed by preserving values of --read and
--postread options when bitbake server starts and restoring
them if client bitbake doesn't explicitly specify them.

(Bitbake rev: 02c64f7487ca8ec5d32c440f5002c4b8f64b76a6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:30 +00:00
Ed Bartosh
6fa3fecb12 bitbake: cooker: normalize build targets
BuildStarted event not fully represents build tasks for
the targets. If -c option is used to specify default task
it's not included into the event.

Made build targets to always look as <target>:do_<task>.
Consider default task (do_build or specified by -c command
line option) when normalizing.

(Bitbake rev: 0b0e214e6f53c97ad3d48f622c7fc0ca149956f6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Alex Franco
7ef2f951d6 bitbake: cooker/runqueue: Allow bitbake commands starting with do_
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).

[YOCTO #7818]

(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)

Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:58 +01:00
Paul Eggleton
294948c732 bitbake: lib/bb/cooker: add interrupted flag to BuildCompleted event
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).

(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-29 14:03:49 +01:00
Paul Eggleton
3c3ff92882 bitbake: cooker: drop appendlist
Now we have the bbappends list and all users have been converted over to
use it, we don't need this anymore.

(Bitbake rev: 279770c42d4c63aa2cebce331b55a92a564b50ac)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:46 +01:00
Paul Eggleton
b84bf58c19 bitbake: cooker: drop appliedappendlist
Whilst collecting this list on the fly may be quicker, doing so within a
function that's meant to *query* the list of bbappends is poor practice.

(Bitbake rev: 5c12aa3b0010d7d1733e54a0ca7d0af465454210)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:46 +01:00
Paul Eggleton
7ce27b9fce bitbake: cooker: ensure prefile/postfile can work in memory resident mode
The prefile/postfile options weren't working in memory resident mode
because they weren't being passed through to the server, so ensure that
they do get passed through and that the server is reset when the values
come through.

(Bitbake rev: a3f7dc042fc7b1c308bfd248431930eb8ba50326)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:45 +01:00
Richard Purdie
37a2c062a5 bitbake: command: ensure sync commands that read configuration see updates
Add a means of ensuring that synchronous commands that read the results
of the configuration trigger a reparse of the configuration if any
underlying files have changed.

(Bitbake rev: aaf3cc024315450c1674819edf2a4e5cdf293f35)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:45 +01:00
Paul Eggleton
6e5ac6ba8e bitbake: cooker: further limit inotify watches
Unfortunately we were acting on inotify notifications about any files
changing within the watched directories, not just the ones we actually
care about. In OE the build directory is in BBPATH and hence it gets
watched, and we write things like bitbake.lock and
bitbake-cookerdaemon.log to that directory, hence effectively
notifications were being tripped on every bitbake invocation. To avoid
this, record which file/subdirectory we're interested in against each
watched directory so we can ignore any events for files/subdirectories
we don't care about.

Additionally, if we move up to the parent dir, ensure we haven't already
seen it before adding a watch on it (we were previously calling
watcher.add_watch() on the same directory multiple times before in a
typical OE configuration).

(Bitbake rev: bc39b8da34c046b629c43fd0a8cac2efbf1c060f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Paul Eggleton
715d857174 bitbake: Fix default function parameter assignment to a list
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:

http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Richard Purdie
a8b723498c bitbake: cooker: Resolve file monitoring race issues when using memres bitbake
If you load memory resident bitbake, then change bblayers.conf or other
configuration files, then execute your first command, bitbake will not
notice the changes. This is because it adds the file watches during
inital parsing, which happens at the time the first command is run.

To fix this, we move the addition of the file watches to earlier in
the process, so bitbake then does track them correctly. This also
avoids some issues Paul was seeing with tinfoil2.

(Bitbake rev: b9375c73e736003e66575969c9ea244403e47aeb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-03 07:36:25 +01:00
Richard Purdie
4c2f28bf48 bitbake: cooker: properly fix bitbake.lock handling
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.

The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.

(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).

Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.

This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.

(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01 07:34:09 +01:00
Richard Purdie
61fbff2cd4 bitbake: cooker/taskdata: Make NoProvider errors non-fatal to -e/-g options
If you have a situation where you are getting a "Nothing PROVIDES" error
(for example when something you request to build DEPENDS on something
that has been skipped or doesn't exist) it would be useful to be able to
use bitbake -g or bitbake -e to debug it, but currently both of those
are blocked by the error.

This patch adds an "allowincomplete" option to taskdata and uses this
for the -e/-g bitbake options. The NoProvider errors are still
printed and bitbake does return an error exist code but the environment
and task graph files are generated.

[YOCTO #7623]

(Bitbake rev: 2ca36a9f088438a8d1db44119c704f9480b04298)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-12 22:50:41 +01:00
Leonardo Sandoval
7bccf59a24 bitbake: cooker.py: Lock/Unlock members function into BBCooker
(Bitbake rev: d66dccf9f9a33bfef5c28cc1c767bfc89faee532)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-09 18:00:21 +01:00
Richard Purdie
25702f3a2f bitbake: cooker: Improve DATE/TIME handling
Currently, once set, DATE and TIME in the metadata remain unchanged.
This is suboptimal for cases where cooker is reused. This ties setting
the variables into the BuildStarted event which seems like a more
appropriate time to do so.

It also changes BUILDNAME to be based off DATE/TIME by default if not
already set so that the data is more consistent. We therefore need to
expand the value rather than the previous default of not doing so.

This change does mean the date/time values are in sync across all
variables too.

It does mean bitbake now has special knowledge of DATE/TIME but that
would seen unavoidable (other than doing this in event handlers which
has its own set of downsides).

[YOCTO #5187]

(Bitbake rev: f883cf240266ee7be2cbd8971a8164cf4df9e372)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-08 00:06:08 +01:00
Richard Purdie
00ca43be05 bitbake: cooker: Fire BuildCompleted before finishing the command
Some handlers hook on BuildComplete so it avoids certain event races
to finish the command after the BuildComplete event is sent out.

This means the UI is available to handle events until the command
completes.

What appears to be a race on one of the sanity tests for event handlers
triggered this change although the failure is hard to reproduce.

[YOCTO #7921]

(Bitbake rev: e42d7c47a06fbb5981e0313478c8e3656b99f4e7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:33 +01:00
Richard Purdie
69b6919341 bitbake: bitbake: Add explict getVar param for (non) expansion
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.

This patch was mostly made using the command:

sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:53 +01:00
Richard Purdie
4bedfeffa2 bitbake: cooker/event: Add an event which lists all stamps reachable after parsing
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.

(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-09 13:30:09 +01:00
Richard Purdie
08b77c8cef bitbake: cooker/utils: Improve parsing profiling
Currently the cooker parsing processes each dump an individual profile
which is ok, but means absolute numbers of function calls for a given load
can be tricky to determine as parsing of recipes may go to different pool
threads on different runs.

This change collects up the individual thread parsing results and processes
them into one profile output. The profile processing function in utils
needed tweaks to allow this to work.

(Bitbake rev: d3d2541aacd1ea560da0d8b25a3ea3f0563dee70)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-29 10:17:16 +01:00
Ed Bartosh
db409697db bitbake: bitbake: reset build mtime cache before the build
Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.

[YOCTO: #7562]

(Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-04-29 10:59:54 +01:00
Richard Purdie
3cb87724a5 bitbake: cooker: Ensure bbappend files are processed in a determistic order
self.appendlist is a dict and as such unordered. This can lead to cases
where appends with different names (e.g. x_%.bbappend vs. x_123.bbappend)
can be reordered in application which in turn reorders the variables
that those bbappend files might touch. Reorderd variables changes the sstate
cache signatures causing real world issues.

To avoid this, use a list for the append files instead.

This patch is conservative and just adds a new data structure alongside
the existing one and uses it to resolve the core issue. Later patches
(post release) can handle some of the wider but less problematic ones
(e.g. issues in bitbake-layers flatten).

[YOCTO #7511]

(Bitbake rev: f980f060cd0d1e7fe5011f3c325c1b254f05eccf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-31 22:39:21 +01:00
Richard Purdie
0467d53017 bitbake: cooker/server: Fix up 100% CPU usage at idle
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.

This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.

(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-10 10:37:28 +00:00
Alexandru DAMIAN
b611de9429 bitbake: cooker: make sure state is properly used to accept clients
This patch fixes a bug where if the build is force stopped,
subsequent clients cannot connect to the server due to
unnecessary limits on setFeature.

Additionally, we make sure that the state is properly reset
even if the BuildCompleted event firing excepts for some reason.

(Bitbake rev: 0b66b05169688aa4ddc4c54d175bb961b2f27fec)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-09 14:33:46 +00:00
Alexandru DAMIAN
a9cd657a01 bitbake: cooker: read file watches on server idle
The inotify facility monitoring changes to the config files
could be overwhelmed by massive changes to the watched files
while server is running.

This patch adds verification the notification watches to the
server idle functions, in addition to the cooker updateCache
command which executes only infrequently, thus preventing
overflowing the notification buffer.

[YOCTO #7316]

(Bitbake rev: 996e663fd5c254292f44eca46f5fdc95af897f98)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-09 14:33:43 +00:00
Paul Eggleton
c4da9b949a bitbake: cooker: rework LAYERDEPENDS versioning so that it is actually useful
We've had versioned dependency support in LAYERDEPENDS for quite a long
time, but I can say with pretty good certainty that almost nobody has
used it up to now because it was too strict - the specified version had
to exactly match the version in your configuration or you would get an
error; there was no "greater than or equal" option, which is usually
what you will want given that LAYERVERSION does get bumped from time to
time.

However, users mismatching layer branches and then having their builds
fail later on with some incomprehensible error is still a pretty common
problem. We can't simply use the git branch because not everyone is
always on a branch and the branch names don't always match up (and
that's not an issue). To provide a practical means to address branch
mismatching, I have reworked LAYERDEPENDS version specifications to use
the more familiar "dependency (>= version)" syntax as used with package
dependencies, support non-integer versions, and clarified the error
message a little. If we then take care to bump the version on every
breaking change, it is at least possible to have layers depend on these
changes when they update to match; we can now even support a major.minor
scheme to allow retrospectively adding a version limiter to old branches
when a new branch is created and yet still allow the old branch minor
version to be bumped if needed.

Fixes [YOCTO #5991].

(Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-02-16 09:10:42 +00:00
Richard Purdie
9538ee9191 bitbake: cooker: Use expanded_data for ASSUME_PROVIDED
Users expect operators like _remove to work on this variable. We need
to use expanded_data to ensure this happens correctly.

[YOCTO #7135]

(Bitbake rev: cc4c8478fc547ea0ebf827a8d319496b39f25684)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-23 11:33:55 +00:00
Richard Purdie
4eb3ffe571 bitbake: cooker: Rename event_data -> expanded_data
event_data would be better named expanded_data, then we can widen its scope
to other places in cooker where we need to access an expanded data store.
We certainly don't want multiple expanded data stores.

(Bitbake rev: 1a3c1c9203e1a1452314954f1cfd771e5c1ce89b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-23 11:33:54 +00:00
Robert Yang
c6f933e5cf bitbake: cooker.py: trap the ENOSPC error and translate it
Trap the ENOSPC error and translate it into a human readable error
message, which is good for debugging.

(Bitbake rev: 2b084dff6ff0d274fbbf7ab07022507f7249e427)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-21 14:37:39 +00:00
Richard Purdie
e63723f59a bitbake: cooker: Further optimise pyinotify
We currently add crazy numbers of watches on files. The per user limit is 8192
by default and on a system handling multiple builds, this can be an issue.

We don't need to watch all files individually, we can watch the directory containing
the file instead. This gives better resource utilisation and better performance
further reverting some of the performance regression seen with the introduction
of pyinotify.

(Bitbake rev: a2d441237916a99405b800c1a3dc39f860100a8c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-21 14:37:39 +00:00
Richard Purdie
db45ddeeaf bitbake: command/cooker/knotty: Fix memres handling of command environment changes
If the environment changes, we need memory resident bitbake to adapt to those
changes. This adds in functionality to handle this alongside the configuration
option handling code. This means that the common usage:

MACHINE=X bitbake Y

now works with the memory resident server.

(Bitbake rev: 4d1343010da757a0c126bc22475354da44aaf8e3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-21 14:37:39 +00:00
Richard Purdie
967f13fda0 bitbake: cooker: Improve pyinotify performance
Benchmarks show that the introduction of pyinotify regressed
performance. This patch ensures we only call the add_watch() function
for new entries, not ones we've already processed which does improve
performance as measured by "time bitbake -p".

This doesn't completely remove the overhead but it does substantially
reduce it.

(Bitbake rev: 493361f35f6cc332d4ea359a2695622c2c91a9c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-16 08:25:30 +00:00
Richard Purdie
eb7480fc4d bitbake: cooker: Fix pyinotify handling of ENOENT issues
We try and add watches for files that don't exist but if they did, would influence
the parser. The parent directory of these files may not exist, in which case we need
to watch any parent that does exist for changes. This change implements that fallback
handling.

(Bitbake rev: 979ddbe4b7340d7cf2f432f6b1eba1c58d55ff42)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-14 13:34:06 +00:00
Richard Purdie
6c3c3e11f6 bitbake: cooker/cache/parse: Implement pyinofity based reconfigure
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).

This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.

Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.

Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.

We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.

For education and QA purposes I can document a workflow that illustrates this:

$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]

As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.

The new pyinotify dependency is covered in the previous commit.

(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-14 11:32:02 +00:00
Richard Purdie
1393e224b5 bitbake: cooker: Shut down the parser in error state
If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.

[YOCTO #6934]

(Bitbake rev: 294bb9cad294423d4f8998405ceff58655f12660)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-08 09:42:24 +00:00
Paul Eggleton
d31b7bdf11 bitbake: cooker: add ability to ignore unmatched regexes in BBFILE_PATTERN
Add a BBFILE_PATTERN_IGNORE_EMPTY variable to allow ignoring the fact
that a regex specified in BBFILE_PATTERN for a particular collection
doesn't match any recipes. This will be used in OpenEmbedded in the
workspace layers created by "devtool" which may not always contain any
recipes (which is not cause for warning the user).

(Bitbake rev: 19c74aaa2836a88d666f0032452fac521689ab6b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-23 10:18:21 +00:00
Alexandru DAMIAN
85a17f86ea bitbake: add option to write offline event log file
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.

The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.

The "-w" option can also be supplied as the BBEVENTLOG
environment variable.

We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.

We modify the toasterui to fix minor issues with reading
events from an event log file.

Performance impact is undetectable under no-task executed builds.

(Bitbake rev: 1befb4a783bb7b7b387d4b5ee08830d9516f1ac2)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-18 10:24:06 +00:00
Richard Purdie
2fc06a12a1 bitbake: cooker: Allow featureset in error state
Currently, if an invalid PR service is selected the server will error
with a traceback. This is because its set into the error state and the
setFeature code will then fail since its not in the initial state.

Modifying the featureset in the error state is acceptable, we just need
to ensure we don't trigger a reset, that would happen from whichever
code handles the error.

[YOCTO #6934]

(Bitbake rev: c52841445d8db8f84c4da34203b195fea5874247)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-28 14:03:02 +00:00
Robert Yang
c09e56168e bitbake: bitbake-worker: exit normally when SIGHUP
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.

(Bitbake rev: 40d2ae0723de2bf5fee343faafb4afda40546839)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-20 14:15:32 +00:00
Brendan Le Foll
edaeb89408 bitbake: cooker.py: fix loginfo op being set to an invalid value
loginfo "op" was being set to invalid values in saveConfigurationVar it was
working because set is a python global but append is not. This replaces both
the "op" with strings and removes the crash when calling appendConfigurationVar

(Bitbake rev: 2a8e847de85546d43600a561f9c63aa36bd69222)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-12 17:04:50 +00:00
Richard Purdie
ff5fba8462 bitbake: knotty: Ensure commandline parameters are updated in memres server
When using options like -k, -f, -v and so on with the memory resident
server, they'd currently only be set on the initial values passed to
the original command. This ensures they now match those specified
on the commandline for the options where this makes sense.

To make this work, a command to update the options on the server side
is required so this is added.

[YOCTO #5292]

(Bitbake rev: 1c75cc4d0c8b606c1fe76e6bf60bf6a32298b105)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-09-23 20:06:06 +01:00
Bernhard Reutner-Fischer
21f41f6c43 bitbake: cooker: tweak CookerCollectFiles::find_bbfiles
since python-2.5 string.endswith() takes a tuple

(Bitbake rev: 86a67a1fd4244da9343dbf14deed1ad0d3003f32)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-23 23:02:00 +01:00
Namhyung Kim
f401e8f88c bitbake: cooker: Get rid of duplicate 'import re'
The re module was imported at the top, no need to do it again.

(Bitbake rev: c29c2370635075949c75937c237602600f4f10dc)

Signed-off-by: Namhyung Kim <namhyung@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-02 09:57:11 +01:00
Robert P. J. Day
653b557770 bitbake: "per-package" should say "per-recipe"
(Bitbake rev: 1cd369883469747a8158826bb8d67dcca2a8577f)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-07-23 10:02:13 +01:00