When in VariableParse, use the expand_cache if possible rather than looking
up data. Ultimately it would come from the same place but this short cuts
a heavily used code block for speed improvements.
(Bitbake rev: f682b8b83d21d576160bac8dc57c4c989b4dc555)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debugging showed the variable expansion regexp was catching python
expressions (starting with @). Since these are caught by their own
dedicated regexp, stop matching these for the plain variable expansion
for small performance improvements.
(Bitbake rev: c630d564285f55f9db10c18269bd310df797430e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All the values we need are already guaranteed to be in the lookupcache
so rather than fetch variables again, just use the cache. This gives a
small performance improvement and simplifies the code.
(Bitbake rev: 8ffaba61da7f195d7c3b64dce35b6a56272aecae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The if statement current causes the size of parent to be calcuated which
is like a len() operation on a datastore. Since we're only interested
whether the value is none, checking explictly for this gives a
small performance gain.
(Bitbake rev: 43a245bde318545ea75ca4ce7894395c1cf9b32a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean-up to avoid duplication and promote code reuse to factor
taskdata creation into a common function.
[RP: minor tweaks]
(Bitbake rev: 468c221449290c4f196e87f7d8e23fcd7db86135)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a runQueueTaskSkipped to notify that the tasks that are not
run either because they are set-scened or they don't need an update
(timestamp was ok).
(Bitbake rev: cf4a0c7aa82090876ae652b611acfab3ce2688f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the path to the logfile for all Task events except TaskInvalid
so that we can trace back the logfile locations at some future point.
TaskInvalid doesn't ever have a logfile.
(Bitbake rev: 8344d84c609446f59f9619cc7ca0d693b7e2bbd6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exiting from the server is antisocial, instead we should raise an exception. This
will correctly fail the current command and reset the server state. We use
the handled exception since for these conditions to occur, something was
already displayed to the user.
(Bitbake rev: dacc94bcace85a2e95aee2dccd8e680c59e4545f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only display a CommandFailed ERROR: message if there is an error to display.
Only display an errors summary if we actually displayed errors.
(Bitbake rev: 568ea00acd226d48e725bb01d4f8c410ed1eaa61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Empty messages should trigger CommandFailed, not CommandCompleted as
otherwise the exit code will be incorrect.
(Bitbake rev: 70a8ead31f9ffc987d9c6db61a926f7a9af8f8b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After running a command on the server, it needs to reset to the initial
state. This ensures that subsequent clients start from a known state
and notice any configuration changes.
Ultimately we may want to do more than this buts a good start and better
than nothing.
(Bitbake rev: dd15648fc2654b8d7c3e00ea7ab3dbf04f24f24b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shutdown state causes the server to finish what its doing, stop was
them meant to completely stop it. It doesn't mean the server is stopped
though. Renaming the current stop event for forceshutdown gives more
meaning to what it actually does. The stopped namespace then becomes
available to indicate a completely stopped server.
(Bitbake rev: 12e9d33bfae5294e3870dfd1202f63383ad05e92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cooker event data isn't rebuilt upon reset and the cache
configuration cannot be changed after init. These are both bad things
and this patch refactors the init/reset code so that it is possible
to reconfigure the server.
(Bitbake rev: 1193b8d76fcb6cb87e9ec135a2514370d7dd90ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This call only ever happens in cooker context now so we can drop the
nasty worker check from here.
(Bitbake rev: bc0b30199a8e3624c5b9914430adbcc7c6bd4497)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BitBakeUIEventServer is an unused class that pushes UI
events over a separate thread.
The current version of XMLRPC server works just fine with
the classic UI event handlers, so this class is not needed.
(Bitbake rev: 8e8e17631d790271b1be747c4b45059ec38ab606)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake wasn't reparsing when _remove items were added to its configuration
and equally, appends/prepends were also being badly tracked. This
change enrures these variables are accounted for in the configuration
hash.
[YOCTO #5172]
(Bitbake rev: 62914f9208ef2427a34daa523af857f4027900eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_PRESERVE_ENV wasn't working since data.inheritFromOS wasn't getting a
correct list of keys to import into the data store. This fixes
things so it does add all environment variables into the data store
when BB_PRESERVE_ENV is used.
(Bitbake rev: 843e9339c5ee3c99657a40a0e2c7dbd777b6ef06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding an event to be fired when a scene task is completed.
It is analogous to the run task completed event, and has
been missing for some reason.
(Bitbake rev: 73b8f4d3fbeaf1b330a66d76012d0a5cef8dbe2d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There would be an race issue if we:
$ bitbake make-3.81 make-3.82
This because they are being built at the same time which would cause
unexpected problems, for example:
[snip]
ERROR: Package already staged (/path/to/tmp/sstate-control/manifest-qemux86-make.populate-sysroot)?!
ERROR: Function failed: sstate_task_postfunc
[snip]
Or there would be python's strack trace such as:
[snip]
*** 0004: mfile = open(manifest)
0005: entries = mfile.readlines()
0006: mfile.close()
0007:
0008: for entry in entries:
Exception: IOError: [Errno 2] No such file or directory: xxx
[snip]
[YOCTO #5094]
We can quit earlier to avoid this kind of issue when two versions of the same PN
are going to be built since this isn't supported.
(Bitbake rev: ab377c00c33a2d296bfda1b0b6c2a62b29d1004f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runtime provider debug message is the same as the build time debug
message, make them different would be better.
[YOCTO #5067]
(Bitbake rev: 92b624cbc2711d3d859994099fb63918dfd0031a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In exclusive mode, we need to complete the transaction for writes to make
it to the database. Therefore add sync calls to ensure this happens.
Autocommit mode is significantly (100 times) slower so caching the
data is of significant benefit.
(Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When displaying larger number of events the client can get caught up in displaying
the footer, then immediately overwriting it. To avoid this, wait for pauses
in the event stream before displaying the footer to give a slightly more
friendly feel to the UI.
(Bitbake rev: 5d706c7cd6ee8d83b67ff18312d4c8119bea8878)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to the worker split the ${DATE} and ${TIME} variables could end up
with different values for different workers.
E.g., a task like do_rootfs that is run within a fakeroot environment
had a slightly different view of the time than another task that was not
fakerooted which made it impossible to correctly refer to the image
generated by do_rootfs from the other task.
(Bitbake rev: 756cc69ebf8bfe8455d0c90f288dd51be2499773)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Selected custom image recipes should be cleared from the combo the moment
you change your selection. The idea is to always perform the selection of
those images in the same way (i.e through the "Select from my image
recipes" option).
[YOCTO #5001]
(Bitbake rev: 94483ee5ae9f4051bccd660c4718c36564e17161)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DISTRO_FEATURES_remove = "opengl" wasn't working as expected. The reason
turned out the be the indirect reference to opengl and the fact _remove was
operating on unexpanded data.
This patch rearranges some code to ensure we operate on expanded data
by moving the expand cache handing into getVarFlags instead of getVar.
(Bitbake rev: 181899bd9665f74f8d1b22d2453616ad30d26d9e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before saving the packages for a custom images in a .bb file,
the packages were saved in bitbake memory. Now all the variables
are saved in conf file, so saving PACKAGE_INSTALL is not needed anymore.
Moved were LINGUAS_INSTALL is set, because both conditions are for testing
if a custom image is saved.
[YOCTO #5101]
(Bitbake rev: 8757f962b92e7668f40d2d8bd9e762b152f91f7b)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BUILDNAME is set from cooker by default, so since the worker split it
will not be set when executing functions. In OpenEmbedded this results
in /etc/version (which is populated from BUILDNAME) not having any
content. Pass this variable value through to the worker explicitly to
fix the issue.
Fixes [YOCTO #4818].
(Bitbake rev: 92940b0427d9b2b3f95e27c230ec1e36638a34bc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cookerdata.findconfigFile method has a new parameter. Changed some calls.
(Bitbake rev: dce0f9d4afe0986e2dd0146944fc4ac9dde275e4)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was missed off in a previous patch.
(Bitbake rev: ad7664edd40fa46e6f6fec2144403e3b6fc3a639)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server only expects a single client to connect (i.e.
no bind parameter set, so there is no way for the
clients to get the server port), stop the server after
the first client exits.
(Bitbake rev: eb6bae56f62082bf147045311154cbae4bca0f4c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using the threading mixin class resulted in large amounts of memory
being used by the PR server for no good reason. Using a receiver thread
and a thread to do the actual database operations on a single connection
gives the same performance with a much saner memory overhead so
switch to this.
(Bitbake rev: e08455d5f3b8e96765942b9c3b9767c30650557d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using the basename of the .bb file is not unique, for example xxx-native
and xxx can overwrite each other. If this happens whilst running, you can
get odd backtraces as one file is parsed as another tries to write out
new data.
Avoid issues by using PN for the output filename instead.
(Bitbake rev: c9534f8e59d44b885334607ed90a3be2e492ec69)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we don't support using multiple servers on the same database file,
don't use the BEGIN/COMMIT syntax and allow writes to the database
to work ~100 times faster with no transaction locking.
(Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only support one server using the database at a time so take an exclusive
lock and avoid later lock overhead.
(Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the database is locked we will get an immediate error indicating so,
there is no retry timeout. The looping code is therefore useless, the loop
count is near instantly exceeded.
Using a time based retry means we can wait a sensible time, then gracefully
exit.
(Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing backend server implementations were inefficient since they
were sleeping for the full length of the timeouts rather than being woken when
there was data ready for them. It was assumed they would wake and perhaps did
when we forked processes directory but that is no longer the case.
This updates both the process and xmlrpc backends to wait using select(). This
does mean we need to pass the file descriptors to wait on from the internals
who know which these file descriptors are but this is a logical improvement.
Tests of a pathaolgical load on the process server of ~420 rapid tasks
executed on a server with BB_NUMBER_THREAD=48 went from a wall clock
measurement of the overall command execution time of 75s to a much more
reasonable 24s.
(Bitbake rev: 9bee497960889d9baa0a4284d79a384b18a8e826)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist"
by retrying if we see the string "is locked".
(Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a better value that the earlier infinite timeout yet still
allows for servers with high loads. It does mean the bitbake process
can hang at exit for the timeout period but that should never happen
and only happened for me in some test cases which wouldn't happen
in normal use.
(Bitbake rev: ab8d926b9bc27c58011e7db9327e031ac76ba34b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It should be possible to run a build anywhere on the filesystem and have
bitbake find the correct build directory if its set somehow. The BBPATH
variable makes perfect sense for this usage. Therefore use any available
value of BBPATH to search for conf/bblayers.conf before walking the parent
directory structure.
This restores the option of being able to run bitbake from anywhere if
the user has set things up to operate in that environment.
(Bitbake rev: e86336b3fe245bc97fe74c9b9d6a21d38a536fb7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By definition, bblayers.conf is at the top of the build tree. We'd like
to support running bitbake anywhere within that build tree but TOPDIR
gets set to wherever cwd is. Change the code to reset TOPDIR
to the top of the build directory.
This shouldn't break anything but does make the system more usable.
(Bitbake rev: b266db27de0bba19a418e4d42e870649136b116b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes the PR server multithreaded and able to handle multiple connections
at once which means its no longer a build bottle neck when serving one connection
at a time. I've experimented and database connection for each thread seems to
cause the least issues, pushing the contention for sqllite to handle itself.
This means moving the db/table connection code into the actual function methods.
It doesn't abstract well as a function since we need the db object around for
the lifetime of the function as well as the table else we lose the connection.
(Bitbake rev: bf9be2029b2bded5f532bdda4c38ae3dff5d1cf6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enabling threading for the PRServer causes a number of issues. Firstly is
the obtuse error:
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type
which is due to the class not being derived from object. See:
http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable
Secondly, we want to enable multithreadded access to the database so we do this
when we open it. This opens the way up to multithreading the PR server.
(Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
commit 972bc43e6d5b1207b944b3baa8f9805adb35dda7 (serv.py: Fix hang
when spawned dynamically with bitbake) introduced a regression,
because the wrong patch was submitted. The syntax was incorrect in
the original patch. The logger iterator must be used with a call to
getLogger().
[YOCTO #5059]
(Bitbake rev: 85fed8acc3af3e15bf119db2f51c486a9de3646b)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The PRServer has the possibility to hang indefinitely blocking on a
semaphore processing a xmlrpc request to send an event back to the
main bitbake instance. This was observed during a "bitbake -e" on a
heavily loaded machine and the main bitbake instance and cooker exited
before the PRServer emitted its first log.
The stack trace is provided below as to show what happens every time a
logger.info() is executed in the PRServer. Not only does it write to
the stream handler but it also tries to send the event to the main
event processor.
self._notempty.acquire()
self.queue.put(event)
_ui_handlers[h].event.send(event)
fire_ui_handlers(event, d)
fire(record, None)
self.emit(record)
hdlr.handle(record)
self.callHandlers(record)
self.handle(record)
self._log(INFO, msg, args, **kwargs)
(self.dbfile, self.host, self.port, str(os.getpid())))
self.work_forever()
pid = self.daemonize()
self.prserv.start()
singleton.start()
self.prhost = prserv.serv.auto_start(self.data)
cooker.pre_serve()
bb.cooker.server_main(self.cooker, self.main)
self.run()
code = process_obj._bootstrap()
self._popen = Popen(self)
self.serverImpl.start()
server.detach()
server = start_server(servermodule, configParams, configuration)
ret = main()
It was never intended for the PRServer to send its logs anywhere but
its own log file. The event processing is an artifact of how the
PRServer was forked and it inherits the event log handlers. The
simple fix is to clean up and purge all the log handlers after the
fork() but before doing any of the typical PRServer work or logging.
(Bitbake rev: 972bc43e6d5b1207b944b3baa8f9805adb35dda7)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an image from scratch is selected, and recipes parsing
is canceled, the image shown by the combo box isn't correct.
[YOCTO #5000]
(Bitbake rev: f8166ace0bd9155199166990ce15da24eb2e793b)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create the _evt_list for hob; it is longer than the knotty
uses because it handles more events.
(Bitbake rev: 715aed74f972bb6e9b6a5130ca9ede48d4f79f0a)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
