7afa726bec
The original implementation of persistent data executed all SQL
statements via sqlite3.Connection.execute(). Behind the scenes, this
function created a sqlite3 Cursor object, executed the statement, then
returned the cursor. However, the implementation did not account for
this and failed to close the cursor object when it was done. The cursor
would eventually be closed when the garbage collector got around to
destroying it. However, sqlite has a limit on the number of cursors that
can exist at any given time, and once this limit is reached it will
block a query to wait for a cursor to be destroyed. Under heavy database
queries, this can result in Python deadlocking with itself, since the
SQL query will block waiting for a free cursor, but Python can no longer
run garbage collection (as it is blocked) to free one.
This restructures the SQLTable class to use two decorators to aid in
performing actions correctly. The first decorator (@retry) wraps a
member function in the retry logic that automatically restarts the
function in the event that the database is locked.
The second decorator (@transaction) wraps the function so that it occurs
in a database transaction, which will automatically COMMIT the changes
on success and ROLLBACK on failure. This function additionally creates
an explicit cursor, passes it to the wrapped function, and cleans it up
when the function is finished.
Note that it is still possible to leak cursors when iterating. This is
much less frequent, but can still be mitigated by wrapping the iteration
in a `with` statement:
with db.iteritems() as it:
for (k, v) in it:
...
As a side effect, since most statements are wrapped in a transaction,
setting the isolation_level when the connection is created is no longer
necessary.
[YOCTO #13030]
(Bitbake rev: e8b9d3f534ef404780be23b601d5a4bb9cec928a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake
BitBake is a generic task execution engine that allows shell and Python tasks to be run efficiently and in parallel while working within complex inter-task dependency constraints. One of BitBake's main users, OpenEmbedded, takes this core and builds embedded Linux software stacks using a task-oriented approach.
For information about Bitbake, see the OpenEmbedded website: http://www.openembedded.org/
Bitbake plain documentation can be found under the doc directory or its integrated html version at the Yocto Project website: http://yoctoproject.org/documentation
Contributing
Please refer to http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded for guidelines on how to submit patches, just note that the latter documentation is intended for OpenEmbedded (and its core) not bitbake patches (bitbake-devel@lists.openembedded.org) but in general main guidelines apply. Once the commit(s) have been created, the way to send the patch is through git-send-email. For example, to send the last commit (HEAD) on current branch, type:
git send-email -M -1 --to bitbake-devel@lists.openembedded.org
Mailing list:
http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Source code:
http://git.openembedded.org/bitbake/