f1b0ab7e4b
The IPC mechanism between runqueue.py and bitbake-worker is currently not scalable: The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker has no information about the size of the message. Therefore, the bitbake-worker is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")" for each chunk received. This does not scale, because queue.find has a linear complexity relative to the size of the queue, and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI. The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup 35 seconds before this fix, and 1.5 seconds after this fix). This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be received, and does not need to constantly search the whole queue for </tag>. (Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake
BitBake is a generic task execution engine that allows shell and Python tasks to be run efficiently and in parallel while working within complex inter-task dependency constraints. One of BitBake's main users, OpenEmbedded, takes this core and builds embedded Linux software stacks using a task-oriented approach.
For information about Bitbake, see the OpenEmbedded website: https://www.openembedded.org/
Bitbake plain documentation can be found under the doc directory or its integrated html version at the Yocto Project website: https://docs.yoctoproject.org
Bitbake requires Python version 3.8 or newer.
Contributing
Please refer to our contributor guide here: https://docs.yoctoproject.org/dev/contributor-guide/ for full details on how to submit changes.
As a quick guide, patches should be sent to bitbake-devel@lists.openembedded.org The git command to do that would be:
git send-email -M -1 --to bitbake-devel@lists.openembedded.org
If you're sending a patch related to the BitBake manual, make sure you copy the Yocto Project documentation mailing list:
git send-email -M -1 --to bitbake-devel@lists.openembedded.org --cc docs@lists.yoctoproject.org
Mailing list:
https://lists.openembedded.org/g/bitbake-devel
Source code:
https://git.openembedded.org/bitbake/
Testing
Bitbake has a testsuite located in lib/bb/tests/ whichs aim to try and prevent regressions. You can run this with "bitbake-selftest". In particular the fetcher is well covered since it has so many corner cases. The datastore has many tests too. Testing with the testsuite is recommended before submitting patches, particularly to the fetcher and datastore. We also appreciate new test cases and may require them for more obscure issues.
To run the tests "zstd" and "git" must be installed.
The assumption is made that this testsuite is run from an initialized OpenEmbedded build
environment (i.e. source oe-init-build-env is used). If this is not the case, run the
testsuite as follows:
export PATH=$(pwd)/bin:$PATH
bin/bitbake-selftest
The testsuite can alternatively be executed using pytest, e.g. obtained from PyPI (in this case, the PATH is configured automatically):
pytest