bitbake: bitbake-worker: Improve bytearray truncation performance

If there are large amounts of data being transferred to the cooker
from the worker, recreating the bytearray becomes inefficient as it
happens for every pipesize block of data, defaulting to 64kb.

Instead we can use the deletion API for bytearrays to make this more
efficient and avoid the object recreation.

We noticed this with a strace ptest image taking days to complete the
build after having 6GB of data in the testimage log. Whilst there are
other issues there, making this code more efficient doesn't hurt.

(Bitbake rev: a4a72b7edb368f352784c856a647236a887010dd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Richard Purdie
2024-11-19 21:40:06 +00:00
parent 17f1b80c06
commit b97de63d2f

View File

@@ -105,7 +105,7 @@ def worker_flush(worker_queue):
if not worker_queue.empty():
worker_queue_int.extend(worker_queue.get())
written = os.write(worker_pipe, worker_queue_int)
worker_queue_int = worker_queue_int[written:]
del worker_queue_int[0:written]
except (IOError, OSError) as e:
if e.errno != errno.EAGAIN and e.errno != errno.EPIPE:
raise