mirrors/poky
1
0
Fork 0
mirror of https://git.yoctoproject.org/poky synced 2026-04-04 23:02:22 +02:00
Code Issues Packages Projects Releases Wiki Activity

bitbake: cooker: Rework the parsing results submission

Browse Source 
The loop submitting the parser results was not efficient on many core systems
as may of the parsers could block for a long time waiting to send the results.
While a result was pending, the code wouldn't parse further jobs.

Change the code to parse further jobs even if the results can't be submitted
and lower the result submission timeout to keep the parsers busy.

(Bitbake rev: 9ece4a2e406509bd89dbce8dac80a02e600b0ecf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Richard Purdie
2023-01-04 17:47:44 +00:00
parent 69abe10566
commit 9043034da5
1 changed files with 15 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
bitbake/lib/bb/cooker.py
View File

@@ -2107,25 +2107,29 @@ class Parser(multiprocessing.Process):
multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1)
pending = []
havejobs = True
try:
while True:
while havejobs or pending:
if self.quit.is_set():
break
if pending:
result = pending.pop()
else:
try:
job = self.jobs.pop()
except IndexError:
break
job = None
try:
job = self.jobs.pop()
except IndexError:
havejobs = False
if job:
result = self.parse(*job)
# Clear the siggen cache after parsing to control memory usage, its huge
bb.parse.siggen.postparsing_clean_cache()
try:
self.results.put(result, timeout=0.25)
except queue.Full:
pending.append(result)
if pending:
try:
result = pending.pop()
self.results.put(result, timeout=0.05)
except queue.Full:
pending.append(result)
finally:
self.results.close()
self.results.join_thread()