Update the description to align with the current goals of the dtc project.
(From OE-Core rev: cfe4b6b4559191a7986ba48f54a1d0350705a86d)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was missed when riscv64nc was added
(From OE-Core rev: 0c549ef5732afdcd96407ceb045983eed2ca75f4)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently binutils in buildtools is searching for /etc/etc/ld.so.conf
which makes no sense. ld_sysconfdir already contains /etc so we need to
drop the /etc from the fixed string.
(From OE-Core rev: ccd28c418ab8390118d738fbe914395b5c2a1f75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add in the new 'check' calls which mean that the code can exit from longer
running code loops if requested by the user. This covers sstate checks and
sstate manifest cleanup code which currently couldn't be interrupted by the
user from the UI.
Increase the minimum bitbake version which brings in this API.
(From OE-Core rev: cb6c50689175668b66ffbe09ff8f250ba9a9034b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that OE-Core can depend on bb.event.check_for_interrupts(), bump our
verison number to a development series version.
(Bitbake rev: dea1b2f3fc31f28daed5da16e62da8895d6e5716)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake is now able to understand when a UI wants it to stop the current
processing. There are some long running functions which currently have no
mechanism to interrupt them however.
This patch adds a call, bb.event.check_for_interrupts(d) which can be
placed in long running code and allows an internal state flag within
bitbake to be checked. If set, that flag will trigger an exit.
This means that Ctrl+C can be made to work in a variety of places where
it currently would not.
Long running event handlers in OE-Core can also then benefit from this
new approach with the addition of the function call as well.
(Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The xattr DISTRO_FEATURE is required otherwise a bitbake build will fail due
to a failure with the 'cp' utility:
Subprocess output:
cp: cannot preserve extended attributes, cp is built without xattr support
ERROR: Logfile of failure stored in: /home/builder/poky/build/tmp/work/qemux86_64-poky-linux/packagegroup-base/1.0-r83/temp/log.do_populate_lic.1001
ERROR: Task (/home/builder/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_populate_lic) failed with exit code '1'
(From OE-Core rev: 7e1b2598fedfdc7c9a79c5c94273218f2eedcba9)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow a user to override the QM_SMP value giving them the opportunity to
select for themselves the number of CPUs to use in qemu.
(From OE-Core rev: 70a91e6d0357149c00b97f7e66e16cbc52997a92)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the user to specify a QB_MEM value of their choosing, otherwise set it
to a default value.
(From OE-Core rev: 2f8c20ab6750bd900b28e6468493cbd010144050)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a /lib64 -> /lib symlink in case the build appliance is 64-bit.
Building rust-native requires cargo to be run. The native cargo is linked to
look for /lib64/ld-linux-x86-64.so.2 and can't be run without it. The build
produces the following error, even though that file does exist:
FileNotFoundError: [Errno 2] No such file or directory: '/home/builder/poky/build/tmp/work/x86_64-linux/rust-native/1.66.1-r0/rust-snapshot/bin/cargo'
Trying to run this binary on its own gives:
qemux86-64:~/poky/build$ tmp/work/x86_64-linux/rust-native/1.66.1-r0/rust-snapshot/bin/cargo
-sh: tmp/work/x86_64-linux/rust-native/1.66.1-r0/rust-snapshot/bin/cargo: cannot execute: required file not found
(From OE-Core rev: a62b32907cb059b4512032f15571705540f1c147)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the taskstats configuration to any kernel built for qemuall for bitbake's
benefit:
WARNING: The Linux kernel on your build host was not configured to provide process I/O statistics. (CONFIG_TASK_IO_ACCOUNTING is not set)
(From OE-Core rev: 351f82b9918188eb4dbb6598733da5f5392fbd8f)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Choose a nicer terminal for the build appliance so that build warnings and
errors show up in colour instead of:
$<50>Setscene tasks: 2844 of 2844
$<50>WARNING: mesa-2_22.3.3-r0 do_fetch: Failed to fetch URL https://mesa.freedesktop.org/archive/mesa-22.3.3.tar.xz, attempting MIRRORS if available
$<50>WARNING: glslang-native-1_1.3.236.0-r0 do_fetch: Failed to fetch URL git://github.com/KhronosGroup/glslang.git;protocol=https;branch=master, attempting MIRRORS if available
(From OE-Core rev: ab0e748c5652a86d6c535c1f0979aa7d3d367d6e)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add missing tools necessary on the host for a build.
(From OE-Core rev: 15ba0ced2e8d4c2f9d6889b9b84c5525bc451e3f)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Several of the lists of packages are alphabetized, but the alphabetization
was lost in a couple areas.
(From OE-Core rev: 4965a14dd51242d4d5ecc4b5f288d0c42bafae18)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
qemu doesn't always behave well and can hang too.
kill it with force if it was still alive. Move clean up
commands into cleanup() function.
(From OE-Core rev: 079c2935d2f585ce49e1c7daab2155fcf0094c48)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When qemu machine hangs, the ssh commands done by tests
are not timing out. do_testimage() task has last logs like this:
DEBUG: time: 1673531086.3155053, endtime: 1673531686.315502
The test process is stuck for hours, or for ever if the
executing command or test case did not set a timeout correctly.
The default 300 second timeout is not working when target hangs.
Note that timeout is really a "inactive timeout" since data returned
by the process will reset the timeout.
Make the process stdout non-blocking so read() will always return
right away using os.set_blocking() available in python 3.5 and later.
Then change from python codec reader to plain read() and make
the ssh subprocess stdout non-blocking. Even with select()
making sure the file had input to be read, the codec reader was
trying to find more stuff and blocking for ever when process hangs.
While at it, add a small timeout to read data in larger chunks if
possible. This avoids reading data one or few characters at a time
and makes the debug logs more readable.
close() the stdout file in all cases after read loop is complete.
Then make sure to wait or kill the ssh subprocess in all cases.
Just reading the output stream and receiving EOF there does not mean
that the process exited, and wait() needs a timeout if the process
is hanging. In the end kill the process and return the return value
and captured output utf-8 encoded, just like before these changes.
This fixes ssh run() related deadlocks when a qemu target hangs
completely.
(From OE-Core rev: 9c63970fce3a3d6029745252a6ec2bf9b9da862d)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to using mkdtemp instead of TemporaryDirectory we needed to
manually cleanup the directory in a try finally block.
With tempfile.TemporaryDirectory we can handle the cleanup with
a "with" statement and not need to manually clean up oursevels.
(Bitbake rev: 36a231d2a45e7c87172196538d18ded81a234774)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
branchname was set but never used in the context of _contains_lfs
method.
(Bitbake rev: a64ec64c2faf1ac6ce995cdd4e311d8be6046155)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.
The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.
lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.
lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.
lfs=0 actually disables git-lfs functionality even if supported.
(Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not restoring the mocked _find_git_lfs leads to other tests
failing.
(Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the inotifier code has an exception, bitbake currently hangs. Catch any
exception and exit if seen. Also check the idle thread is alive and exit
if it disappears. This should stop bitbake hanging if such a situation arises
in future such as this example:
3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
self.cooker.process_inotify_updates()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
n.read_events()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)
(Bitbake rev: 358b5b02d5de1ab0f98104c4ec4953e46999b9a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen a couple of cases which bitbake hangs due to an inotifer exception
such as:
3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
self.cooker.process_inotify_updates()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
n.read_events()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)
Ensure we don't destory the inotifier when the idle thread is reading is
by holding the lock during setup/teardown.
(Bitbake rev: 8fc5c50c2e23017833f93bcd514d708a14fa4266)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
