If you send this forked process a SIGTERM, it will execute all of the
parent's exit code leading to two sets of console/exit output which is
extremely confusing. Wrap the code in a try/finally to ensure we always
call os._exit() to avoid this.
I spent far too long trying to work out the crazy console output from this.
(From OE-Core rev: 652e40bfae24b8e23bbf7a7f35d900d2ab8d0f92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ifconfig is obsolete and being removed, convert to use ip instead.
(From OE-Core rev: 3b2fb477750606976359884b18c33a37832e5a78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid "RuntimeError: release unlocked lock" since the lock shouldn't
be locked even in the error path. Add a try/finally path to ensure
this.
(From OE-Core rev: b0732ee009ca47580d1d2ad75334f4aa50e6efd5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows defining non-standard patterns from QEMU tests just as they
are already supported by the testimage.bbclass.
Will allow testing non-Linux shells in QEMU, too (e.g. a U-Boot shell or
another bootloader shell).
(From OE-Core rev: f9e6c3deee5a4159a09396b625a7327f6c204df5)
Signed-off-by: Enrico Jorns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we talk to terminals that like colors, we need to ignore the vt100
escape sequences when matching strings.
An unprocessed barebox console prompt would e.g. look like:
ESC[1;32mbarebox@ESC[1;36mARM QEMU virt64:/ESC[0m
where we cannot match for something like "barebox@ARM QEMU virt64:/".
The same applies to colored Linux terminal output of course.
The "\x1b\[" from the regex catches the standard start of ANSI escape
sequence while the rest catches the actual command code executed.
(From OE-Core rev: 33bbe4cb040f890121681865fbcf28bc8213a170)
Signed-off-by: Enrico Jorns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A change in QEMUMonitorProtocol.cmd() requires that we either pass in
kwargs instead of an argument dict or move to cmd_raw()
cmd() was renamed to cmd_raw() (and command() was renamed to cmd())
See:
37274707f6684750ab4f
My concern with this patch is that I haven't seen this come up with
utils/dump.py which also uses QemuMonitor's run_monitor. If it is
occuring, this should fix issues there as well
(From OE-Core rev: 9665d38ab60c1c3b27887c2b1a6396f13a1b33ea)
Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <pidge@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The pid file can disappear when qemu is shutting down leading to a
file not found race before it is read.
Tweak the code to handle this and fix a rare but annoying race error
case.
[YOCTO #15036]
(From OE-Core rev: 8c07aac9d55f92fe5fbe3cab9f006efecf266328)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we are only tracking stdout and are not using self.readsock we end
up throwing an exception blocking further action from the thread. Fix
this by checking self.readsock is not None first.
While we are at it split even into fd, event to make things clearer
and handle the fail path of stringify_event by echoing the hex value
of the unknown flag.
(From OE-Core rev: 5e58737c66090fe009ec49296f3e7d687eb05766)
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Mikko Rapeli <mikko.rapeli@linaro.org>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similar to stdout in the previous commit, we need to ensure serial output
if written is read and put somewhere, else qemu might block on writes to
the serial port leading to hangs in the kernel. Use our existing logging
thread to log data when run_serial is not in use.
(From OE-Core rev: 05761282ba31e4ba3594f7321e2162d01fe12a5f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to ensure we read from and log the output from qemu stdout
as otherwise the buffers can fill and block, leading qemu to hang.
Use our existing logging thread to do this.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
(From OE-Core rev: a9c46ee014ef1e6436b39fdd4fd15d15388ea795)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop the newlines serial workaround, it doesn't seem to fix things
enough of the time to be useful.
(From OE-Core rev: 6cd57b2d148c5de7839d52d8cec359e4a10e0cd6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the BrokenPipeError occurs when writing to the serial port to wake it
up, defer the write and try again (which will happen on the 5s timeout
of the select call). Why it should return ESHUTDOWN and then work later
I'm not sure but it does appear to make it work. For now we need 'working'
QA tests whilst the issue is debugged.
(From OE-Core rev: ea9e6ba0ab31a0b20012c283aa768496a50b527a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the console doesn't appear, breifly sleep and try an extra read. This
is a useful debugging trick which we may as well preserve in the code. We're
already failing at this point so extra data is useful.
This means if we perturb things on the other port, we may gain useful logging
insight.
(From OE-Core rev: be3724854947ded160789d0353b7efb8c593040f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This disables Nagle's algorithm for our tcp serial connections which may
be causing data transfer issues.
(From OE-Core rev: f8eff4c427881a98333fdf7c42f66ed6603e4f03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're struggling with the 6.5 kernel as the serial port getty doesn't appears sometimes
leading to failures in CI. Add a workaround of sending some newlines as a way of
unblocking the kernel/release issues whilst we try and work out how to get to the bottom
of the issue.
(From OE-Core rev: 0a65f0d272895ba13c8c133ee71f3605d765a8a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In case utf-8 conversion of the serial console data is flaky
due to bad characters in the stream.
(From OE-Core rev: 3f4921ced1e2aae546ff7cef232eec3e214c28be)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is unclear when things fail which output was on which serial port. Improve the output
to show the last lines of both data to improve debugging.
(From OE-Core rev: 605938b3bb19dcf6c7218648b4d16df9eaa675fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To aid debugging, always log the second serial console as well as the first
to a seperate log file. This should make it clearer what happened when we
see test failures.
(From OE-Core rev: 372cd58fe672726900af80f3e65a108984da2750)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch to the backslashreplace error handling when decoding strings so that
invalid characters are clear in the stream to improve debugging.
(From OE-Core rev: 8ec86d46f736ec3e625b741e97545377c79414ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no point in decoding binary data only to encode it again risking
conversion issues. Write the raw data to the log file as binary and skip
the conversion.
Also always update self.msg even if a logfile isn't specified to improve
logging/debug.
(From OE-Core rev: b5d67471dddd198723c4f711747783a33e8e5987)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The intent behind these functions was to dump the system state when issues occured
but it has never really worked as we'd planned. Regular monitoring as the build
runs has largely replaced this as that allows a trend to be seen rather than a spot
value which was never really useful. The code is bitrotting and not functioning
correctly so drop it.
[YOCTO #13872]
RP: Reword commit message
(From OE-Core rev: dea37ba49a236029da73d5cfbfc069bffc38b508)
Signed-off-by: Thomas Roos <throos@amazon.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The error message is a little misleading as the qmp module is a
directory with __init__.py file, not qmp.py file. Also, put the
path where we try to import it from in the error message to make
the message more indicative.
(From OE-Core rev: 08bacbf797f6a50ae8abe8fc3455b3a15a0a94b3)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Complex boot sequences take a lot of time to reach serial console login.
In our case boot involves multiple bootloaders like u-boot and grub,
then hypervisors like xen, then initramfs images which do things like
validating secure boot and encrypting the main rootfs. All these happen
before main rootfs serial console login prompt is reached and thus many
things can go wrong, including race conditions, hangs and deadlocks, and
thus logging the details is important for development.
This change adds the qemu serial console logs to do_testimage task logs while
they are being read out. The logs are not added to bitbake output. This
enables debugging and tracing the do_testimage via task log file while the boot
and tests are running.
(From OE-Core rev: cb62fae6ea3e411643395b12e56c363ce4d73e5d)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
runqemu script works with qemu machines which provide login
and serial console to the qemu process stdout. Add the same support
to qemurunner so that testing with testimage.bbclass is possible.
Default qemu machines provide serial console boot logs and login via
socket to qemu process but I don't see a reason why qemu process stdout
should not be supported too since they work with runqemu as well.
(From OE-Core rev: a3d4b80c5b4cce933c759d023c75b8671c56fe12)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* There is no need to be that precise. It just irritates in the logs.
* There is also no point in printing plain time.time() value a single
time while only using formatted printout everywhere else, thus remove
it.
* Use %d for printing integer times
(From OE-Core rev: f0988cb8cf8d03708490cca4eba345492ef78d52)
Signed-off-by: Enrico Jorns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since login user is configurable with boot_patterns['send_login_user'],
unconditionally using 'root' in the debug message can be confusing.
Also fix the debug message to say 'Logged in' instead of 'Logged'.
(From OE-Core rev: 900e3d42b918b5a33d8d952b3a8078fbe72ba98f)
Signed-off-by: Enrico Jorns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Read from serial console with a small delay to bundle data to e.g.
full lines. Reading one character at a time is not needed and causes
busy looping.
(From OE-Core rev: 0049f6757f6f956fb4cc77b3df6a672c20b53cf4)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a qemu machine hangs, the QMP calls can hang for ever
too, and when this happens any failing test commands from ssh
runner may be followed by dump_monitor() calls which
then also hang. Hangs followed by hangs.
Use runqemutime at setup and run_monitor() specific timeout
for later calls.
(From OE-Core rev: 3a07bdf77dc6ecbf4c620b051dd032abaaf1e4ff)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a short sleep to bundle serial console reads so that
we are not reading one character at a time which reduces busy
looping.
(From OE-Core rev: cafe65d8cf7544edbd387f7f5f6d77c64c6b18fa)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This does not actually guarantee that the child runqemu process has completely exited:
poll() may return prematurely while the SIGTERM handler in runqemu is still running.
This thwarts the rest of the processing, and may terminate the handler before
it completes.
Use Popen.communicate() instead: this is what python documentation recommends as well:
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate
(From OE-Core rev: cd3e55606c427287f37585c5d7cde936471e52f4)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Qemurunner should not pass rootfs to runqemu in case
rootfs is not a filesystem itself. Some images could
be built into some disk format like vmdk and this
commit makes qemurunner handle such images properly.
(From OE-Core rev: 5aea74046cf4c1aa7fa9e2402788d662268ccf53)
Signed-off-by: Sergei Zhmylev <s.zhmylev@yadro.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default host side IP address is not set and qemu listens
on all IP addresses on the host machine which is not a good
idea when images have root login enabled without password.
It make sense to listen only on localhost IP address 127.0.0.1 using
config change like:
QB_SLIRP_OPT = "-netdev user,id=net0,hostfwd=tcp:127.0.0.1:2222-:22"
This config works for qemu itself, but breaks runqemu which tries to
parse the host side port number from qemu process command line arguments.
So change the runqemu side hostfwd parsing for port number to ignore
the host IP address field.
Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
(From OE-Core rev: bdbd52082eb26f418000eb4e424baae9babc272c)
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using a binary string as the login banner search expression is fraught with
risks. We've seen cases on the autobuilder where "login:" is clearly shown
but the code hasn't triggered. The most likely cause is hidden control characters
in the output causing the search to fail.
Take the opportunity to remove the horrible binary string search, at the expense of
decoding the bootlog multiple times.
Tweak the logging so we can know which log was printed (self.msg or bootlog)
just in case this isn't the issue and we need more information in future.
(From OE-Core rev: 91b9e30e08695e715ef14c3df7471e8c99f9deb5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updates the log message printed when login banner is seen in QEMU to
report the UNIX Epoch time in addition to the human readable time. This
makes it much easier and accurate to correlate logs with the guest, in
particular with the guest journalctl which prints log timestamps in
human readable format and the oeqa SSH debug logging which prints the
UNIX Epoch.
(From OE-Core rev: 275b6f3c8d0eeafa3902c48a49655491a89c47bc)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Upstream made changes to the qmp module. We need to use the legacy one
for now since that matches the interface we use, ultimately we likely
need to update our code.
Also fix the generic exception handler to show the actual exception
which helps debugging when something does break.
(From OE-Core rev: 6e90b0d909d3c8b1be5cb19f2411cd0e89735c84)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a comment explaining the non-obvious return codes.
(From OE-Core rev: 6572baffa02ba6b8a686490d55af17cacb528920)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sys.exit will cause finally statements and other code to run at exit. Since
we're using os.fork() here, os._exit() is apprioriate in this codepath.
(From OE-Core rev: ec08498ff29de9ccd23be88b9d7af3dab6bbb81e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is done when starting up qemu has failed, but is not done
when qemu started ok, but fails later in QMP communication.
Output from runqemu does contain valuable information to find out
why, so rather than fix all the QMP fails to include it, let's just
print it in stop().
(From OE-Core rev: 6e2bf68e4401db747484c2c8ba0f77500b1d2d49)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Files in /proc/xxx/map_files/ may no longer exist, just ignore this rather than
raising an exception.
(From OE-Core rev: fb1027896a263cd91e2378a4e97dbdf0807b306b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
