This avoids some tracebacks we've seen on failed autobuilder builds which would
allow the real error to be seen.
(From OE-Core rev: a4031935a7b8ea4f61b9020c1aa5598e186e7ad7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not all QEMU machines are capable of having more than one serial port,
this is due to the machine emulating a physical device/board.
Rework QemuRunner to handle machines that only have 1 serial port, where
the serial port shares output of the kernel log buffer and a login
console. In this case the output is mixed but enables the machine to
boot and have QemuRunner detect the login prompt.
QemuTarget uses SERIAL_CONSOLES to determine the number of available
serial ports.
(From OE-Core rev: 333897c4a00b41681ffe34312a08cae09274327e)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some commands like to look for libraries at runtime manually (e.g.
Python's ctype.utils.find_library() function). For this to work
properly, the libraries in the native sysroot must be findable. To
accomplish this, set LD_LIBRARY_PATH to search library paths in the
native sysroot.
(From OE-Core rev: 3a7305bdfded3c8988484c3f430110cc121123b5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a nfs module into oeqa utils. This module provides unfs_server which
allows a test case to build unfs3-native and setup the unfs server on a
target directory of the host. This directory is then shared and can be
mounted by the host or a target device attached to the host (e.g. qemu
via tap or slirp). The nfs server is setup over UDP and automatically
assigns user privileged ports. The function provides the UDP ports for
the server as part of a returned python contextmanager which handles
cleanup of the server process on completion or exception.
Also add a 'udp' arg to get_free_port to get a free UDP port.
Note: unfs3 still requires the host to have rpcbind or portmap running.
(From OE-Core rev: c754fd85be85ad0a381b642365eca17cea8eb627)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, 'nographic' was set first, and was overriding anything set via runqemuparams.
Particularly, egl-headless from the virgl headless selftest was overriden,
which broke the test in non-X environments.
(From OE-Core rev: 344c55a58042ecd6ffed8b20add41b6b335c51bf)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst seemingly a good idea, this exception doesn't really serve any purpose
that bb.fatal() doesn't cover. Wrapping exceptions within exceptions isn't
pythonic.
Its not used in many places, lets clean up those and remove usage of it
entirely. It may ultimately be dropped form bitbake entirely.
(From OE-Core rev: efe87ce4b2154c6f1c591ed9d8f770c229b044ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of hiding problems in the debug log let's print them as warnings
instead.
(From OE-Core rev: 088f5d97001bd4b573f00cfca93b8d24e814fd64)
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When handling pid file, qemu would first create the file, stat it,
lock it and then write actually contents to it.
So it's possbile that when reading the pid file, the content is empty.
[YOCTO #13390]
(From OE-Core rev: 170e59b203a02f8438b9aeab3a45f6fcd6608b1f)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output format for Python and GLib both can be of this form:
FAIL: foobar (Segmentation fault)
In this case the test is called foobar not foobar_segmentation_fault.
(From OE-Core rev: 0219e9bd0273661b4b70df97e5762f77b3ac3e8c)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While hacking on this I got an Exception. It's better to define
variables also in python.
Signe:-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
(From OE-Core rev: 1ea225a86cdee4ed932ede509d3351d5aecae497)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the commit ad522ea6a6 ("runqemu: Let qemuparams override default
settings"), the order of the two "-serial" parameters when running the
qemu have been switched. The effect of this is that the logging thread
will use ttyS1 (of course can't capture the kernel boot message anymore),
and the test command will run on the ttyS0. So the output of the test
command may be mangled by the kernel message (such as call trace), and
let the test command produce a fake timeout error message. We can't fix
it by just adjusting the order of the threadport and serverport, since
it will break some machines such as qemuarm64 which use the virtio
serial. So using the tcpserial to setup both the threadport and
serverport.
[YOCTO Bug 13309]
(From OE-Core rev: 9f2005dee41b1ef5a0d1f7b69bcd6c8352dac016)
Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
self.msg is a str(), bootlog is b'' so this code clearly doesn't work. Add
in a decode since its being used as a string.
(From OE-Core rev: e7546759d3102b576974bbab52594796f084721e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If python3 git module is not installed, we fall back to some git
command to get info. A previous commit (b8d22ed6) accidently deleted
the return statement, causing errors like below.
Exception: UnboundLocalError: local variable 'InvalidGitRepositoryError' referenced before assignment
(From OE-Core rev: cd8aedc408afa4f8589f79e4d85befc7cac76ac6)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if python3-git isn't installed we can get odd behaviours when the
commit_count is absent. Avoid this set of bugs by adding a fallback here.
(From OE-Core rev: b8d22ed681141ce360d742a96cec5f2925a20222)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The parent code currently assumed that any parent branch is locally
checked out which may not be the case.
Use the local branch by default but fall back to the origin. This
also means removing the later saftey check as the branch may not exist
locally.
This fixes the autobuilder resulttool test pushing code.
(From OE-Core rev: 0ee636bc89d93282cb3d3cd747b3dd046f6e5a9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These functions can be reused by the resulttool code so move to the common
function library for this purpose.
(From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.
(From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Get rid of further unneeded code complications:
* value mappings we could just direct use
* ftools when we can write files easily ourself
* test result status filtering we don't use
* variable overwriting module imports
(From OE-Core rev: d6065f136f6d353c3054cc3f440a4e259509f876)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Merge the results handling into the ptest log parser as a seperate
method.
Drop the weird "pass.skip.fail." prefix to the results filename, its
just bizarre.
Drop the code turning a list into a regex then searching the regex for
an item, "x in y" is perfectly capable.
Use a dict, sort the keys as needed and drop the list sorting code.
(From OE-Core rev: f317800e950b4a37b4034133bc52e0c47f04dc29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow parsing of the ptest duration, exit code and timeout keywords
from the logs, returning data on each section.
Also include the logs broken out per section.
(From OE-Core rev: a9a67dccaa5be0f06eedcab46dcff7cbf9202850)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have a dedicated ptest parser, merge in the remaining ptest
specific pieces to further clarify and simplify the code, moving to
a point where we can consider extending/enhancing it.
(From OE-Core rev: 05991bb5bc8018275d03fdeecee3d5a757840c7c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename the paster to be ptest specific and apply some further cleanups
to the code to simplify and clarify what its doing.
(From OE-Core rev: 45a5886f1ec458d4c306b8d68fd31d568bc36b47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
logparser is only used by ptest. Its slightly overcomplicated as it was
intended to be reusable but wasn't. Simplify it as a dedicated parser is
likely to me more readable and maintainable.
(From OE-Core rev: c7478345b2b4a85cb1fec40e762633871f0e94cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are failures when multiple users run oe-selftest on the same
host:
PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved-
tests/201812250324_qemu'
This is because /tmp/oe-saved-tests was created by user A, while user B tries
to write data in it, then the error will happen. This patch can fix the
problem.
Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than
/tmp/oe-saved-tests/ to fix the problem.
(From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 5s timeout for non-kvm is too short, especially when the load is high,
which leads to unexpected errors, so set timeout to 60s by default.
(From OE-Core rev: 8197d0f638a760fc03062c7a9009117d083d7ead)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we need to create a temporary directory in targetbuild or buildproject use
tempfile.TemporaryDirectory so that when the test case is finished, the
directory is deleted.
Also synchronise the logic and don't possibly store the temporary directory in
self.tmpdir as nothing uses that.
(From OE-Core rev: db0e658097130d146752785d0d45f46a3e0bad71)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reorder the shutdown/teardown to avoid:
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 224, in launch
op = self.getOutput(output)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 90, in getOutput
fl = fcntl.fcntl(o, fcntl.F_GETFL)
ValueError: I/O operation on closed file
(From OE-Core rev: 8e7d756862d2a8d62f3c87497d6d65ddb3c1b962)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runqemu fails it would leak an unclosed socket and file. Ensure we
close these in all cases to remove the resource warning.
(From OE-Core rev: ed80e46ccbc8fe8e9148d80723152066fa00ba28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Each time a runqemu() fails, the log handler would be left behind meaning
messages from any subsequent run would be duplicated (or worse/more).
This ensures we remove the handler regardless and means we no longer
have the duplication.
(From OE-Core rev: 532984708436bdfa3a8cac2c684a425eb249bad0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just referring the user to the logs containing the failure, print
them on the console. This aids debugging with oe-selftest with parallelisation
as the logs may otherwise be lost.
(From OE-Core rev: 36a018e245a232f520ff946f152cc875927a6fb4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable qemurunner for slirp. Retrieved the ip & port from host machine
to connect to qemu from host machine.
[YOCTO#10713]
(From OE-Core rev: 1db6a6fc9cde28d0a29bcf6d24a8bfbe51d120b1)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage.bbclass installs a SIGTERM handler which conflicts with the
use of multiprocessing here. This is paritcularly problematic if the http
service is terminated before its started and hence before its had a chance
to reset the default signal handler (as the code was written).
Instead, temporarily remove testimage's handler whilst forking the http process
which means the correct handler is installed and won't deadlock.
Also take the opportunity to add in some log messages about the server start
and shutdown so that future debugging is easier and its clearer what the code
is doing.
(From OE-Core rev: 0762b6021b87ceb1f37952f3a6d64a36e99ae6a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should only be wiping out things on target if the tests have
actually run.
(From OE-Core rev: d38c3eac0a5a1a9b0eb98385832e92f48145655e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The python git module may or may not be enabled, allow this code to
function without it, falling back to the same method as metadata_scm.bbclass
uses. This will be cleaned up in the next round of feature development.
(From OE-Core rev: 6350586ba9f4a4107a2d457590824cd4d662d5b9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The qemu shutdown can race with the signal handler removal leading to
confusing tracebacks on slower/loaded systems.
Remove the signal handler first before shutting down.
(From OE-Core rev: 1b63ce7bb83134f84bf07d1075d5ca0d5466ed3d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similar to rpm, use copies of the ipk/deb directories for rootfs construction.
This means the image creation code can no longer "see" recipes wich aren't in its
dependency chain which is good for a variety of reasons including determinism,
incompatible recipe (e.g. systemd/sysvinit) package conflicts and locking
performance.
(From OE-Core rev: c7c5f4065c102fde4e11d138fb0b6e25bffe0379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #12493]
Fixed:
- qemu started with pid 10000
- qemu exited unexpectedly
- The pid 10000 is re-used by another different process.
The is_alive() returned True in such a case because both qemu_pidfile and
/proc/10000 exist, but it's another process, this patch fixed the problem.
(From OE-Core rev: 78033af85698b4e112ae78e3140b241b1b3c06a3)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warn method is deprecated. We should use the documented warning instead.
Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
As warn is deprecated, please do not use it - use warning instead.
"""
(From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we see warnings if LAYERSERIES_COMPAT is unset, the auto generated
code from devtool/oeqa needs to set this to avoid warnings which break
various tests.
(From OE-Core rev: 27568410ebb0d40db3428550704f35199df0e034)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently do_rootfs gets to see all rpms in the deploy directory. This filters
that view to only rpms which the image recipe has actual depends upon which
potentially removes some sources of confusion in the image construction.
This makes builds more reproducibile and also fixes contamination issues
where dnf picks up packages it shouldn't be able to 'see'.
[YOCTO #12039]
(From OE-Core rev: 85e72e129362db896b0d368077033e4a2e373cf9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
