create_socket: Use urllib because urllib2 is now urllib in python3
and proxies as argument are deprecated so export them in the environ
instead.
get_links_from_url: Change usage of sgmllib for parsing HTML because
is deprecated in python 3, use instead bs4 that is already imported
in the bitbake tree.
[YOCTO #9744]
(From OE-Core rev: ee26ecf58277560459dd01992bb3f486f92c1531)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On python 3, bytes variable types must be decoded if these are intended to be
used as strings, otherwise we get the following error exception:
TypeError: Type str doesn't support the buffer API
(From OE-Core rev: b950539c911b7945d652b05616164828e711ac7f)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exported tests is not been tested right now and it get
broken very easily with import errors from bitbake.
This adds a new test in order to verify in the CI if
the testexport gets broken.
[YOCTO #8384]
(From OE-Core rev: b6e0a762f8a17ee323cef02934f9a624faf706ed)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support to export the SDK tarball needed when a test
system doesn't have the required software to perform runtime
tests.
The support is when exporting the test and when running
the test on a remote system. The user of this feature just
need to set TEST_EXPORT_SDK_ENABLED to "1" and declare
the sdk packages in TEST_EXPORT_SDK_PACKAGES.
[YOCTO #7850]
(From OE-Core rev: a6041f81b81baa7564e4c712fc88de2b997e52e4)
(From OE-Core rev: 05e6c89f0f71311f8bd32cdb86a2deb789c58035)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the functionality to install/unistall packages in the
DUTs without the use of the package manager. This is possible
with the extraction introduced in package manager class.
testimage and testexport bbclasses has been modified in order
to support this new feature.
[YOCTO #8694]
(From OE-Core rev: b7111d9e9d64d21f57729d1ac1865aea6e54cc8b)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python3 the functionality to import modules has been changed and
this broke the capability to add runtime tests from other layers.
This commit returns this capability to testimage and testexport.
[YOCTO #9705]
(From OE-Core rev: a26f23d3ce8f7e9f59dbc9bf27516377fd7a0a6d)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
string.rstrip() doesn't exist in Python 3, so use the .rstrip method on the
object itself instead.
(From OE-Core rev: 554719e3f65e468f89a96d4869766d3ff7a8012e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The read method of the data_stream File object now returns bytes,
not a str, so we must decode it.
(From OE-Core rev: cfae302c4996c49a8754497ea9f13f8331d6975d)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove deprecated attributes in order to use python3.
runexported was changed to use python3.
[YOCTO #9702]
(From OE-Core rev: 9129af6dc421455c0253be313bf5781b913dc5fd)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We are observing cases (see below) where the 'WARNING:' prefix is not at the
beginning of a line, so instead of expecting it in the beginning, match it
within the string.
======================================================================
FAIL: test_recipetool_appendfile_patch (oeqa.selftest.recipetool.RecipetoolTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/recipetool.py", line 285, in test_recipetool_appendfile_patch
self.fail('Patch warning not found in output:\n%s' % output)
AssertionError: Patch warning not found in output:
Parsing recipes..WARNING: File /etc/selftest-replaceme-patched is added by the patch /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta-selftest/recipes-test/recipetool/files/add-file.patch - you may need to remove or replace this patch in order to replace the file.
NOTE: Writing append file /tmp/recipetoolqaug0kdb4x/recipes-test/recipetool/selftest-recipetool-appendfile.bbappend
NOTE: Copying /tmp/recipetoolqagci9tita/testfile to /tmp/recipetoolqaug0kdb4x/recipes-test/recipetool/selftest-recipetool-appendfile/testfile
done.
(From OE-Core rev: 2289138bdaa17b764821f41a3b3fd0a01cda7440)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since the stream can contain invalid binary characters (e.g. from
ppc's bootloader) use surrogateescape decoding to ensure we do process
the character stream, else it can hang/timeout.
(From OE-Core rev: 28a0030430d4cfcaf5dfc3e71bda07cdbfbbf4a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current parsing was picking wrong targets, leading to the following problem:
AssertionError: Command 'bitbake Parsing recipes..done. -e' returned non-zero exit status 1:
(From OE-Core rev: eaf83a58825d91c7445835b27d843da7532c208b)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bytes type variables are threat different as strings variables in python3,
so convert bytes to strings. This was found when using the
script buildhistory-diff, where the interpreter was yielding this error
TypeError: Type str doesn't support the buffer API
(From OE-Core rev: 3064d36186b47954eb94095217f7bb37e3fce651)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sets are safer when comparing internal elements and positions are not
important. This commit avoid errors observed on python3 builds as reported
on the below bugzilla entry.
[YOCTO #9661]
(From OE-Core rev: f6df164d09a4d4cf58977bf6cc0bc4f4bc71183a)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.
(From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved call of decode('utf-8') as close as possible to
call of subprocess API to avoid calling it in a lot of
other places.
Decoded binary data to utf-8 where appropriate to fix devtool
and recipetool tests in python 3 environment.
(From OE-Core rev: 30d02e2aa2d42fdf76271234b2dc9f37bc46b250)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Converted str to bytes before sending to socket.
Converted bytes to str after receiving from socket.
This should fix TypeError: 'str' does not support the buffer interface
for qemurunner.run_serial method.
(From OE-Core rev: 210e290c9251839dc74e3aabdcea3655dd707a50)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
xrange() no longer exists in python 3, use range()
(From OE-Core rev: d022b4335100612d6596cc4c4956cb98ed5873cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.
(From OE-Core rev: caebd862bac7eed725e0f0321bf50793671b5312)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.
(From OE-Core rev: 2476bdcbef591e951d11d57d53f1315848758571)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.
(From OE-Core rev: bb4685af1bffe17b3aa92a6d21398f38a44ea874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The syntax for octal values changed in python3, adapt to it.
(From OE-Core rev: 737a095fcde773a36e0fee1f27b74aaa88062386)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If fetching source from a git repository, typically within OpenEmbedded
we encourage setting SRCREV to a fixed revision, so change to do that by
default and add a -a/--autorev option to use "${AUTOREV}" instead.
(From OE-Core rev: 000480c42797dd2f03ebc3bc6d1dabfc6a7b75f5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some more variables in appropriate places in recipe_progression such
that the patch_recipe_* functions are able to insert variables in the
right place within a recipe.
(From OE-Core rev: c33ba6cc5d14b1da96f6d906836c50e0346dcf06)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function was assuming that what you wanted was that output lines
had trailing newline characters. If you're just outputting each line
verbatim to a text file then that's fine, but sometimes you start with
the assumption that the lines don't have trailing newlines; thus we
shouldn't allow for the possibility that the caller doesn't want them
and add a parameter to control it.
(From OE-Core rev: fb2bb509ff5c7bd71b41a1dcba3b1bff1d18cf5d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Split out a function from patch_recipe_file() that takes just the lines
as input so we can edit recipe lines in memory. This will be used within
recipetool to ensure we insert new values in the right place.
(From OE-Core rev: d780642f950fb3a9699f466a405a2710d870dd08)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having the package path with all the other package info allows to
reuse more code and have this information outside the package manager,
without additional processing.
[YOCTO #8536]
(From OE-Core rev: 343f762792cbfccffaf3aa901289f9bb0f8cef3d)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes is needed to have a package without extraction when
running a test. This patch adds the functionality.
[YOCTO #8536]
(From OE-Core rev: 49234fe926224c21ef6c8292132620b4716c5263)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This new method extracts the content of package (RPM, DEB, or IPK)
to a directory inside of WORKDIR. The extraction is needed for later
install in the DUTs without using a package manager.
[YOCTO #8694]
(From OE-Core rev: 90d585f59f217f23694a9b02a73b79d18dfdb579)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the functionality to use a json file to
specify the packages needed for a particular test.
The content of the json file is a dictionary with
dictionaries inside, using the test name as the hash.
The json file must have the same name as the class
module name and must be in the same path.
[YOCTO #7850]
(From OE-Core rev: 1f24ef9a339a2ad34e010f39aa93abdc8d085c85)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This new file just contain one function to return an OE package
manager, this depends in the current packaging method.
[YOCTO #8694]
(From OE-Core rev: 9b4b150ccbcd2f7a0b27d083eb406578584af3fa)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update older exception syntax to modern one required by python 3.
Compatible with python 2.7.
(From OE-Core rev: d13f0ac614f1d1e2ef2c8ddc71cbfcf76a8dc3f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
file() API doesn't exist in python 3, convert to open(). Also handle
some cases where files aren't closed. Compatible with python 2.7.
[Contributions from Ed and Richard]
(From OE-Core rev: 0f4ec13e11bb8abe21aba2a28547dfb9372bc377)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used print function instead of print statement to make
the code work in python 3.
[Changes from both Ed and Richard]
(From OE-Core rev: ced1995694c394d92cb82fb9c25a33dc027a3b69)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having two code paths here makes maintenance difficult, and it doesn't
seem likely that you would use the local case in real usage anyway, so
drop the local support entirely.
This should allow us to resolve [YOCTO #9301].
(From OE-Core rev: 7a4c9c96fee4fb514c2b69b52e811c4f896a16f1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 20ea6d274bb0a9a5addb111f32793de49b907865, debugfs generation
for images using opkg, which included openssl was fixed.
However, that broke the generation of the opkg-based images, that lacks
openssl. The error is a python stack trace, showing that shutil.copytree
tries to copy a non-existing directory.
This relates to [YOCTO #9040].
(From OE-Core rev: f6b0b260ce18a30d04edfb0afb7942b9f9a5480b)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a common _log_check_common() function (based on the old
_log_check_warn() function) to implement the logic for both
_log_check_warn() and _log_check_error().
The main benefit of this is that now all error messages will be
reported again, not just the first one found. Additionally the output
will now look the same for both error and warning messages.
This removes the context for the error messages. However, since there
was no indication in the output that some of the lines were context,
they were more confusing than helping.
(From OE-Core rev: dd73dcac36b80b3b886a2e9bf575b91c4f60d039)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For each warning found in the log, _log_check_warn() would output a
line stating that it had found a warning, then the actual warning and
finally an empty line. This is quite excessive when there are many
warnings in the log.
With this change the output is instead a line stating how many
warnings were found, followed by the warnings. This makes the output
much more compact and actually much more readable.
(From OE-Core rev: d6e3477749b1d09d40a773e0ac857a24d5851984)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will make _log_check_warn() exclude the same lines as
_log_check_error() does.
(From OE-Core rev: 85f64c68278f797c6f73f002f63d7f46fe80aef4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before there were three different ways to exclude a line from being
searched for error messages in _log_check_error(). Now there is only
one: an array of regular expressions. This should make it easy to add
more excludes if nedded.
(From OE-Core rev: 321df88088fbfa657b61b2bae32751f03daec46f)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 0387d095 lines with "NOTE:" in them were excluded from the
log check for warnings. However, those lines were only there in the
first place since the regular expression that is used to find warning
messages explicitly included those lines...
(From OE-Core rev: 865ab39f18a52ed84217df56d0e65113e2894d02)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fact that this function was overridden in the RpmRootfs class
seems to have led to a number of misstakes when changes have been made
to the base function in the Rootfs class. E.g., this change will
properly solve ticket 7789, which was supposedly solved in 38871dc0,
but that change had no effect in practice as the log_check_regex that
was modified for RpmRootfs class was not used by the RpmRootfs version
of _log_check_error()...
The only thing _log_check_error() in RpmRootfs did that the base
function in Rootfs did not do was to skip lines in the log that begin
with a + sign. This has now been moved to the base function instead.
[YOCTO #7789]
(From OE-Core rev: 1eb0a46502fca4b2ee30ccd2508f4e21a40c25ca)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The reverted commit added a warning with the log line that triggered
the log check for error messages before the warning that states that
an error has been found in the log. However, the error line is output
by the call to bb.fatal() that follows immediately after the original
warning, which makes it redundant. Additionaly, having two warnings
contradicts the intent of commit 8dfdd329 where the log warnings were
tidied up.
This reverts commit f9cf31525fc885e1a0f65bd55654631257f87078.
(From OE-Core rev: e509a83c8eb5a572aad7e7451d019fa2469d70c3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tested generation of .bmap file using wic --bmap command
line option.
(From OE-Core rev: 1f21c46e517685ff1363d3b6f4fd840300b1930b)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are parts of the fuctionallity missing when using the
dummy ExportTestContext class in runexported.py.
This changes the use of ExportTestContext dummy class from
runexported.py to the real class in oetest.py.
(From OE-Core rev: b0f105eecf25d334dd5fd0d936ec98cb8430ed23)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
