The high-level method tempfile.TemporaryDirectory give us no way to ignore erros on
removal thus use tempfile.mkdtemp instead. Ignoring possible issues on removal
is neccesary because it contains gpg sockets that are automatically removed by
the system once the process terminates, otherwise the following log is observed:
File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.browser'
This is the same fix as 7e3a7cd2426feac757def81850dc44156cd0f33e, but this applies
to runtime (instead of signing).
[YOCTO #11821]
(From OE-Core rev: f28e8131f69913ff90ed210c7a58726d3ef37db6)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Search made with the following regex: getVar ?\((.*), True\).
(From OE-Core rev: dbc0eaf478feb3f752ae22fd184984494fc85d0a)
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are cases where the 'while loop' waiting for login prompt fails
and the bootlog variable does not get populated, thus use the the new
qemurunner member (self.msg) which stores all output coming from the qemu
process.
[YOCTO #12113]
(From OE-Core rev: 39ffa0f3779305c5e8ef86fe4572e961c5912021)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It contains cached metadata for a transient repository that is used
only when creating images on the host, and so is of no use on target
images. Dnf will recreate the cache on target when needed.
(From OE-Core rev: 89ddf2516b713e6622df95ab391423b033d4271a)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GitHub dynamically generates the /archive/ tarballs but we're encoding checksums
in the test suite. Change the URL to use a static tarball, and update the
checksums.
(From OE-Core rev: 9c668f9ff989a34e615e2ecc051dadbfe24a5bb4)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since it was first implemented, devtool's source extraction (as used by
the devtool modify, extract and upgrade subcommands) ignored other recipe
dependencies - so for example if you ran devtool modify on a recipe that
fetches from svn or is compressed using xz then it would fail if those
dependencies hadn't been built first. Now that we can execute tasks in
the normal way (i.e. tinfoil.build_targets()) then we can rework it to
use that. This is slightly tricky in that the source extraction needs to
insert some logic in between tasks; luckily we can use a helper class
that conditionally adds prefuncs to make that possible.
Some side-effects / aspects of this change worth noting:
* Operations are a little slower because we have to go through the task
dependency graph generation and other startup processing. There's not
really any way to avoid this though.
* devtool extract didn't used to require a workspace, now it does
because it needs to create a temporary bbappend for the recipe. (As
with other commands the workspace be created on the fly if it doesn't
already exist.)
* I want any existing sysroot files and stamps to be left alone during
extraction since we are running the tasks off to the side, and
especially devtool extract should be able to be used without touching
these. However, this was hampered by the automatic removal process in
sstate.bbclass triggered by bb.event.ReachableStamps when the task
signatures change, thus I had to introduce a way to disable this
removal on a per-recipe basis (we still want it to function for any
dependencies that we aren't working on). To implement this I elected
to use a file written to tmp/sstate-control which gets deleted
automatically after reading so that there's less chance of stale files
affecting future sessions. I could have used a variable but this would
have needed to be whitelisted and I'd have to have poked its value in
using the setVariable command.
Fixes [YOCTO #11198].
(From OE-Core rev: 830dbd66992cbb9e731b48d56fddf8f220349666)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need QEMU PID in order to access "/proc/<qemupid>/cmdline"
Having a valid QEMU PID does not mean we can access the proc entry
immediately, we need to wait for the /proc/<qemupid> to appear
before we can access it.
(From OE-Core rev: d2d069fa9910d1c7a94c898355a63fca03ec5ad8)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code in scriptutils which implements the logic for running the
editor used by devtool edit-recipe looks at the VISUAL environment
variable before EDITOR, and thus if VISUAL is set in the environment it
will override the EDITOR value we are setting here, the editor (usually
vim) launches and there's nothing to stop it running forever short of
manually killing it. Set VISUAL instead to fix this.
Apparently VISUAL is in fact the variable we should really be preferring
here - I don't think I knew that but somehow I got it right in the code,
just not in the test. Here are the details for the curious:
https://unix.stackexchange.com/questions/4859/visual-vs-editor-whats-the-difference
Fixes [YOCTO #12074].
(From OE-Core rev: 6a7c50def569b5e86aa17bd9b287e8c63781dcb0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The high-level method tempfile.TemporaryDirectory give us no way to
ignore erros on removal thus use tempfile.mkdtemp instead. Ignoring possible issues
on removal is neccesary because it contains gpg sockets that are automatically removed
by the system once the process terminates, otherwise the following log is observed:
..
..
File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.browser'
[YOCTO #11821]
(From OE-Core rev: 7e3a7cd2426feac757def81850dc44156cd0f33e)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a tempdir to copy the .config file from the kernel instead of being
copied to build directory.
(From OE-Core rev: ccccf975c75055639c0ed5052ac0ad53ebbf53ca)
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we capture stderr, always write the output to the log, and don't be so
emotional in the warning message.
(From OE-Core rev: 87110e971c548d9d3eb7296ea6772d18e0226e53)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recipeinc method returns the absolute path of the test_recipe.inc
file of a specified recipe. It replaces four instances of identical
code, and make it possible to access the filename from a testcase for
cleanup.
The write_recipeinc and append_recipeinc methods are changed to return
the path to the file in case that is useful.
The test_recipe.inc file is usually cleaned up in a finally block,
but that block executes before any teardown operations. This blocks
any teardown that requires the presence of the test_recipe.inc file.
(From OE-Core rev: cdb431676456f47da1a3b70caddf49f083948798)
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ask QEMU to store the QEMU process PID in a file, this way we don't
have to parse running processes and analyze descendents.
This is done via QEMU command line argument "-pidfile".
[YOCTO #12001]
(From OE-Core rev: 67612dcd2a8a1aa1d683dddb0bd2f592886ff020)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python unittest intercepts stdout and stderr however qemurunner sets
up a streamhandler before that interception occurs, hence the messages
spam the unittest output.
By moving the logging init to the class init time, we use the unittest
stdout/stderr and this means unittest can only show the log output upon
failure. This cleans up the selftest and testimage output whilst still
showing logging upon failure.
(From OE-Core rev: 9099cecc727fe0ae5f1559582426d30ba7a9f4d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If this fails the exception will now be raised. Lets use that and drop
all this 'None' return value ugliness.
(From OE-Core rev: b3c63b3b816179b96f1ed9b5baaf6e1f1c3c7b80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently exceptions that happen in pool commands are ignored. Any errors
would be printed on the console but everything else is silent.
Switch to use pool.map_async which allows for an error_callback which
we can use to detect exceptions and make sure these errors are handled.
(From OE-Core rev: 7f2f9b3ff011b340b5d23bb7c47b12c357dc9f02)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the exit code of the spawned program isn't checked so it can
fail and the do_package task will continue merrily upon its way.
Use subprocess.check_output() to ensure we check the exit code and
redirect stderr to stdout so if it fails, we see the error output.
We can then drop the existing exception handling as the subprocess
exception gives a much better error.
(From OE-Core rev: ce11cb449222bc47fea4f6d66ff1cc7cdc529ab9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the memory resident mode, the user may not see a message about the server
starting, it would be in the cookerdeamon logfile. We don't need this to
test the server is functioning correctly so just drop the test.
Add in an extra check that the file we expected to be created was
created when exporting PR values.
(From OE-Core rev: 811edd95420e907e71b5c7646bde5013b43d4c73)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I recently found that variable history wasn't working properly for
recipes when we enable history tracking, resulting in minor
functionality loss in devtool upgrade, so add a test to ensure this
doesn't regress now that it's fixed.
(From OE-Core rev: f49042e707d641b59a71c687374b76df97c64c34)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We weren't testing the devtool find-recipe and edit-recipe subcommands,
with the result that when they regressed recently we didn't notice. Add
some code into the test_devtool_add to test this (since we need a
recipe in the workspace, and adding a new test with all that preamble
would seem a bit excessive for these simple checks). Also take the
opportunity to refactor the test a little bit so that the recipe name
and version are variables rather than hardcoding them everywhere.
(From OE-Core rev: 355d8f42679e37610c2947dece597ed7db774bee)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're upgrading a recipe that appends additional patches for, say,
class-native, and we're just upgrading the target variant, then when we
copied the recipe into the workspace we skipped copying the additional patches
for the native variant. This caused warnings because the workspace
recipe is preferred. Look at SRC_URI for all variants when copying files
to work around this.
More work is needed to make it easier to work with recipes that use
BBCLASSEXTEND where you need to build more than one variant at once, but
this at least fixes the immediate ugliness.
(From OE-Core rev: 56bf5e93358187e31160d7893f57906bb3dc7ad7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If your BBLAYERS has non-absolute paths in it (e.g.
"${COREBASE}/../something") then none of the paths matched in
copy_recipe_files() with the result that no files got copied and you
ended up with an error later on because the recipe file couldn't be
found at the destination. Fix this as well as adding an explicit check
to see if no files got copied - error out earlier if so.
Fixes [YOCTO #10981].
(From OE-Core rev: 3861486ad06f90c8644ebab119bbc5ddb9e693ca)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating Packages.gz, do not save the time stamp in the compressed
file metadata.
(From OE-Core rev: 693fc377aa0788825b921d1b7804f5b2eafec76f)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DISTRO_VERSION may contain the current date, and so is prone to mismatches
particularly when keys are created on one date, and dnf is configured to use the keys
on another date.
[YOCTO #11983]
(From OE-Core rev: b8f33f3f0a1e24f89151743d5c0b82efe45a9df1)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing timeouts in the autobuilder testing code. Increase these
timeouts to 120, print the length of time we're having to wait, change
the error messages to really be errors and don't print empty logs, its
not helpful, print a message about the empty log instead.
(From OE-Core rev: b87a33d9c86cc1d2ea196259020e1d3dc712ccef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added test_expand_mbr_image test case to the wic oe-selftest suite.
The test expands directdisk wic image to 1Gb target, checks if it's
expanded correctly and boots it in qemu to make sure the image
is bootable, i.e. bootloader, kernel, boot and root partitions
are still functional.
(From OE-Core rev: 83739174be5b3fabd7df559a35ac6407d82a6280)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not currently used, but include for completeness.
(From OE-Core rev: dcceb90516efc77e4d0cddc3caf4e203ab9321d6)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without the --disable-factory option, mate-terminal fails to start
with the error:
| There was an error creating the child process for this terminal
| Failed to execute child process "oe-gnome-terminal-phonehome" (No such file or directory)
The --disable-factory option was removed by:
http://git.openembedded.org/openembedded-core/commit/?id=e8dca725ed8211a874472300a3ed50e494039ab9
apparently based on an assuption that mate-terminal continues to
track gnome-terminal since forking from it. However, based on the
mate-terminal man page in the upstream master branch, the option is
still supported:
https://github.com/mate-desktop/mate-terminal
(From OE-Core rev: c7f769bd400756d180abf80dbfdf4ed2703fab60)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the less common terminal types haven't been tested with the
recent phonehome pid file changes and there may be error cases where
the pid file is never created.
(From OE-Core rev: 6b0cf568e9fbe28fb6e7b17f4ad92348d33e2bf4)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- rename type to exec_type not to shadow type
- rename isELF is_elf
(From OE-Core rev: c72bd726d3e8495aae3e57f524c43b3be6367796)
Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A gpg command, e.g, called by get_gpg_version(), may trigger such a warning:
gpg: WARNING: unsafe ownership on homedir '/home/user/.gnupg'
This sort of warning is harmless so it can be depressed through specifying
--no-permission-warning.
(From OE-Core rev: 0a740aa6f02deada56e0b7deb2bc1f7401e58726)
Signed-off-by: Jia Zhang <lans.zhang2008@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There might be wild strings when read from target (especially when
reading ptest results), replace the errors to avoid breaking the test.
Fixed: (Not always happen)
$ bitbake core-image-sato -ctestimage
[snip]
status, output = self.target.run('ptest-runner', 0)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 84, in run
status, output = self._run(sshCmd, processTimeout, True)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 55, in _run
status, output = SSHCall(command, self.logger, timeout)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 258, in SSHCall
run()
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 236, in run
output = process.communicate()[0].decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4906: invalid continuation byte
[YOCTO #11547]
(From OE-Core rev: d0d2f892f0bed6adb5ec6fb59d64efcc97c83e19)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The section might be None or '', so use "if section" for it.
Fixed:
File "/buildarea/lyang1/poky/meta/lib/oeqa/utils/logparser.py", line 113, in log_as_files
prefix += section
TypeError: Can't convert 'NoneType' object to str implicitly
[YOCTO #11547]
(From OE-Core rev: 320ea01f9eb33df462534bf08488ff6ada9bbe7b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Make it work with current oeqa
* Skip the test if ptest is not in DISTRO_FEATURES
* Skip the test if ptest-pkgs is not in IMAGE_FEATURES
* The logs are saved to:
testimage/ptest_log -> testimage/ptest_log.<datetime>
* This provides data that could be used to detect regressions in ptest results
[YOCTO #11547]
(From OE-Core rev: f1dfb59495db9e79441c8aa623ede7ef20045a20)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The IDE driver in the kernel is fragile and in 4.12 is causing backtraces.
To unblock 4.12 kernel merging use the virtio CD driver instead to mount
iso images which should be faster and more stable.
(From OE-Core rev: f59e729f98ef9b506b0cfdc415567e03ec87f2a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
THe newner kernels have ope firmware support enabled which leads to warnings
during boot on mips. These aren't interesting and we should ignore them.
(From OE-Core rev: 4ba9a6ffb4e7f6c7eee47aa13252fd981cfe5618)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before GNU tar 1.24, only the archive creation command had the '-h'
argument to preserve and follow symlinks. After >= 1.24 via commit
14efeb9f956e38d7be (tar: --dereference consistency) the capability to
preserve symlinks was also added to the archive extraction command.
-h is default at archive creation but is not default at extraction,
meaning that it will replace symlinks with directories even if the
original filesystem directory tree and archive contains them.
Add -h to the copyhardlinktree extraction step so the build can
support symlinks in variables like ${DEPLOY_DIR_IPK/RPM/DEB}.
(From OE-Core rev: c1b73fe26540c38086e259be6eb313e0826dceed)
Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The eSDK installation requires the meta-skeleton layer.
The build system might use the meta-skeleton recipes as layout
to create custom recipes. An example is the recipetool script
that uses the meta-skeleton kernel recipe when creating a custom
kernel recipe.
[YOCTO #11102]
(From OE-Core rev: 5c9ef0734d23909b5694ed43cdbb205c2ba9ca95)
Signed-off-by: Juan M Cruz Alcaraz <juan.m.cruz.alcaraz@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When buildsystem with layer structure is going to be copied, only the last
meta-XXX layer is taken.
For example, during ext_sdk bblayers creating:
layers/oe/meta \
layers/oe/meta-oe \
layers/oe/meta-networking \
layers/oe/meta-webserver \
...
It restructured meta-oe, meta-networking,... contents into meta-oe.
Recipes from meta-oe will be on the same level like meta-networking,
meta-webserver, ... .
It should take the whole meta path instead of the last one.
layers/oe/meta \
layers/oe/meta-oe/meta-oe \
layers/oe/meta-oe/meta-networking \
layers/oe/meta-oe/meta-webserver \
...
Now the directory structure is the same like during build creation.
(From OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b)
Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit d58b1d196 moved from chunk to serial signing, but neither of both approaches
allowed the user to select the chunks size. This patch allows the user to select
a chunk size through RPM_GPG_SIGN_CHUNK defaulting to BB_NUMBER_THREADS, considered
a good default. Indirectly, this change reduces the number of processes spawn
to number-of-packages/RPM_GPG_SIGN_CHUNK.
(From OE-Core rev: f7f78e73f1cd15f4233a231364b14438af758628)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
