Commit Graph

2670 Commits

Author SHA1 Message Date
Paul Eggleton
14f3d4e58a recipetool: create: fix incorrect URL variable usage
We have two variables here, srcuri and fetchuri. srcuri is what
eventually ends up in the recipe, whereas fetchuri is what we actually
pass to the fetcher when we fetch the source within recipetool -
sometimes these need to be different particularly for an upcoming patch
to handle automatically setting the branch parameter. In OE-Core
revision 9a47a6690052ef943c0d4760630ee630fb012153 I erroneously changed
the call to scriptutils.fetch_url() to pass srcuri instead of fetchuri -
this likely didn't have any ill effect, but change it back to passing
fetchuri to match the original intent.

(From OE-Core rev: b66b73bcf5ee7e4488970576fdc31dfa25b35f5e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-17 00:21:14 +01:00
Andrea Galbusera
1df5a58b12 scripts/oe-publish-sdk: use hook to call git update-server-info
The author's initial intent was to use a git hook to automatically call
update-server-info, but the wrong hook type was chosen (post-update). A
post-commit one will do the job, hence allowing to drop the explicit call to
update-server-info.

(From OE-Core rev: 0a8ae997c782794b6ef32654cb245b496ab1de8e)

Signed-off-by: Andrea Galbusera <gizero@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:16 +01:00
Andrea Galbusera
6346d12d8d devtool: sdk-update: fix pulling updates from git
Commit 4657bc9d165e51981e034e73e7b92552e873eef7 replaced the git pull logic with
the git fetch + git reset --hard combo, but resetting to HEAD does not really
pull in new commits from remote... Replace with resetting to the upstream branch
instead.

(From OE-Core rev: 0dcdb146f59a184419bffd4f24cdf8343a43c0ea)

Signed-off-by: Andrea Galbusera <gizero@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:16 +01:00
Paul Eggleton
1cce7f885c devtool: fix handling of errors during task execution
* If an error is logged while executing a task, we need to ensure we
  exit instead of assuming everything went OK.
* If we receive CookerExit, the server is shutting down and we need to
  stop waiting for events and probably exit (knotty does this). This
  will occur if an exception or bb.fatal() happens during an event
  handler.

This fixes a couple of issues highlighted when using devtool upgrade or
modify on a non-supported recipe with intel-iot-refkit together with
bitbake master, but I'd be very surprised if it were hard to reproduce
in other scenarios.

(From OE-Core rev: 65e644368fc9c294af96906528ee0cf30305e0a6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:15 +01:00
Mark Asselstine
c760d99466 python: don't include -tests with modules
Although 'test' is listed in the python module list
(https://docs.python.org/3/py-modindex.html) it is meant only to be
used 'internally' by folks developing python itself. Per the
documentation:

  Note The test package is meant for internal use by Python only. It
  is documented for the benefit of the core developers of Python. Any
  use of this package outside of Python’s standard library is
  discouraged as code mentioned here can change or be removed without
  notice between releases of Python.

Per the above it is best not to include this module to discourage
folks who might not head the above warnings. Additionally this module
is one of the largest py modules going, by dropping this unneeded
module from the 'modules' package we can reduce overall image size,
something which is important for many embedded deployments.

The generator scripts as well as the manifests have thus been modified
accordingly, providing a generic mechanism to exclude modules from the
'all modules' package.

(From OE-Core rev: a5bb13a5d7d7a668ca61da6b17884e3b05b95355)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-03 11:14:13 +01:00
Jonathan Liu
843a74d708 wic: improve generating disk system identifier
This should reduce the chance of generating 0xffffffff as the disk
system identifier.

(From OE-Core rev: 5619e72facbc9228d3cf8f844f198e03b536ac8c)

Signed-off-by: Jonathan Liu <net147@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-03 11:14:13 +01:00
Tom Rini
03d5daf1ae wic: Switch to using --use-uuid by default
The most portable way to specifiy a root device in a disk image that we
create is to use PARTUUID rather than /dev/sda2.  As background, both
GPT and MBR tables provide valid UUID values for each partition and the
Linux Kernel contains the logic to parse this value.  With this change
we can now boot the default disk images when used as any valid block
device that the included kernel uses.  This for example means that
VirtualBox can be used to run vmdk without changes as it uses IDE for
the virtual disk controller.

Cc: Matt Porter <mporter@konsulko.com>
(From OE-Core rev: 8a58e9bb3e76a9962f1d14a8bdd3f7de675c3492)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:20 +01:00
Tom Rini
9d07c736e3 image: Convert vmdk/vdi/qcow2 to strict CONVERSION_CMD types
The vmdk/vdi/qcow2 IMAGE_FSTYPEs predate wic.  As such, they provide
some similar underlying functionality in order to produce a "disk" image
that in turn can be converted into different formats that various
hypervisor types work with.  They do not however provide the ability for
other disk image types to be converted into these same output types.
Furthermore, they are less flexible than what wic does provide.  This
drops the old style vmdk/vdi/qcow2 types and re-introduces them under
the CONVERSION_CMD framework.  The equivalent of vmdk is now wic.vmdk
and so forth for the other types.

(From OE-Core rev: 929ba563f1bc7195c4981b8e139c432b2cc388ea)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:20 +01:00
Jonathan Liu
408c65613a wic: ensure generated disk system identifier is non-zero
Zero may be interpreted as no MBR signature present and another
partitioning program might install a new MBR signature.

(From OE-Core rev: f53b5555d6eaa171bc5882cfee807cf62576862d)

Signed-off-by: Jonathan Liu <net147@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:20 +01:00
Ed Bartosh
ae4ee4d511 wic: add /boot mount point to fstab by default
wic avoided adding /boot to fstab for no reason.
This exception was hardcoded in the wic code.

There is no need for this as mountpoint in .wks file is an optional
field. It can be used only if user wants to have partitions
automatically mounted on system boot.

[YOCTO #11662]

(From OE-Core rev: 2376b05512ddb8c4ec3aaf1df11071f536a76bd9)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:19 +01:00
Ed Bartosh
092f4fafaa wic: rootfs: make copied rootfs unique
Used unique suffix (line number from .wks file) for the
copied rootfs directory to avoid possible conflicts.

(From OE-Core rev: 01efc234a8caab67ed3138ab2de9bbd82ce97b44)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:19 +01:00
Ed Bartosh
d4e60fe6a4 wic: rootfs: fix rootfs path reporting
wic gets rootfs paths from partition object property
'rootfs_dir' and shows them in final report.

rootfs plugin sets this property to the temporary path,
which causes temporary paths appearing in the report.

Changed the code to prevent storing temporary rootfs path
in part.rootfs_dir. This should fix the report.

(From OE-Core rev: 28d2d7d6f79df08431187c7debaab2a3fa516671)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:19 +01:00
Ed Bartosh
5541191b99 wic: use absolute paths in rootfs plugin
Using relative paths can cause copyhardlinktree API to fail as
it changes current directory when working. Converted all paths
to absolute paths using os.path.realpath.

(From OE-Core rev: a1c83cebe986e211dfc31be5cbd748f53fc298df)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:19 +01:00
Ed Bartosh
c4f9d8151f wic: copy rootfs directory before changing fstab
wic updates /etc/fstab on root partition if there are
valid mount points in .wks

When wic runs from bitbake this can cause incorrect results
or even breakage of other tasks working with the same rootfs
directory in parallel with do_image_wic.

Implemented copying rootfs directory to a temporary location
using copyhardlinktree before updating fstab to avoid conflicts with
other tasks working with the same rootfs directory.

(From OE-Core rev: 92e1c7d47e695eb4ce1a863cd0f6c49dca1c2339)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:19 +01:00
Leonardo Sandoval
c147d7753f bsp/help: include a warning showing deprecation of create-layer plugin
This plugin will be removed starting 2.5 development in favour
of using 'bitbake-layers create-layer' script/plugin, offering a single
script to manage layers.

(From meta-yocto rev: 76dd79e345d8edb22fc7aefd31f2a1f150916718)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Robert Yang
74b8f86a05 runqemu: chmod 0o777 for lockdir
Multi-users may run qemu on the same host, all of them should be able to
create or remove lock in lockdir, so set lockdir's mode to 0o777.

Note, os.mkdir()'s mode is default to 0o777, but the current umask value is
first masked out, so use os.chmod() to set it.

(From OE-Core rev: 4a5d21dbdc88982c2c90e660811b84983eaebeb7)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 22:36:52 +01:00
Robert Yang
5c7d025ced runqemu: validate combos
Error out ealier if the combos is invalid, e.g.:
$ runqemu tmp/deploy/images/qemux86/bzImage-qemux86.bin tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic

This will fail at kernel panic, no we check and error out early. We can
add other checkings in the future.

[YOCTO #11286]

(From OE-Core rev: 8c6f253dfb4899324e91dd5d082190909e2bd25d)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 22:36:52 +01:00
Robert Yang
0420171c08 runqemu: check tar.bz2 and .tar.gz
Handle them as nfs, so that cmd like the following can be boot:
$ runqemu tmp/deploy/images/qemux86/core-image-minimal-qemux86.tar.bz2

[YOCTO #11286]

(From OE-Core rev: 552093d1f60ca335d95bcfc9d6070ec551ebe6c0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 22:36:52 +01:00
Robert Yang
b9568cdd46 runqemu: check qbconfload before running bitbake
If qbconfload (.qemuboot.conf is found) is present, we can get
DEPLOY_DIR_IMAGE from it rather than "bitbake -e".

(From OE-Core rev: 89e97033a8a27a695567c321ed0ebf17f23f8d9b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 22:36:52 +01:00
Robert Yang
b8c208e8e8 runqemu: add --debug and --quiet
And move some debug info into logger.debug(), this can make it easy to
read key messages like errors or warnings.

I checked meta/lib/oeqa/ they don't depend on these messages. And I have
run "oe-selftest -a", it doesn't break anything.

[YOCTO #10474]

(From OE-Core rev: e696425e7627edada128b40304fddc84d8d56ba7)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 22:36:52 +01:00
Ed Bartosh
7a2730fe7a wic: add description of --mkfs-extraopts
Updated help contents with the description of new
wks option --mkfs-extraopts

(From OE-Core rev: 69c95f74013d0d8b199abc38ba0d6f3ff952eed3)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:38 +01:00
Ed Bartosh
9f481f5ac9 wic: implement wks option --mkfs-extraopts
This option specifies extra options to pass to mkfs.<fstype> utilities.

[YOCTO #11709]

(From OE-Core rev: 67b7c67edba305fbd31967baa10d27c2e603ec77)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:38 +01:00
Chang Rebecca Swee Fun
c2318ef416 devtool: standard: enable options for PREMIRRORS and MIRRORS
Since we have provide an option to manually enable PREMIRRORS and MIRRORS
in recipetool, we need to make sure devtool is having the same options
as devtool uses recipetool in creating new recipes.

(From OE-Core rev: 198bddeb928a318c3ad168bcb4f83b5cd9a604fb)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:37 +01:00
Chang Rebecca Swee Fun
819179d775 devtool: find-recipe: enable new subcommand for devtool
devtool find-recipe will prints out the path to the recipe
in a workspace.

This subcommand can also help to find recipe outside of
current workspace using "-a" or "--any-recipe" option.

This enhancement helps developer to get the recipe path
when working with devtool.

[YOCTO #11434]

(From OE-Core rev: 5a16b3c804c5eca331a1c08a7ce31a54909af105)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:37 +01:00
Chang Rebecca Swee Fun
5778e35378 scriptutils: pass in logger as parameter
logger was not defined in scriptutils.py based on the
observation in python traceback.

Traceback (most recent call last):
  File "/workdir/poky/scripts/devtool", line 351, in <module>
    ret = main()
  File "/workdir/poky/scripts/devtool", line 338, in main
    ret = args.func(args, config, basepath, workspace)
  File "/workdir/poky/scripts/lib/devtool/utilcmds.py", line 55, in
edit_recipe
    return scriptutils.run_editor(find_recipe(args, config, basepath,
workspace))
  File "/workdir/poky/scripts/lib/scriptutils.py", line 141, in
run_editor
    logger.error("Execution of '%s' failed: %s" % (editor, exc))
NameError: name 'logger' is not defined

We pass in logger as parameter to run_editor() from where it has
been called (devtool/utilcmds.py and recipetool/newappend.py),
which both modules already has logger setup.

(From OE-Core rev: 21f04b61973dd9029f0e6bff5445e31cd762bf32)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:37 +01:00
Chang Rebecca Swee Fun
2c035b266f scriptutils: exit politely when no text editor available
devtool edit-recipe now has ugly tracebacks if executed without an
editor available. This happens in the build containers whenever no
text editor is available.

subprocess.check_call will run text editing command with recipe path
provided. It will wait for command to complete. If the return code
was zero then return, otherwise raise CalledProcessError exception.

This enhancement will suppress the traceback by catching the exception
and prompt the error messages in a proper manner shown below:

pokyuser@59c99c507238:/workdir/docker-dbg$ devtool edit-recipe ifupdown
/bin/sh: 1: vi: not found
ERROR: Execution of 'vi' failed: Command 'vi
"/workdir/docker-dbg/workspace/recipes/ifupdown/ifupdown_0.8.16.bb"'
returned non-zero exit status 127

[YOCTO #11434]

(From OE-Core rev: c536ccfd162fc8871838fa229012c562e3f8e0d9)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:37 +01:00
Ed Bartosh
8fb8a21b7d wic: fix calculation of partition number
Total number of partitions should be taken into account when calculating
real partition number for msdos partition table. The number can be
different for the 4th partition: it can be 4 if there are 4 partitions in
the table and 5 if there are more than 4 partitions in the table. In the
latter case number 4 is occupied by extended partition.

[YOCTO #11790]

(From OE-Core rev: 5689139b3cd862e2df49f6b21171f513e8a46c60)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:37 +01:00
Aníbal Limón
3aa299d51b argparse_oe: Add int_positive type
Sometimes only expect positive values from cmdline so it's better
to filter at parsing cmdline step instead of validate later.

(From OE-Core rev: 3ef5b518febd047bf90a0955fa2b9fb78ba6dde5)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Richard Purdie
e69665f3e5 recipetool: Setup logging before calling prepare
This avoids test failures like:

======================================================================
ERROR [0.946s]: test_recipetool_load_plugin (recipetool.RecipetoolTests)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/recipetool.py", line 514, in test_recipetool_load_plugin
    with open(srcfile) as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'NOTE: Starting bitbake server...\n/media/build1/poky/meta-selftest/lib/recipetool/bbpath.py'
INFO - ----------------------------------------------------------------------

which comes from:

$ recipetool --quiet pluginfile
NOTE: Starting bitbake server...
/media/build1/poky/meta-selftest/lib/recipetool/bbpath.py

since there is corruption in the output despite the --quiet option. Setting
the logging up before calling tinfoil.prepare() resolves this.

(From OE-Core rev: e124eb73e6a66abbf43dffd82b834e906163614b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Richard Purdie
821cb0b819 devtool: Setup logging before calling prepare
This avoids test failures like:

INFO - ======================================================================
INFO - FAIL [1.755s]: test_devtool_layer_plugins (devtool.DevtoolTests)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/devtool.py", line 1354, in test_devtool_layer_plugins
    self.assertEqual(result.output, s[::-1])
AssertionError: "NOTE: Starting bitbake server...\noY senu[36 chars]rciM" != "oY senuZ s'enoynA morF tiforP oN edaM tfosorciM"
- NOTE: Starting bitbake server...
  oY senuZ s'enoynA morF tiforP oN edaM tfosorciM
INFO - ----------------------------------------------------------------------

since there is corruption in the output. Setting the logging up before
calling tinfoil.prepare() resolves this.

(From OE-Core rev: 3c479fb17ae4d3e7e5f0889af0f68257ef66475c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Richard Purdie
ac4682fc9c oe-pkgdata-util: Setup logging before calling prepare
This avoids test failures like:

INFO - ======================================================================
INFO - FAIL [1.046s]: test_find_path (pkgdata.OePkgdataUtilTests)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/media/build1/poky/meta/lib/oeqa/selftest/cases/pkgdata.py", line 50, in test_find_path
    self.assertEqual(result.output, 'zlib: /lib/libz.so.1')
AssertionError: 'NOTE: Starting bitbake server...\nzlib: /lib/libz.so.1' != 'zlib: /lib/libz.so.1'
- NOTE: Starting bitbake server...
  zlib: /lib/libz.so.1
INFO - ----------------------------------------------------------------------

since there is corruption in the output. Setting the logging up before
calling tinfoil.prepare() resolves this.

(From OE-Core rev: 39c1bf1feccc8dca57160a1e3cd8a50e8c30ef02)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Paul Eggleton
0d72748e81 recipetool: create: refactor code for ensuring npm is available
Across devtool and recipetool we had an ugly set of code for ensuring
that we can call an npm binary, and much of that ugliness was a result
of not being able to run build tasks when tinfoil was active - if
recipetool found that npm was required and we didn't know beforehand
(e.g. we're fetching from a plain git repository as opposed to an npm://
URL where it's obvious) then it had to exit and return a special result
code, so that devtool knew it needed to build nodejs-native and then
call recipetool again. Now that we are using real build tasks to fetch
and unpack, we can drop most of this and move the code to the one place
where it's still needed (i.e. create_npm where we potentially have to
deal with node.js code in a plain source repository).

(From OE-Core rev: 8450de16ddb02d863204b411a94c6d84e0f88817)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Paul Eggleton
e4346e8be5 recipetool: create: reimplement fetching with normal fetch/unpack tasks
Now that we have the ability to run the tasks in a more standard context
through tinfoil, change recipetool's fetching code to use that to fetch
files using it. This has the major advantage that any dependencies of
do_fetch and do_unpack (e.g. for subversion or npm) will be handled
automatically. This also has the beneficial side-effect of fixing a
recent regression that prevented this fetch operation from working with
memory resident bitbake.

Also fix devtool's usage of fetch_uri() at the same time so that we can
completely replace it.

Fixes [YOCTO #11710].

(From OE-Core rev: 9a47a6690052ef943c0d4760630ee630fb012153)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Paul Eggleton
7d474e27bc recipetool: create: eliminate second fetch for packages
When dealing with package files (.rpm, .ipk etc.) we need to unpack them
ourselves to get the metadata, which is thrown away when the fetcher
unpacks them. However, since we've already fetched the file once, I'm
not sure as to why I thought I needed to fetch it again - we can just
get the local path and then unpack it directly.

(From OE-Core rev: be45e9b17e9dbc8c2594d3a939be377ab0720a7c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Paul Eggleton
047d37633b recipetool: create: ensure meaningful error for malformed tarballs
If you pointed recipetool at a URL that should be a tarball e.g.
https://tls.mbed.org/download/start/mbedtls-2.4.2-apache.tgz but instead
it returns an HTML page, we try to unpack it, gzip complains but the
operation doesn't seem to fail - instead we just get back an empty
source tree. Change the checks to account for this - if the source tree
is empty, check if the downloaded file in DL_DIR looks like an HTML file
and error accordingly if it is. If it's not, error out anyway because
no source was unpacked and it should have been (otherwise we just
blindly set up EXTERNALSRC for this which is pointless).

Fixes an aspect of [YOCTO #11407].

(From OE-Core rev: 8496113b63d5a5d1f99056610c0fdb972a6200d4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Paul Eggleton
a9848fdd46 devtool: extract: refactor kern-tools-native handling
When extracting linux-yocto kernel source, we don't need to dance around
shutting down and starting up tinfoil anymore, we can just execute the
tasks as needed when needed using tinfoil's new build_targets()
function. This allows us to tidy up the code structure a bit.

(From OE-Core rev: 5c7f5031023fb74b5f2f26d6b3c829981f2f54d2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Richard Purdie
0ad65be8bf oe-init-build-env-memres: Drop it
With the new server structure we no longer need this separate
environment init script. Just set BB_SERVER_TIMEOUT to be greater
than zero and bitbake will remain in memory and the UI will auto-reconnect
to it.

Also clean out the old shutdown code from oe-init-build-env which
also doesn't make sense now.

(From OE-Core rev: a17724d687f9c04dc21f2f369da2b492f7119ba0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Stanley Cheong Kwan, Phoong
0df60c7de2 recipetool: git reformat URI mangling & parameter stripped
recipetool seems to be mangling and stripping out the parameters for git
URI. This will fix this issue as well as resolve the conflict of
protocol parameter added by user. If a user adds their own protocol as
an argument, it'll be honored.

[YOCTO #11390]
[YOCTO #11391]

(From OE-Core rev: 0cd2fc8ca278ebaa76de95545eef26a07b350c8e)

Signed-off-by: Stanley Cheong Kwan, Phoong <stanley.cheong.kwan.phoong@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:38 +01:00
Ed Bartosh
c23be1034b wic: isoimage-isohybrid: use grub-efi from deploy dir
isoimage-isohybrid plugin tries to build grub-efi in its working
directory if it can't find efi binary. Wic should avoid doing anything
in working directories of other recipes. It should use artifacts from
the image deployment directory instead.

(From OE-Core rev: 9a1709278de877085a8d92f6361624e279a603bf)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Ed Bartosh
f58ea8c814 wic: isoimage-isohybrid: check result of glob()
isoimage-isohybrid plugin uses result of glob call to
get path to initrd image. When glob returns empty list
the plugin crashes with IndexError.

Checking if result of glob call is not empty should fix
the breakage.

(From OE-Core rev: ad02f253f08a3da3fa5c86ae4f6ba7f94b070578)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Ed Bartosh
116e14fc45 wic: build wic-tools only if wic is run manually
When wic can't find native sysroot it tries to build wic-tools
However, it's not possible when wic is run from bitbake.
Moreover, it's not even feasible anymore as wic-tools should be
used only when wic is run manually.

Checked if wic is run manually before building wic-tools.

(From OE-Core rev: 522ac21831944b06fbcc372ac61762576d792c22)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Ed Bartosh
57a334c10a wic: isoimage: do not remove temp directory
Removed isodir subrdirectory instead of removing temporary
working directory as working directory can contain copy of
rootfs partition and shouldn't be removed by any plugin.

(From OE-Core rev: fc12ce9d1b92cc0104cf456af1e3d5f146b9219d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Ed Bartosh
b6dd87f2a8 wic: get rid of using wic-tools
At the moment, when building images with IMAGE_FSTYPES=wic one ends up
depending on wic-tools and thus syslinux and grub-efi even when not
using those at all. Ideally, building an image with wic should only
build the tools and components really needed.

The problem is that "wic-tools" is needed also for the manual
invocations of wic, in which case everything that might be needed has to
be built in advance.

Replaced dependency on wic-tools with dependency to a much shorter set
of tools that wic uses almost for any image: 'parted', 'gptfdisk',
'dosfstools' and 'mtools'.

[YOCTO #11552]

(From OE-Core rev: 33ca15b94dbe7204c556c4b5526edd529f6d85f4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Ed Bartosh
1b11a653d0 runqemu: check exit code of 'ls -t'
Used check_output instead of Popen as it raises CalledProcessError
exception when command exits with non-zero exit code.

Catched the exception to produce user-friendly output.

[YOCTO #11719]

(From OE-Core rev: dac68d2323b0b630c019ce4d5256ed567eaf00da)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:36 +01:00
Ed Bartosh
96a87592d4 runqemu: reworked exception handling
Introduced custom RunQemuException that script raises on known
errors. This exception is handled in one place and prints
error output without printing Python traceback. This shoud make
error output less scary for the end user.

Handling of unknown errors has not been changed - both error and
traceback will be printed.

Reimplemented OEPathError exception code to handle it similarly
to RunQemuException.

Moved exception handling code into main() to keep it in one place.

[YOCTO #11719]

(From OE-Core rev: a779a382b66e7b43ac53286758b4370dc14b193b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:36 +01:00
Choong YinThong
5b23e19841 linux-yocto-custom: include a user feature description file
Missing user feature description file
when select linux-yocto-custom in script/yocto-bsp
and causes failure in script/yocto-kernel feature add.

[YOCTO #11585]

(From meta-yocto rev: 9939e7eacf8a205fb0b10de8a6cc065a8048b103)

Signed-off-by: Choong YinThong <yin.thong.choong@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 15:03:10 +01:00
Peter Kjellerstedt
2d8e3e9120 oe-pkgdata-util: package-info: Allow extra variables to be displayed
By specifying the -e <var> option one or multiple times, extra
variables available in the pkgdata can be displayed, e.,g,
`oe-pkgdata-util package-info -e SUMMARY -e LICENSE ...`. The extra
variables displayed are quoted since some of them may contain
whitespace.

(From OE-Core rev: 6f054fcf08f6d5aeb6b5ade74ac5cc61a99fb8fe)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 14:38:14 +01:00
Patrick Ohly
76f149e384 yocto-compat-layer.py: make signature check code reusable
This moves the main content of test_signature into a helper
function. It can be reused by arbitrary tests that need to do
a before/after signature comparison. Long-term this might even
be useful in oeqa itself.

(From OE-Core rev: ecc9a1f9ceec9996aeb2c602846d51277de0b4a5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 14:38:14 +01:00
Patrick Ohly
17266ae2dd yocto-compat-layer.py: allow README with suffix
It may be useful to append a suffix denoting the file format. For
example, README.rst is rendered differently when viewed on Github, and
also helps editors to switch to a mode more suitable for the format.

The tests uses a file pattern to find the README file(s) and treats
the one with the shortest name as the main one which must not be
empty.

(From OE-Core rev: 501b5b7f338396a4a115355b8a78ae5b03f67d9a)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 14:38:14 +01:00
Patrick Ohly
d58c365a07 yocto-compat-layer.py: add test_world
"test_signatures" ignores wold build breakage for the sake of
reporting differences also when a world build is broken. Therefore we
need a dedicated test that a world build at least theoretically can
proceed without obvious parse time problems (dependencies, parse
errors, dangling .bbappends, etc.).

This is similar to the BSP test_machine_world. The difference is
that test_world doesn't change the MACHINE.

(From OE-Core rev: 1ca35d8571a92c7f8f80c909ca38666da82eb929)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 14:38:14 +01:00