3090 Commits

Author SHA1 Message Date
Richard Purdie
05e6558514 resulttool/resultutils: Fix unicode error handling
This error handling didn't work as expected since upon failure it would
inject bytestreams back into the code leading to tracebacks.

Instead, ignore the decode errors. Fixes:

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 78, in <module>
    sys.exit(main())
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 72, in main
    ret = args.func(args, logger)
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/store.py", line 70, in store
    resultutils.save_resultsdata(results, tempdir, ptestlogs=True)
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/resultutils.py", line 178, in save_resultsdata
    f.write(sectionlog)
TypeError: write() argument must be str, not bytes

(From OE-Core rev: b63955977ebbf9fba291faa1b30c8dba9bd52869)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-15 11:55:45 +01:00
Lee Chee Yang
51586b5612 wic: align layer plugin path
wic look for external layer source plugin under scripts/lib/
while other tools look for lib/.

allow wic to check for source plugin at both scripts/lib/ and
lib/ to align with other tools while avoid breaking any existing
source plugin in external layer.

[YOCTO #13056]

(From OE-Core rev: 8351cb19e8a83157143b1df61cd9bbb71282e9fc)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 986baff26bd96a6265f5fe2d631818fff9f66374)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-29 16:11:56 +01:00
Chee Yang Lee
5531ffc566 wic/direct: reserve 2 sector for extended partition
extended partition should reserve at least 2 unallocated sectors
between the start of the extended partition and the start of a
logical partition.

[YOCTO #13658]

(From OE-Core rev: 884c8a0e50a49bdca5e048197a9dc0ff37cc8d1a)

(From OE-Core rev: 0743dcd0cd70dee87a49d3bcd017168352e60982)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 590555e3b8)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-30 17:41:56 +01:00
Armin Kuster
f42a79c02d wic/engine: lets display an error not a traceback
If the requested partition does not exist in this request "wic ls {path}:pnum"
display a nice message not a trackback

Also fix displaying the pnum and not "%s"

(From OE-Core rev: 29a1d9bed5bf7ed024870a0323f9afdf88346e4d)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 15d1722950a22649905cf8a5789d3cfe48a2a892)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-16 16:44:53 +00:00
Tom Hochstein
7435786516 devtool/standard.py: Allow recipe to disable menuconfig logic
u-boot.inc supports u-boot recipes with or without menuconfig [1].
However, running devtool on a u-boot recipe that does not support menuconfig
results in an error:

cp: cannot stat '/home/r60874/upstream/fsl-xwayland/tmp/work/imx8mmevk-fsl-linux/u-boot-imx/2018.03-r0/u-boot-imx-2018.03//.config': No such file or directory

The problem is the devtool logic assumes that any recipe with a do_menuconfig task
will generate a .config in do_configure().

Fix the problem by removing the assumption with a flag that the recipe can control,
like this:

do_configure() {
    if [ menuconfig-supported ]; then
        ...
    else
        DEVTOOL_DISABLE_MENUCONFIG=true
    fi
}

[1] 11278e3b2c

(From OE-Core rev: 803391ef7ba662a5ee58609d9c81aeffa494287c)

Signed-off-by: Tom Hochstein <tom.hochstein@nxp.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-11 23:05:12 +00:00
Richard Purdie
167bd3e6b6 scripts/oe-build-perf-report: Avoid buildstats warning
When a comparision branch is set we see warnings. If two branches are used,
look for buildstats information from both to avoid this kind of warning:

INFO: Parsing buildstats from 'refs/notes/buildstats/perf-centos7.yoctoproject.org/master-next/qemux86'
WARNING: Buildstats not found for perf-centos7.yoctoproject.org/master/qemux86/56808-g08220028e52992dcba667fc97bf3afe8be1949fb/0
WARNING: Buildstats not found for perf-centos7.yoctoproject.org/master/qemux86/56808-g08220028e52992dcba667fc97bf3afe8be1949fb/1
INFO: Buildstats were missing for some test runs, please run 'git fetch origin refs/notes/buildstats/perf-centos7.yoctoproject.org/master-next/qemux86:refs/notes/buildstats/perf-centos7.yoctoproject.org/master-next/qemux86' and try again

(the warning is due to stats being looked for in both branches, choose
the other branch and different warnings would be shown)

(From OE-Core rev: 232b2f6dabaedff087df15a7e6d431dd9f314b4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit af58c18fcd0451540dd77ee862956b2b9dee17c4)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-28 11:51:03 +00:00
Kalle Lampila
88448accde wic/filemap: If FIGETBSZ iotctl fail, failback to os.stat
Some file systems don't support fetching the block size (notably the
file system Docker uses for containers), so if iotctl() fail, try to use
failback via os.stat() to get block size.

(From OE-Core rev: 996013b4e5d3cd1c054e87539ac1a8d95581a87f)

Signed-off-by: Kalle lampila <kalle.lampila@lempea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e219f5175177a640dd62833082ea19adc1c13d42)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-28 11:51:03 +00:00
Alexander Kanavin
e8e3b2ce49 Revert "devtool/standard.py: Not filtering devtool workspace for devtool finish"
This reverts commit 41d225f4a3.

Unfortunately this change broke 'devtool upgrade' functionality,
causing 'devtool finish' to write out an upgraded recipe that no
longer includes the original upstream source in SRC_URI.

(From OE-Core rev: 2d6e55192dba0bf7f6e23e5ab5b3dbc68835bb28)

(From OE-Core rev: 2bb221ee5689f13d44b7452738f5a97baa1815a1)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-25 21:37:40 +00:00
Chee Yang Lee
b3d7793ccd wic/engine: use 'linux-swap' for swap file system
[YOCTO #13312]
see https://bugzilla.yoctoproject.org/show_bug.cgi?id=13312

wic/engine.Disk._get_part_image was looking at variable fstypes for
supported fstype which is 'swap' but image build with 'linux-swap'.
supported fstype should be 'linux-swap'.

(From OE-Core rev: 759a37e315d72e462a617f1cda326b8f198e3d4e)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7e6da22fe4faf841bcec02e55f376b4dae04d6a8)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-29 09:08:18 +00:00
Eugene Smirnov
6fb5afb65d wic/rawcopy: Support files in sub-directories
If the source file is located in a subdirectory of DEPLOY_DIR
rawcopy will currently fail in sparse_copy function on
open(dst_fname, 'wb'), as the parent directory for destination
file does not exist.

This patch helps to avoid that by recursively creating
parent directories.

(From OE-Core rev: 03796926046e5b5b57349d66de6bb009a34d2cb2)

Signed-off-by: Eugene Smirnov <evgenii.smirnov@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 073c435644091c2801e45c6d02afa917de575082)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-29 09:08:17 +00:00
Joerg Vehlow
5fb336e957 runqemu: Remove disabling of high resolution timer
The option 'highres=off' sneaked itself into the runqemu script for all
configurations, where the root filesystem type is not 'cpio' or 'cpio.gz'.
See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13590

(From OE-Core rev: f71b1a96529d658d1c6c7dc04dca1957ea647e4e)

Signed-off-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-29 09:08:17 +00:00
Alexander Kanavin
527456a57a runqemu: unset another environment variable for 'egl-headless'
Some host distributions (opensuse for example) are using 'pkgconf',
which, unlike the original pkg-config, appends PKG_CONFIG_SYSROOT_DIR
to every directory from the .pc file.

(From OE-Core rev: 5f51c9303940423b05ba8cfeaa1c9c2b24e36a53)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-29 09:08:17 +00:00
Richard Purdie
4964e42e19 scripts/gen-lockedsig-cache: Don't list paths which don't exist
This avoids failures seen on the autobuilder when generating eSDKs
and release sstate copies.

(From OE-Core rev: 8ffe6ca984e034976322beea9b16c92d46708d15)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-09 22:28:37 +01:00
Michael Cooper
7a1d68d136 wic/direct: Partition numbering is broken for MBR primary partition #4
When wks describes extra partitions that aren't in the partition
table (e.g. boot loader) and exactly four primary MBR partitions, the
last partition gets added to fstab as partition #5 instead of #4.

[YOCTO #13560]

(From OE-Core rev: 7537580b3dd21bd512fb26e56e92b6553c549fa8)

Signed-off-by: Michael Cooper <michaelcooper81@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-08 11:22:24 +01:00
Jaewon Lee
41d225f4a3 devtool/standard.py: Not filtering devtool workspace for devtool finish
All devtool commands right now are filtering out the devtool workspace
bbappends in build/workspace/appends when calling parse_recipe. While
this may make sense for devtool add and modify, we need devtool finish
to include those appends.

A specific breakage that is caused because devtool finish filters devtool
appends is the cmake/cml1 flow where a file is created in the WORKDIR
that finish needs access to, to commit those files. Particularly for
git packages with SRCPV in PV, SRCPV is only changed to 999 when using
external source, hence when creating the cfg or cmake config files using
for instance bitbake -c diffconfig, these files are created in the
git999 workdir correctly (as in the devtool bbapends, we are inheriting
externalsrc class). But when devtool finish is run, the devtool appends
are not parsed, hence SRCPV is not changed to 999 and devtool is looking
for the fragment files in the wrong WORKDIR.

Changing the parse_recipe call just in devtool finish to not filter out
the devtool workspace.

Fixes [YOCTO #13533]

(From OE-Core rev: aa94f00eec64ef936acc80dfa826fd309daa294f)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-08 11:22:24 +01:00
Konrad Scherer
e1c173621a gen-lockedsig-cache: Replace glob lookup with hash to filename lookup
Using the glob function to map signatures to sstate files is very slow
when the sstate is large and accessed over nfs. The lookup now only
loads the necessary prefixes and doesn't use glob as all.

Unfortunately I don't have access to the systems where the performance
isse was noticed and on my test system the glob is fast enough that
the performance numbers aren't useful. I could verify that file list
returned by the new code is the same.

[YOCTO #13539]

(From OE-Core rev: ad36335b8592e0387dd36066920cd5ffefd375f8)

Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 20:22:25 +01:00
Nathan Rossi
ec1104fd7f oeqa/core/case.py: Encode binary data of log
Do not decode the log content into a string only to re-encode it as
binary data again. Some logs might un-intentionally contain bytes that
do not decode as utf-8, as such preserve the log file content as it was
on disk.

Handle the decoding on the resulttool side, but also handle the failure
to decode the data.

(From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Peter Kjellerstedt
5af64b55f3 devtool: finish: Add suppport for the --no-clean option
This works just like the already existing --no-clean option to the
`devtool reset` command.

(From OE-Core rev: 4433d3133ce53cdf7db17f5e7914fe8c4d78dba3)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:17 +01:00
Niclas Svensson
a70e1fdba2 devtool: finish: Keep patches ordered when updating bbappend
The _get_patchset_revs() function returns the patches in an
OrderedDict to keep them ordered. However, this information was lost
when the patches were added to the bbappend file.

(From OE-Core rev: 69b7a2ba3af1280cc220ab236032f8466246ee93)

Signed-off-by: Niclas Svensson <niclas.svensson@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:16 +01:00
Alessio Igor Bogani
b404c0cd3c wic: Using the right rootfs size during prepare_rootfs
The commit 8e48b4d6c4 makes wic ignores IMAGE_ROOTFS_SIZE for rootfs
size and makes it uses the computed one only. Re-add support for
IMAGE_ROOTFS_SIZE variable and compute roots size only if the former
is not defined. Moreover the size of a provided directory with
--rootfs-dir="" in the .wks file should always be computed on the fly,
else every partition will be constrained to be the same size as what
ever value was in ROOTFS_SIZE.

(From OE-Core rev: 0217a0392b8cc534fcc0d760b7663a1845b306f5)

Signed-off-by: Alessio Igor Bogani <alessio.bogani@elettra.eu>
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:16 +01:00
Richard Purdie
cf37ca58c2 runqemu: Mention snapshot in the help output
This is a useful option but not documented in the help text.

(From OE-Core rev: 95bad67a802f956082bd0af5615c1dbaae01be0f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 12:09:02 +01:00
Alexander Kanavin
63f08ef496 runqemu: decouple gtk and gl options
This will allow not having to multiply these options for the sdl
frontend, instead combining them as needed.

(From OE-Core rev: 922eb5012364b1603338cfa617712b941e892bbf)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
4b0bf01c14 oe-git-proxy: fix dash "Bad substitution"
The script claims it works with dash, make sure that is actually the
case.

(From OE-Core rev: cb373201464f4a0a90482f62a24a4043abe73fd6)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
7ffd9ee9c3 oe-git-proxy: NO_PROXY suffix matching without wildcard for match_host
NO_PROXY can also contain just suffixes that do not start with a "*". We
failed to match those so far. Just add an extra "*" to also match those
suffixes. If one was there we get "**" which does not hurt.

(From OE-Core rev: 12f0cc209aaba48f846c62663e0b9e5efd253d71)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
1b970e222e oe-git-proxy: disable shell pathname expansion for the whole script
This truly fixes the issue that cbc148d5d9
wanted to solve, without breaking the iteration over multiple entries.

(From OE-Core rev: 477ee7e673684db988c66a75b6400e33509730b4)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
f107c9f815 Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"
This reverts commit cbc148d5d9.

The quoting causes H to be one string with spaces, so looping over
multiple entries does not work anymore.

(From OE-Core rev: 0fdc12dac6244be135ea519fe9c39109e7cfc6d6)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
49bb6cefb5 oe-git-proxy: allow setting SOCAT from outside
This allows to write selftests where we can mock the real socat.

(From OE-Core rev: ea2690e867ff11250d3dd143184432dd03909910)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Nathan Rossi
e08f657220 oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.

This class also provides default compression of log text and log files.

Also add support to resulttool for decoding/decompressing log files
embedded in the test results.

(From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Nathan Rossi
05e583ae84 resulttool: Handle multiple series containing ptestresults
Handle multiple results series having ptestresults content. The contents
are merged on a per-result basis where duplicates are ignored (with a
warning message printed). The 'ptestresults.sections' collection is also
merged on a per-suite basis.

(From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Joshua Watt
bff47225c6 resulttool: Add reproducible log extraction
Adds an argument to the log subcommand to extract the raw logs from the
reproducible selftest.

To prevent ambiguity, the "--raw" argument has been renamed
"--raw-ptest", although the old "--raw" argument is kept around for
compatibility.

[YOCTO #13324]

(From OE-Core rev: 7a4ebb361ff1efc22e7dafadfa60c98bc8a79ed4)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:58:10 +01:00
Drew Moseley
7ded301c3c runqemu: Fix typo in help text.
(From OE-Core rev: c745185b0c112530d1a3d1c57feea0754d68ebb2)

Signed-off-by: Drew Moseley <drew.moseley@northern.tech>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:58:10 +01:00
Nathan Rossi
4364a26db1 oe-selftest: Implement console 'keepalive' output
Similar to bitbake, implement a 'keepalive' output to the console to
ensure CI systems do not kill the process. The default timeout for
bitbake is 5000s.

(From OE-Core rev: 77939cca96fa5467c88eafa3ac0db2db4aef09d6)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 08:15:45 +01:00
Alistair Francis
b8c2575002 qemuriscv: Generate a wic rootFS with a larger filesystem
This allows us to generate a rootFS with a large filesystem for use with
QEMU.

(From OE-Core rev: e06439200e44999c1e2f88d7d6c651da13698ca7)

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-03 09:55:50 +01:00
Peter Kjellerstedt
397d1432cb devtool: Avoid failure for recipes with S == WORKDIR and no local files
When extracting the sources for a recipe that has S == WORKDIR and no
local files in the SRC_URI (which, e.g., can happen for a recipe with
a URI that has the unpack=false attribute), the extraction fails with
the following backtrace:

  Traceback (most recent call last):
    File ".../scripts/devtool", line 344, in <module>
      ret = main()
    File ".../scripts/devtool", line 331, in main
      ret = args.func(args, config, basepath, workspace)
    File ".../poky/scripts/lib/devtool/standard.py", line 762, in
    modify
      initial_rev, _ = _extract_source(srctree, args.keep_temp,
      args.branch, False, config, basepath, workspace,
      args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
    File ".../poky/scripts/lib/devtool/standard.py", line 647, in
    _extract_source
      bb.process.run('git %s commit -a -m "Committing local file
      symlinks\n\n%s"' % (' '.join(useroptions),
      oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree)
    File ".../poky/bitbake/lib/bb/process.py", line 178, in run
      raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
  bb.process.ExecutionError: Execution of 'git commit -a -m
  "Committing local file symlinks

  %% ignore"' failed with exit code 1:
  On branch devtool
  nothing to commit, working tree clean

This is because no files were found in the oe-local-files directory
and consequently no symbolic links were added using `git add`, but the
`git commit` command was still executed.

(From OE-Core rev: 3fdf304e72a1fb5de8bf9bc21e5b598fefb08648)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-01 22:33:08 +01:00
Alexander Kanavin
58e85c60cd qemu: switch to '-vga std' emulated hardware from vmware/cirrus for x86/mips
This is the qemu default since qemu 2.2, is generally supported better,
and is recommended by upstream. It also has already been in use for arm/risc
and ovmf.

Additional information:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13466
https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/

'-vga virtio' emulated hardware remains in use when virgl is enabled via a runqemu override.

Also, adjust the error whitelist, as there is a number of new messages
coming from the drivers that are not actual errors.

(From OE-Core rev: 73cb104f3307736f4922f2e0c9648f9b2d3b3b6b)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-01 22:33:07 +01:00
Mark Hatle
b4932a18ab yocto-check-layer: Allow any case for README file detection
It's become more commone for files to be named "readme" or "Readme" on github servers
in recent time.  So adjust the scanning to allow any mix of case.

(From OE-Core rev: afe46eca15b6ddfa15c75cb7b707d6dd9aae3eae)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30 17:10:28 +01:00
Alexander Kanavin
b0bad37101 qemu: set default RAM to 256M for all machines
There was a discussion about what amount of RAM is appropriate for a
default; the outcome was that for now it is still 256M. Some qemu machine
definitions have however set this to 512M so for the sake of
treating all architectures fairly, they are reset back to 256M.

Also runqemu is adjusted to use 256M if QB_MEM is not set at all.

http://lists.openembedded.org/pipermail/openembedded-core/2019-August/285900.html

(From OE-Core rev: 04c01b6cc5be3e6d45d0e04571640648a5655a8b)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30 17:10:28 +01:00
Nathan Rossi
6e48f9aa23 scripts/lib/resulttool/report.py: Add more result types
Add additional result types into the dictionary to handle dejagnu style
test results. These include PASS, FAIL, XPASS, XFAIL, UNSUPPORTED,
UNTESTED, UNRESOLVED and ERROR.

(From OE-Core rev: 684fc36402a23760b203f4761f284043031c799c)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-28 16:44:09 +01:00
Jaewon Lee
8f29534ccb devtool: build: Also run deploy for devtool build if applicable
Right now `devtool build` runs populate_sysroot and packagedata tasks.
Adding deploy to this list, if the recipe has the deploy task, so that
the newly built artifacts are available in the deploy directory.
Applicable only for packages with deploy task, such as kernel.

[YOCTO#13382]

(From OE-Core rev: b38a1328f0c7bc4b4102a05daee4058fd3214489)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 21:52:59 +01:00
Jon Mason
87138c4c2b resulttool: Prevent multiple results for the same test
Currently, if a test occurs multiple times over different series, the
code will sum these.  This can lead to confusion over the desired
results.  Change the code to report the redundant tests and skip adding
an additional values.

(From OE-Core rev: caeaa0648dff64c56a33f52e45e82bfab6719e3e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 21:52:59 +01:00
Alistair Francis
23662395e2 scripts/runqemu: Add support for the BIOS variable
Add support for specifying a BIOS the same way that the KERNEL variable
is specified. This includes specifying a QB_DEFAULT_BIOS variable.

(From OE-Core rev: fc2a2260aa22a81da6619b4affaf8ae0b5556a34)

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:29:02 +01:00
Richard Purdie
d67fbf68f6 yocto-check-layer: Ensure we use OEBasicHash as the signature handler
The layer checks are designed to work with OEBasicHash so ensure that handler
is in use rather than the new hash equivalency one as an example.

(From OE-Core rev: a10bf92516a4771e2dc49ba9f74323d7a87a1619)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:29:01 +01:00
Jason Wessel
7365a605ae wic: Make disk partition size consistently computed
When using different root directories with a wks file wic is using the
value from the original ROOTFS_SIZE which is not correct.  Example:

Number  Start   End     Size    File system  Name       Flags
 1      20.5kB  318MB   318MB   fat16        otaefi     legacy_boot, msftdata
 2      318MB   636MB   318MB   ext4         otaboot
 3      636MB   1709MB  1074MB  ext4         otaroot
 4      1709MB  2027MB  318MB   ext4         otaboot_b
 5      2027MB  3101MB  1074MB  ext4         otaroot_b
 6      3101MB  5249MB  2147MB  ext4         fluxdata

The partitions 1, 2, and 3 incorrectly inherit the size, instead of using
a computed size.  With the patch applied it is working properly:

Number  Start   End     Size    File system  Name       Flags
 1      20.5kB  14.5MB  14.5MB  fat16        otaefi     legacy_boot, msftdata
 2      14.5MB  65.3MB  50.8MB  ext4         otaboot
 3      65.3MB  1139MB  1074MB  ext4         otaroot
 4      1139MB  1190MB  50.8MB  ext4         otaboot_b
 5      1190MB  2264MB  1074MB  ext4         otaroot_b
 6      2264MB  4411MB  2147MB  ext4         fluxdata

As for the removal of the bb.warn, if the size is not specified, it is
not something to warn the end user about.  Some of my default images
make use of the head room + a computed directory size or while generating
images.

(From OE-Core rev: 8e48b4d6c4d0ed213089a7449fea63aa0656e786)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Jason Wessel
351a10ccd6 wic: Add partition type for msdos partition tables
In order to create a msdos partition table disk image that can auto
expand after the image is copied to an SD card, wic needs the ability
to have a primary partition as the last entry.  The desired use case
is to be able to create an A/B update partition image scheme with a
/var volume that can be auto expanded to the remainder of the SD card
at run time.

The typical .wks file will look similar to the following:

bootloader --ptable msdos
part / --source rawcopy --sourceparams="file=u-boot.imx" \
   --ondisk mmcblk --no-table --align 1 --size 1
part /boot --source bootimg-partition \
   --ondisk mmcblk --fstype=vfat --label boot --active --align 4 --size 16
part / --source rawcopy --sourceparams="file=imx6_boot.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaboot --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaroot --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6_boot.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaboot_b --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaroot_b --align 4 --type logical
part /var --source rawcopy --sourceparams="file=imx6_var.otaimg" \
    --ondisk mmcblk --fstype=ext4 --label fluxdata --align 4

Without the patch applied, wic will generate an SD card image that looks like:

Disk boot.img: 890940s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start    End      Size     Type      File system  Flags
 1      2056s    48001s   45946s   primary   fat16        lba
 2      48008s   132467s  84460s   primary   ext4
 3      132472s  454467s  321996s  primary   ext4
 4      454471s  890939s  436469s  extended               lba
 5      454472s  538931s  84460s   logical   ext4
 6      538936s  860931s  321996s  logical   ext4
 7      860936s  890939s  30004s   logical   ext4         boot

With the patch applied a primary partition can be created at the end
of the image which can be expanded to fill the free space on the media
where the image has been copied, which looks like:

Disk boot.img: 890940s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start    End      Size     Type      File system  Flags
 1      2056s    48001s   45946s   primary   fat16        lba
 2      48007s   860931s  812925s  extended               lba
 5      48008s   132467s  84460s   logical   ext4
 6      132472s  454467s  321996s  logical   ext4
 7      454472s  538931s  84460s   logical   ext4
 8      538936s  860931s  321996s  logical   ext4
 3      860936s  890939s  30004s   primary   ext4         boot

(From OE-Core rev: 56add7cc547e0113cdf980579d1421b14cc233e5)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Ricardo Neri
9b90717e91 runqemu: Add support to handle EnrollDefaultKeys PK/KEK1 certificate
The EnrollDefaultKeys.efi application (distributed in ovmf-shell-image)
expects the hypervisor to provide a Platform Key and first Key Exchange
Key certificate.

For QEMU, this is done by adding an OEM string in the Type 11 SMBIOS
table. The string contains the EnrollDefaultKeys application GUID followed
by the certificate string. For now, the string is passed in the command
line until QEMU understands OEM strings from regular files (please see
https://bugs.launchpad.net/qemu/+bug/1826200).

If runqemu detects it is given an OVMF binary with support for Secure Boot
(i.e., ovmf.secboot* binaries), extract the certificate string from the
OvmfPkKek1.pem certificate and modify the command-line parameters to
provide the key. Such certificate is created when building OVMF with
support for Secure Boot.

Cc: Ross Burton <ross.burton@intel.com>
Cc: Patrick Ohly <patrick.ohly@intel.com>
(From OE-Core rev: 5e47316ae62f7632fb62bc3b8093ac42f9e3541c)

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Changqing Li
f2a951f479 runqemu: fix get portlock fail for multi users
when runqemu with slirp option on same host with different
users, it will report PermissionError: [Errno 13] Permission
denied: '/tmp/qemu-port-locks/2222.lock'
and during handle this exception, another exception happened since
key not exist. Fix by check if key exist first

(From OE-Core rev: 56f30e5377ebe5cc4544f081e001934706a0d8d3)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-03 23:56:01 +01:00
Paul Eggleton
bbd51cbc4a scripts/create-pull-request: fix putting subject containing / into cover letter
If a single-commit series had a shortlog containing a "/" character then
that prevented putting the shortlog into the subject of the cover letter
message. Use a different separating character with the sed command (one
much less likely to appear) in order to fix it.

(From OE-Core rev: d4c3f93d7407ac1ea20b33149f20153972d631c0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-31 13:50:46 +01:00
Paul Eggleton
f3649983d3 scripts/create-pull-request: improve handling of non-SSH remote URLs
When attempting to create a pull request, we look at the remote URL in
order to extract information to include in the cover letter. However,
the assumption was that the remote is an SSH URL i.e. containing '@'
which is not always the case (the pull and push URLs might be different,
or we might be pushing via https) - if it wasn't the script just gave up
leaving you to manually edit the URLs in the email. With a few minor
tweaks to the regexes the script will work for these cases as well.

(From OE-Core rev: 675e88e6e0bbd5ab2dcd4bdf97b0de59925a1be6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-31 13:50:46 +01:00
Changqing Li
09af4dafc7 runqemu: add lockfile for port used when slirp enabled
There is race condition when multi qemu starting with slirp,
add lockfile for each port to avoid problem like:

runqemu - ERROR - Failed to run qemu: qemu-system-x86_64: Could not set up host forwarding rule 'tcp::2323-:23'

[YOCTO #13364]

(From OE-Core rev: ceb3555a40ba06e58914465376aaf41392c12a7c)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-26 08:41:38 +01:00
Daniel Ammann
d3ed42ef27 image_types: Remove remnants of hdddirect
hdddirect was removed in commit 929ba563f1bc7195c4981b8e139c432b2cc388ea.

(From OE-Core rev: 7a91a9818057d69cba32db2428ffc1a1ebb3876e)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-26 08:41:38 +01:00