Commit Graph

2979 Commits

Author SHA1 Message Date
Changqing Li
030b0223aa runqemu: add lockfile for port used when slirp enabled
There is race condition when multi qemu starting with slirp,
add lockfile for each port to avoid problem like:

runqemu - ERROR - Failed to run qemu: qemu-system-x86_64: Could not set up host forwarding rule 'tcp::2323-:23'

[YOCTO #13364]

(From OE-Core rev: ceb3555a40ba06e58914465376aaf41392c12a7c)

(From OE-Core rev: 9f9657683df90c18c1dfc7e65715b134a44a9d5a)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
a872c96192 scripts: Remove deprecated imp module usage
The imp module is deprecated, port the code over to use importlib
as recently done for bb.utils as well.

(From OE-Core rev: f3ba6cee5927c7475c3dc47658fa0548aec52115)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-29 23:50:43 +01:00
Chee Yang Lee
34503c75b0 wic/bootimg-efi: replace hardcoded volume name with label
volume name should refer to --label in .wks.
Replace the hardcoded volume name  with label.
set "ESP" as default name when no lable specified.

(From OE-Core rev: becb0a3f855eff7700fa284a0a7981d6a260a1c5)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Joshua Watt
2d1077bb4f resulttool: Add option to dump all ptest logs
Adds an option to dump all the ptest logs to individual files in a
specified directory. If multiple test runs are present, the
'--prepend-run' argument will create separate directories for each test
run under the target directory and put the logs there to prevent each
test run from clobbering the others.

[YOCTO #13331]

(From OE-Core rev: 7d0dfd6ada9b2fdf0c14833c388730ffc887af49)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Ross Burton
c965cb80ed wic: change expand behaviour to match docs
The documentation says that --expand takes a comma-separated list of
partition:size pairs, but the code was splitting on hyphens.

Hyphens are not a transitional separator for a list of items, so change the code
to reflect the documentation.

(From OE-Core rev: 8f6d55056a1c6f9fd00b09a8e91b3e888750e793)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Alex Kiernan
fa3daedf7c recipetool: fix unbound variable when fixed SRCREV can't be found
If attempting to find a fixed SRCREV fails because the directory doesn't exit,
avoid failing with:

  Traceback (most recent call last):
    File "/home/vagrant/poky/scripts/recipetool", line 121, in <module>
      ret = main()
    File "/home/vagrant/poky/scripts/recipetool", line 110, in main
      ret = args.func(args)
    File "/home/vagrant/poky/scripts/lib/recipetool/create.py", line 707, in create_recipe
      srcrev = stdout.rstrip()
  UnboundLocalError: local variable 'stdout' referenced before assignment

Fixes: 000480c42797 ("recipetool / devtool: set a fixed SRCREV by default when fetching from git")
(From OE-Core rev: 7dbaaf8a823527a3523187327f5b0885fc682510)

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
sangeeta jain
a005b1897a resulttool/manualexecution: Enable test case configuration option
Current manualexecution required user to exceute all test cases defined inside a "modulename.json" file in oeqa/manual

There are cases when all test cases all not required to run for a module on specific DUT.

Enable manualexecution to have the optional feature where it will use pre-defined json format test case configuration file
where user will be able to select test cases from the "modulename.json" instead of running all of them. This will help
in reducing testing time and reporting unneccesary skip or failures.

Example pre-defined json format test case configuration file (for build-applince):

{
    "testcases" : [
        "build-appliance.build-appliance.Create_core-image-sato-sdk_using_build_appliance",
        "build-appliance.build-appliance.Build_a_image_without_error_(added_recipe)"
    ]
}

(From OE-Core rev: 1d3696bd3b6d05b91c119ade79c40510d1426a56)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Richard Purdie
ee7cffefbf resultool/resultutils: Fix module import error
Fix AttributeError: module 'urllib' has no attribute 'request' when
using remote http urls.

(From OE-Core rev: 170a233452539b0b6b8fc4db50cf69d848d1ac5b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Joshua Watt
02d88de4b5 resulttool: Add log subcommand
Adds a subcommand for dumping various logs from test results

(From OE-Core rev: 7bfb8bef5ec9c4137f439b35c56856697c6b79a4)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Joshua Watt
35fa3e7e18 resulttool: Load results from URL
Adds support for resulttool to load JSON files directly from a http://
or https:// URL

(From OE-Core rev: 3e48404afe27c93fa6ffbd8d66bc52dcd6216005)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Yeoh Ee Peng
47d3b81ad3 resulttool/manualexecution: Refactor and remove duplicate code
Remove duplicate codes. Replace unnecessary class variables with
local variables. Rename variables and arguments with simple and
standard name.

(From OE-Core rev: 3dae8d5a02c29523dc640bee31a362f46ffde6ca)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Yeoh Ee Peng
559176c191 resulttool/manualexecution: Enable creation of configuration option file
Allow the creation of configuration option file based on user inputs.
Where this configuration option file will be used by the the manual
execution to display options for configuration rather than user
need to inputs configuration manually.

(From OE-Core rev: 17fd507e218bfd0f6745737bcd1a170d82e712e5)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:49 +01:00
Chen Qi
a84ff1c062 runqemu: do not check return code of tput
The subprocess.run was replaced by subprocess.check_call because
of compatibility support down to python 3.4. But we really don't
care about whether that command succeeds. Some user reports that
in some tmux environment, this command fails and gives some
unpleasant traceback output. So we use 'call' instead of 'check_call'
to avoid such problem.

(From OE-Core rev: ced3c75fa75d9b9373d695d9204b197b98ea3bd9)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:48 +01:00
Lei Yang
70846334d1 recipetool: add missed module
When I use recipetool to add a new recipe, it says:

[snip]
sys.exit(14)
NameError: name 'sys' is not defined
[snip]

(From OE-Core rev: 4cf8d01d0066699f39162e25d1cdc847c0175473)

Signed-off-by: Lei Yang <Lei.Yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:48 +01:00
Yeoh Ee Peng
3cbf1342dd resulttool/manualexecution: Enable configuration options selection
Current manualexecution required user to input configuration manually
where there were inconsistent inputs and human typo issues.

Enable manualexecution to have the optional feature where it
will use pre-compiled configuration options file where user will
be able to select configuration from the pre-compiled list instead
of manual key-in the configuration. This will eliminate human error.

Expect the pre-compiled configuration options file in json format below

{
    "bsps-hw": {
        "IMAGE_BASENAME": {
            "1": "core-image-sato-sdk"
        },
        "MACHINE": {
            "1": "beaglebone-yocto",
            "2": "edgerouter",
            "3": "mpc8315e-rdb",
            "4": "genericx86",
            "5": "genericx86-64"
        }
    },
    "bsps-qemu": {
        "IMAGE_BASENAME": {
            "1": "core-image-sato-sdk"
        },
        "MACHINE": {
            "1": "qemuarm",
            "2": "qemuarm64",
            "3": "qemumips",
            "4": "qemumips64",
            "5": "qemuppc",
            "6": "qemux86",
            "7": "qemux86-64"
        }
    }
}

(From OE-Core rev: 4c6bac8453768cab9c416c012019ead5edeaafa5)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:48 +01:00
Yeoh Ee Peng
08230d15a8 resulttool: Enable report for single result file
Current validation check function inside resulttool disallow the
report for single result file although the underlying library
was able to handle both directory and file as source input to report.
Removed the validation check as it was no longer needed and to
enable report for single result file.

(From OE-Core rev: dc15a21cbb3090dfb56e78f622486f843be4f54a)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:48 +01:00
Taras Kondratiuk
711064cad9 devtool: prevent starting git gc in background
Devtool creates a git repository for extracted sources in a temporary
directory and then moves it to a final destination after patching is
done. Unfortunately devtool is not aware that some of its git operations
may have caused git garbage collector to start in background. If timing
is just right a repository move fails because GC is removing git objects
while they are being moved.

The issue was hit on Krogoth release, but the code that moves created
git repository is still the same.

Disable gc.autodetach to make GC run synchronously and block git until
it is done.

Traceback (most recent call last):
  File "<snip>/openembedded-core/scripts/devtool", line 342, in <module>
    ret = main()
  File "<snip>/openembedded-core/scripts/devtool", line 329, in main
    ret = args.func(args, config, basepath, workspace)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 352, in extract
    initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, rd)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 644, in _extract_source
    shutil.move(srcsubdir, srctree)
  File "/usr/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/usr/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e', '<snip>/build/grpc/grpc_src/.git/objects/5e', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e'"), ('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57', '<snip>/build/grpc/grpc_src/.git/objects/57', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57'"), many git objects ... ]

(From OE-Core rev: eae5caaf050f54a065e2654a60f55fda2c3afd95)

Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:48 +01:00
Yann CARDAILLAC
bc4556512d correct do_patch for kernel bbappend in sdk
do_patch rule of SDK's workspace/appends/linux-*.bbhappend may fail if script are not written in Python

that was the case with Phytec's BSP, the fix was to replace the do_patch rule with :

do_patch[noexec]="1" when the file was generated in scripts/lib/devtool/standard.py

(From OE-Core rev: e041812a84025a9ff9121efabc3805ff2dfe6375)

Signed-off-by: Yann CARDAILLAC <yann.cardaillac@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-22 00:31:48 +01:00
Yeoh Ee Peng
f2f7099b8a resulttool/manualexecution: Refactor and simplify codebase
Simplify and removed unnecessary codes.
Refactor to allow pythonic loop.

(From OE-Core rev: 31449f3a7649be781b7b61f915d5e879728e87af)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 22:15:46 +01:00
Yeoh Ee Peng
e29a39142e resulttool/manualexecution: Fixed step sorted by integer
Currently the manual execution display step by sorting
the step as string, where steps were not being sorted
correctly when there are more than 9 steps.

Fixed the step sorting by sorting step as integer.

(From OE-Core rev: 2d96a9228fb451752c061c25582a5cc1735099bf)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 22:15:46 +01:00
Yeoh Ee Peng
4a3d52537a resulttool/manualexecution: Enable display full steps without press enter
Current manualexecution required pressing enter button to show each step
information, where this was wasting execution time. Enable display
full steps without needing to any press enter button.

(From OE-Core rev: 6bb181b1e5f14d166cab8023b14316e54012c583)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 22:15:46 +01:00
Yeoh Ee Peng
2147e148d1 resulttool/manualexecution: Standardize input check
Current input checking does not match the standard input practiced
by QA team. Change the input checking to match the standard
input practiced by the QA team.

(From OE-Core rev: f28cfbdad40a69f0cc1a7e71d20260c9621e6194)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 22:15:46 +01:00
Richard Purdie
056ff02e98 resulttool: Allow extraction of ptest data
Rather than simply discarding the ptest data, change the code to discard
it when writing out the new testresult files, or optionally either preserve
it, or write it as seperate discrete logs.

This means the autobuilder should start writing out individual ptest log
files as well as allowing ueers to extract these manually.

(From OE-Core rev: 3c5e5730031ccfb8e851734db23814a4d4fcae4a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 22:15:46 +01:00
Richard Purdie
d80bcba0e3 resulttool: Allow store to work on single files
Store operations using a single file as a source weren't working as the os.walk
command didn't like being given a single file. Fix the store operation to
work for single files.

(From OE-Core rev: 5173954c1ec75629bedbe06d6979dae36eb71b6f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 22:15:46 +01:00
Richard Purdie
9505d6c48e scripts/wic: Be consistent about how we call bitbake
Whilst its tempting to call bitbake_main(), the script also directly executes
bitbake. By doing both this creates all kinds of races. Therefore
change to use subprocess execution in all cases.

(From OE-Core rev: 0507ec4c6069490c9a503e9e2b65f0e7b1962d34)

(From OE-Core rev: c48c37a436b3ebfe4d8ca5d12e97a0cdff0e20a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Mazliana
df4d56b25a resulttool/manualexecution: To output right test case id
We found that manualexecution does not capture test suite values
correctly if there are more than one test suite in test cases.
After verification has made we found out we should retrieved
full test cases value <test_module.test_suite.test_case> from
oeqa/manual/ json file rather than split it them into new
variables test_suite and test_cases.

(From OE-Core rev: c4c63cdc1eca2acebc4ec9486b0d0592e47a8ae2)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Yeoh Ee Peng
d41c0a34e7 resulttool/report: Enable roll-up report for a commit
Enable roll-up all test results belong to a commit
and to provide a roll-up report.

(From OE-Core rev: f2d6019f39a4037cf0914b1cfb3ccb3a03dacfc6)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Yeoh Ee Peng
1050296150 scripts/resulttool: Enable manual result store and regression
To enable store for testresults.json file from manualexecution,
add layers metadata to configuration and add "manual" map to
resultutils.store_map.

To enable regression for manual, add "manual" map to
resultutils.regression_map. Also added compulsory configurations
('MACHINE', 'IMAGE_BASENAME') to manualexecution.

(From OE-Core rev: 3e711cd0482e9df644b011a7d6dadffdfe139f8e)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Yeoh Ee Peng
b6fa6f7e6c resulttool/regression: Ensure regressoin results are sorted
Sorted regression results to provide friendly viewing of report.

(From OE-Core rev: cbe96a6dcb95032c61393bac348141e15a6b6590)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Richard Purdie
dcb9804dff resulttool/store: Fix missing variable causing testresult corruption
(From OE-Core rev: 0d98b9957c611af354efb33f4e915c5166c87ad3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Richard Purdie
38bbbb4ee1 resulttool/report: Ensure ptest results are sorted
(From OE-Core rev: 2e3ffad9e97c5f757e1457721308eb8da63bcabf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Richard Purdie
19a3f1ba47 resulttool/report: Ensure test suites with no results show up on the report
ptest suites with no results don't show up on the reports even though we have
a duration for them. Fix this so the fact they report no tests is visible.

(From OE-Core rev: eb67a3cc4f8f57e85f1f95ced8f2c32ab5090866)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Richard Purdie
4b5686a0fd resulttool/report: Handle missing metadata sections more cleanly
Currently some older results files cause the code to give tracebacks.
Handle these missing sections more cleanly.

(From OE-Core rev: 93b87a9a7056fb507a585c454cef8db7194d7393)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Richard Purdie
3cbd483c42 resulttool/store: Handle results files for multiple revisions
Currently we cant store results if the results files span multiple
different build revisons. Remove this limitation by iterating.

(From OE-Core rev: 63d2c53c01d2aec90b3f0ab298a61a551b23d8fc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Richard Purdie
094b6ea1fc resulttool/resultutils: Avoids tracebacks for missing logs
(From OE-Core rev: faacf35d748067fd17d169c09bc2759606e6e819)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:23 +00:00
Richard Purdie
10b6fbfb8a resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: b4513e75f746a0989b09ee53cb85e489d41e5783)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:22 +00:00
Mazliana
12385bc69b scripts/resulttool: enable manual execution and result creation
Integrated “manualexecution” operation to resulttool scripts.
Manual execution script is a helper script to execute all manual
test cases in baseline command, which consists of user guideline
steps and the expected results. The last step will ask user to
provide their input to execute result. The input options are
passed/failed/blocked/skipped status. The result given will be
written in testresults.json including log error from the user
input and configuration if there is any.The output test result
for json file is created by using OEQA library.

The configuration part is manually key-in by the user. The system
allow user to specify how many configuration they want to add and
they need to define the required configuration name and value pair.
In QA perspective, "configuration" means the test environments and
parameters used during QA setup before testing can be carry out.
Example of configurations: image used for boot up, host machine
distro used, poky configurations, etc.

The purpose of adding the configuration is to standardize the
output test result format between automation and manual execution.

To use these scripts, first source oe environment, then run the
entry point script to look for help.
        $ resulttool

To execute manual test cases, execute the below
        $ resulttool manualexecution <manualjsonfile>

By default testresults.json store in <build_dir>/tmp/log/manual/

[YOCTO #12651]

(From OE-Core rev: 07054cc2691fd2822028a3fd55185af457f79ebf)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:22 +00:00
Yeoh Ee Peng
661fa1335b resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "<testresult_1>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
    ...
    "<testresult_n>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store <source_dir> <git_branch>

To merge multiple testresults.json files, execute the below
    $ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below
    $ resulttool report <source_dir>

To perform regression file analysis, execute the below
    $ resulttool regression-file <base_result_file> <target_result_file>

To perform regression dir analysis, execute the below
    $ resulttool regression-dir <base_result_dir> <target_result_dir>

To perform regression git analysis, execute the below
    $ resulttool regression-git <source_dir> <base_branch> <target_branch>

[YOCTO# 13012]
[YOCTO# 12654]

(From OE-Core rev: 39cda3498ec68b91a671ead256231324d74f7d4c)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:22 +00:00
Robert Yang
5b80b1eabf send-error-report: Add --no-ssl to use http protocol
The script use https protocol by default, but the error-report-web server's
https connection may not work (e.g., doesn't work with python 2.7.6), so add an
option --no-ssl to make it use http.

(From OE-Core rev: a4c89902a24c0961657c8281425d480097aadfa6)

(From OE-Core rev: f9dac6e62623b0ac1a2b2f40b54337721afab7a5)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:48:38 +00:00
Richard Purdie
80adac85bc oe-build-perf-report/gitarchive: Move common useful functions to library
These functions can be reused by the resulttool code so move to the common
function library for this purpose.

(From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385)

(From OE-Core rev: 4b1bd35030c5502873106782a35c4f5a9446e20c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
Richard Purdie
59714c52f0 scripts/oe-git-archive: Separate out functionality to library function
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.

(From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0)

(From OE-Core rev: e47c48768a01191ec36b9732288e05c9f5e80d52)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
Richard Purdie
4c25cd5f7a lib/buildstats: Improve error message
Just stating the recipe counts are different isn't helpful, showing
the differences makes it much easier to understand what changed.

(From OE-Core rev: 1d84b782e3af6f0e6922d72895c905877cc33739)

(From OE-Core rev: 047fd7cf6fc2a3d5d170241f5cc5b61b8d9d8522)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
Robert Yang
a0f99f3bcf runqemu: Make QB_MEM easier to set
It only could be set as the following in the past:
QB_MEM = "-m 256"

Now it also can be set as:
QB_MEM = "-m 256M (or m)"
QB_MEM = "256M (or m)"

[YOCTO #11522]

(From OE-Core rev: ad246f5ce0652bd917d85884176baa746e1379ff)

(From OE-Core rev: ed10ef4f8deaec739ce3713b6ce3b63e134a8c4a)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
Robert Yang
18f000c708 runqemu: Let qemuparams override default settings
Fixed:
In meta/conf/machine/include/qemuboot-x86.inc:
QB_CPU_x86-64 = "-cpu core2duo"

$ runqemu qemux86-64 qemuparams="-cpu coreduo"
Check /proc/cpuinfo, it should use coreduo rather than core2duo since user
specifies it, but it doesn't, append qemuparams to the last can fix the
problem.

[YOCTO #11773]

(From OE-Core rev: a847dd7202a2c493788c45d11eb86866264af7a4)

(From OE-Core rev: 81ab6332bee0426201626cc8a0339ce3e6b81d6f)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
Richard Purdie
11ec3b28b4 wic/engine: Fix missing parted autobuilder failures
OE-Core rev: a88bcbae850a2e6d182291d3f8e167aabdbe4842 broke the ability
to find parted as it may be in sbin which is not in PATH for some users
on some distros.

Iterate on the original patch to fix this and also fix the original problem.

(From OE-Core rev: af3803e5189d7814f9dbd238fb6dab200f351e1a)

(From OE-Core rev: 6e2bf8691ffd6004cb40e71a2d1d002b5ed5808f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
William Bourque
b55c0d0a17 wic/engine.py: Load paths from PATH environment variable
Load self.paths from environment variable and
if it fails, fall back to hardcoded list.
This is required for users that would need to load
different e2fsprogs binaries if their system's ones
are not recent enought

(From OE-Core rev: a88bcbae850a2e6d182291d3f8e167aabdbe4842)

(From OE-Core rev: 319ee1bb146b91f78d7a141def2b1cf92cf88887)

Signed-off-by: William Bourque <wbourque@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
Leonardo Augusto
11e09611df scripts/lib/wic/engine: Fix cp's target path for ext* filesystems
Python subprocess' shell=True defaults to /bin/sh[1][2], which
often refers to a POSIX-compliant shell. As the -e flag is
not defined in the POSIX standard[3], some shells may
interpret "-e" as the first argument to the echo command,
causing the rest of the command line to fail.

In this particular case, "echo -e 'cd {}'" is interpreted
as "-e cd {}", which causes the first line of the command
to fail, and causing cp to always place the source file
in the filesystem's root.

Replacing "echo -e" for a printf command makes this
more portable.

This issue only affects "wic cp" for ext* filesystems.

[1] https://docs.python.org/2/library/subprocess.html
[2] https://docs.python.org/3/library/subprocess.html
[3] http://pubs.opengroup.org/onlinepubs/9699919799

(From OE-Core rev: 9a5b703d4e60c3ccdf80b5289b8e6fc91133fcde)

(From OE-Core rev: bf51d6da725a3d5850c3cbd70fb91b91c08e507f)

Signed-off-by: Leonardo Augusto <leobsd@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:40 +00:00
Robert Yang
422710e0db checklayer: Avoid adding the layer if it is already present
* Rename add_layer() to add_layers() so that add_layer_dependencies() can
  re-use it.

* Avoid adding the layer if it is already present

[YOCTO #13148]

(From OE-Core rev: b9cc18d83f55ff48c3d6e60c56359f6736d5a06a)

(From OE-Core rev: 64310d9d18bb1751595c197e6955774ad5514844)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:39 +00:00
Changqing Li
35015f1751 checklayer: generate locked-sigs.inc under builddir
yocto-check-layer will find locked-sigs.inc under builddir,
but locked-sigs.inc is generated under current bitbake working
dir. if run yocto-check-layer outside builddir, we will met error
like "No such file or directory: *locked-sigs.inc". change to
run bitbake -S under builddir to fix this problem.

[YOCTO #12973]

(From OE-Core rev: 2ec0bc0b038bc6413978c8f34ef5c0d22b4bc3e7)

(From OE-Core rev: f236d8782958df953ca1e2a6f09779cbc8f2d600)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:39 +00:00
Robert Yang
3fd921d551 yocto-check-layer-wrapper: Fix path for oe-init-build-env
We only could run it in top of oe-core dir since it assumed oe-init-build-env
was in cwd, this patch fixes the problem.

[YOCTO #13148]

(From OE-Core rev: 47d7b5fcad127171243cc95c586c2c7f35cfac3b)

(From OE-Core rev: 09e4b9bead32f143f4a685794c4709fe41bf014d)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:39 +00:00