Commit Graph

5798 Commits

Author SHA1 Message Date
Armin Kuster
6d2e12e792 bitbake: tests/fetch: Resolve fetch error in bitbake-selftest
FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring
      self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
      AssertionError: '' is not true : Could not find upstream version for db

[YOCTO #13496]

The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget.

Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the
recipe. Also change the version being checked.

(Bitbake rev: 164c0df429fc9be771c54c241b17d7afb8849afc)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-09 09:58:02 +01:00
Martin Jansa
78ac8af785 bitbake: fetch2: show warning when renaming the archive with bad checksum failed
* noticed on read-only sshfs premirror
* it was showing the warning about renaming the file:
  WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441

  and then failed because of movefile() issue with python3 (fixed in previous commit):
  ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:

  with movefile() fixed, it let do_fetch continue and re-fetch locally with the right
  checksum, but still the renamed file didn't exist, because of movefile failure - add
  another warning when the movefile fails - for whatever reason - unfortunately movefile
  prints error messages with just print() so the real error is hidden only in log.do_fetch
  in this case:
  movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441'

(Bitbake rev: 24e44c412ac36987c1f65ced50030064ec58100d)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 11:58:50 +01:00
Martin Jansa
6cd3eee5fa bitbake: utils: Fix movefile() exception handling with python3
* with python3 this fails with:
  File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile
       0795:        try:
       0796:            os.rename(src, destpath)
       0797:            renamefailed = 0
       0798:        except Exception as e:
   *** 0799:            if e[0] != errno.EXDEV:
       0800:                # Some random error.
       0801:                print("movefile: Failed to move", src, "to", dest, e)
       0802:                return None
       0803:            # Invalid cross-device-link 'bind' mounted or actually Cross-Device
  Exception: TypeError: 'OSError' object is not subscriptable

(Bitbake rev: 16415c6e0f9e3d5c9fd81c9aabaea11d61b14187)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 11:58:50 +01:00
Richard Purdie
048c1467e2 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 34ed28a412af642a993642c14bd8b95d5ef22cd8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 18:04:01 +01:00
Richard Purdie
e1db6a1690 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ac556588fac55e91b7ce4839a975eb9ebb5aa192)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 18:04:01 +01:00
Matthias Schiffer
b164f0dc3e bitbake: fetch2: runfetchcmd(): unset _PYTHON_SYSCONFIGDATA_NAME
Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME;
unfortunately, this also affects Python scripts run as fetch commands like
git-make-shallow, breaking it with a message like

    Failed to import the site module
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site.py", line 570, in <module>
        main()
      File "/usr/lib/python3.6/site.py", line 556, in main
        known_paths = addusersitepackages(known_paths)
      File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages
        user_site = getusersitepackages()
      File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages
        user_base = getuserbase() # this will also set USER_BASE
      File "/usr/lib/python3.6/site.py", line 254, in getuserbase
        USER_BASE = get_config_var('userbase')
      File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var
        return get_config_vars().get(name)
      File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars
        _init_posix(_CONFIG_VARS)
      File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix
        _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
    ModuleNotFoundError: No module named '_sysconfigdata'

on an Ubuntu 18.04 system (and likely others) when building with
BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS.

Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this.

(Bitbake rev: 6783104f18a8a14ddd88932720cd80dffeada63d)

Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18 17:00:37 +01:00
Mark Hatle
12a8a51e96 bitbake: gitsm: Fix a bug where the wrong path was used for the submodule init
Because we are trying to avoid network activity and use our own fetcher,
the system emulates the behavior of 'git submodule init'.

git submodule init uses the .gitmodules file, where typically the module
name and path are the same.  However, in this case the module name and
path (in the tree) were different. i.e.:

[submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"]
        path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared
        url = https://github.com/Azure/azure-c-shared-utility.git

Previously the code assumed the 'path' was both the checkout location
under .git/modules, as well as the path to extract the components.  This
proved to be incorrect as the .git/modules path needs to match the submodule
'name'.  This causes the components that were fetched to be initialized in
the wrong location, which later caused the 'git submodule update' process to
skip not properly initialized modules.

A test case was added for this specific case to ensure a regression does
not appear in the future.

(Bitbake rev: 2a823584a5dbc2427060352f3b4798dbb2640cfd)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18 11:43:22 +01:00
Joshua Watt
d6462e0149 bitbake: bitbake: cooker: Rename __depends in all multiconfigs
The renaming of the __depends variable to __base_depends and file
watches needs to occurs for all multiconfigs, not just the base config.
Failing to do this for all multiconfigs will result in a huge increase
in the size of the parsing cache (about 5x for a single mulitconfig)
because all multiconfig caches will still depend on the base config
files. This will also seen a similar jump in the amount of time required
to load the parsing cache from memory, both because the cache is larger
and because of explosion of additional existence checks that must be
done for the base files.

[YOCTO #13359]

(Bitbake rev: 2ea1e6720c66841baa5e01c523c86f8ecb03eb7d)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:28:26 +01:00
Richard Purdie
38a6b64f1c bitbake: cooker: Ensure mcdeps are processed even if only one multiconfig
If you have no BBMULTICONFIG set but set mcdepends, they're currently
ignored. We can handle them correctly with this small tweak.

(Bitbake rev: 2aaea2f72e0c37451e7c7851b3a32abd733b38e1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:28:26 +01:00
David Reyna
254bef1d42 bitbake: toaster: Fix Thud Bitbake release metadata
Fix the Bitbake version ID for Thud (1.40).

[YOCTO #13356]

(Bitbake rev: 7cda98d64693d978ad1387fed05a83aaa681e097)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 09:44:36 +01:00
Naveen Saini
6e3a4d7926 bitbake: bitbake: fetch2/git: git-lfs check
Build will fail if repository has lfs contents in absense of git-lfs tool on host.
Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

[YOCTO #13198]

(Bitbake rev: 7107c1953b4c949b81491295e03736beb085b055)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-15 21:52:57 +01:00
David Reyna
eab2d05025 bitbake: toaster: update to Warrior
Update the fixture files to add Warrior and remove Sumo.

[YOCTO #13287]

(Bitbake rev: 014287facfcf4598b6fd24487f44109bfb4cff57)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-15 21:52:57 +01:00
Scott Rifenbark
1b425a8450 bitbake: bitbake-user-manual: Added section on modifying variables
Fixes [YOCTO #12548]

I created a new section titled "Modifying Variable Values" that
provides instruction on how to use the "bitbake -e" command to
be sure your configuration and variable values are as expected.

(Bitbake rev: cd8d4ec8ec6650b756dafe750436528f736b3938)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-23 16:33:57 +01:00
Scott Rifenbark
80baa2d1fb bitbake: poky.ent: Removed "ECLIPSE" entity variables.
These have never been used in the BB manual and were an
artifact from copy-and-pasting the poky.ent file from
the main yocto-docs repo to this bb repo when the BB
manual was origially created.  They are useless.  I have
removed them.

(Bitbake rev: cb185efe9e88cfb12b7a3fd08f3086ca0b69c8e2)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 22:44:16 +01:00
Richard Purdie
0c3423aae9 bitbake: bitbake: Update version to 1.42.0
(Bitbake rev: 00b133af009f9e7c1a4c751b6ef4902f47a81fe2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 13:48:04 +01:00
Andre Rosa
9be9876760 bitbake: utils: Let mkdirhier fail if existing path is not a folder
Let mkdirhier fail if existing path is not a folder instead of assuming a
directory hierarchy already exists.

(Bitbake rev: a8d9b82ccf93dcb74258693f62d88be380b1c0b7)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 13:48:04 +01:00
Andre Rosa
27cbc8c5f0 bitbake: utils: Make prune_suffix prune a suffix
Make prune_suffix prune a suffix instead of replacing a substring that could
happen more than once and not only when it ends with it.

(Bitbake rev: 57e765e38c6382a9b36d5ee2a6f3fa96ac905b82)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:45:15 +01:00
Mark Hatle
2030e815bb bitbake: gitsm: Add need_update method to determine when we are going to a new SRCREV
If the system had previously fetched a source repository for use by gitsm,
and then the SRCREV was updated and the new commit already existed, the system
would not re-evaluate the submodules and update them accordingly.

The cause of this issue was that need_update was being used, unmodified, from
the base git fetcher.  It did not have any knowledge, nor did it care if we
were moving commits and needed to re-evaluate what was happening due to this
switch.

To fix the issue, during the download process we add all processed (by
gitsm) srcrevs to the git config file, as bitbake.srcrev.  This allows us to
use a new need_update function that not only checks if the git commit is
present, but if we have previously processed this commit to ensure all of the
submodule components are also present.

This approach is used, instead of iterating over the submodules in need_update
to avoid a potential race condition that has affected us in the past.  The
need_update is called only with the parent locking.  Any time we need to dive
into the submodules, we need to lock, and unlock them, at each stage.  This
opens the possibility of errors in either the code, or unintended race
conditions with rm_work.

This issue was discovered by William A. Kennington III <wak@google.com>.  The
included test case was also written by him, and included unmodified.

(Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-01 14:48:44 +01:00
Robert Yang
b1641008f0 bitbake: fetch2: Use bb.utils.to_boolean() for BB_NO_NETWORK
Make it consistent with BB_FETCH_PREMIRRORONLY.

(Bitbake rev: 4c6013f47ef36b03f590c909d7c9a2f50b698620)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
64c21baf63 bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLY
The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but
fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous,
and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case,
so unify it to the previous one. (As BB_NO_NETWORK does).

And also use bb.utils.to_boolean() to make them consistent.

(Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
acc242c7ec bitbake: fetch2: Print SCMs list when SRCREV_FORMAT is not set
This makes it easier to debug, especially when multipe SCMs like gitsm,
otherwise we don't know why there are multiple SCMs.

(Bitbake rev: 313fe5e86b254eadfdead706be4bd7b274d5e3c0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
db91c23871 bitbake: fetch2/git: Fix clean to remove clonedir
The localpath is a symlink to clonedir when it is cloned from a mirror, for
example:
$ bitbake systemtap-native -cfetch
$ ls downloads/git2
sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git
mirror.path.git.sourceware.org.git.systemtap.git

There are both sourceware.org.git.systemtap.git and
mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink
sourceware.org.git.systemtap.git is created by try_mirror_url(), but
do_cleanall" only removed the symlink, didn't remove the real dir
mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions,
for example, I assumed that do_cleanall removed everything, but it didn't, and
it would the re-used next time when do_fetch. This patch fixes the problem.

(Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
4b2b3339df bitbake: cooker: Fix bbfile_config_priorities when BBFILE_PATTERN is empty
The layer was not in bbfile_config_priorities when BBFILE_PATTERN is empty,
this caused "bitbake-layers show-layers" can't show these layers, this was
incorrect since these layer did exist. Add these layer to
bbfile_config_priorities can fix the problem.

Fixed:
Add BBFILE_PATTERN_core = "" in oe-core/meta/conf/layer.conf
$ bitbake show-layers | grep oe-core

There was nothing, now the layer is shown

(Bitbake rev: 0ff5cdb0cca9266ca29127639494bcfd95e36831)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
a7f3cbb4a2 bitbake: fetch2: runfetchcmd(): Print workdir in debug message
The workdir is an important message when run git command, for exmample, before the patch:
Running git -c core.fsyncobjectfiles=0 branch --contains [snip]

We don't know where it is running, now it is:
Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git

Which is easier to debug.

(Bitbake rev: f25c05b73218d0e8edf2dd63b36bc09e547c5455)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Richard Purdie
c96d8de352 bitbake: fetch2: Fix undefined variable issues
Various refactors have left hanging variables, often in debug messages, hence
why they haven't been spotted. Fix them (pylint spotted the issues).

(Bitbake rev: e1f252408982a62485bbf874a01fd30a02aeed32)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
28fb15c56f bitbake: fetch2/wget: Remove pointless lambda function
(Bitbake rev: 329986c3f2d98c4cd1a43b725194003575e718d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
588a4c58e8 bitbake: fetch2/wget: Clean up whitespace/comments
Clean up the whitespace and comments style, keep pylint happy.

(Bitbake rev: ee59fdaae68543ade03cacfdbbf14fdc7e469412)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
9a1faba274 bitbake: fetch2/wget: Fix typo in proxy handling
Fix a typo in a variable which would affect proxy handling spotted by
pylint.

(Bitbake rev: 19f4ca0119e33df64da629253d5002fcdb385f4f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
202b7d0521 bitbake: fetch2/wget: Cleanup module imports
(Bitbake rev: ef062a3251af474fdeb480149856be3887b8c1c9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Stefan Klug
0a9161855c bitbake: fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
I wonder how this used to work for anybody.

(Bitbake rev: 9481e9cd688f635a7fe2ac60d5826ac26f0cb9ba)

Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 15:29:31 +00:00
Richard Purdie
c5882f9212 bitbake: .gitattributes: Add to improve git diff for minified css/js files
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-07 12:18:48 +00:00
Alexander Kanavin
984a4def83 bitbake: bitbake: fix version comparison when one of the versions ends in .
Previously, this would happen:

======================================================================
ERROR: test_vercmpstring (bb.tests.utils.VerCmpString)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alexander/development/poky/bitbake/lib/bb/tests/utils.py", line 45, in test_vercmpstring
    result = bb.utils.vercmp_string('1.', '1.1')
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 143, in vercmp_string
    return vercmp(ta, tb)
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 135, in vercmp
    r = vercmp_part(va, vb)
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 124, in vercmp_part
    elif ca < cb:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

----------------------------------------------------------------------

(Bitbake rev: fef56d28c3efec4876c379898cbc4d4c65303aee)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 16:35:33 +00:00
Richard Purdie
4b02d1fa21 bitbake: cooker: Tweak multiconfig dependency resolution
There were a couple of problems with the multiconfig dependency resolution:

- the "if mc" condition triggering this code wasn't correct, it needs
  to be "if more than one multiconfig" configured
- after adding providers we need to call add_unresolved again
  and rebuild mcdeps within the "while new" loop

By fixing these issues we allow various other combinations of multiconfig
builds to work which previously didn't.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: 4359b037de578095db2595f119dfb8e3340e1414)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
3bf0602713 bitbake: siggen: Fix multiconfig corner case
There was already a fix to ignore some multiconfig dependencies but its
'opposite' case wasn't covered. Cover that combination to so as to avoid
tracebacks in multiconfig builds.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: c883dfe378af9dfc192a8e392e84325d68648806)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
0b19e52b85 bitbake: runqueue: Filter out multiconfig dependencies from BB_TASKDEPDATA
The consumers of BB_TASKDEPDATA in OE metadata can't cope with multiconfig
dependencies. The choice is either to start adding code to each of them to
filter out multiconfig dependencies, or do this at source.

After consideration we've decided to do this at source as doing otherwise
is code duplication and error prone and in any case we've looked at, they
don't make sense.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: 531dcd221a10853f45cc057b52bb2d5083e0ee42)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
8dac3ce424 bitbake: runqueue: Fix dependency loop analysis 'hangs'
Currently the mechanism for breaking out of the dependnecy loop analysis
code is broken and doesn't work leading to bitbake appearing to hang.

Add in a custom exception for this purpose and fix the code to exit
as intended, fixing the hang and making the dependency loop code
usable again.

(Bitbake rev: 8756e4ade67c16e35269ea0659e10b9ebaa6117f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17 10:20:21 +00:00
Joshua Watt
9025bd11ec bitbake: bitbake: Allow arguments in FAKEROOTCMD
Changes FAKEROOTCMD so that it can accept additional arguments to pass
to the fakeroot implementation instead of being treated as a simple
command

(Bitbake rev: 4fa51afb56b090cf1f746842acd602c9536715d5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15 08:18:19 +00:00
Robert Yang
5fa3b5b152 bitbake: bitbake: cookerdata: Check duplicated BBFILE_COLLECTIONS
It shouldn't work when there are duplicated BBFILE_COLLECTIONS.

(Bitbake rev: a1f251e5ab859d6d4a2cb908408d4ddcab5a5de1)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-14 11:21:58 +00:00
Alexander Kanavin
2f17b87f97 bitbake: build.py: add unhandled exception information to the logger when executing python tasks
Previously this information was simply discarded, which in some cases
resulted in generic, unhelpful failures.

With this change the user would see what the exception is, but
without ugly tracebacks or bulky, irrelevant information from the log file.

(Bitbake rev: 8da9242702fbef8cf3156b95a1076802e0f653c7)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-14 11:21:58 +00:00
André Draszik
2fcbc079e2 bitbake: bitbake: remove True option to getVar calls
getVar() has been defaulting to expanding by default for
a long time (2016), thus remove the True option from
getVar() calls with a regex search and replace.

Search & replace made using the following command:
    sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
        -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
             | cut -d':' -f1 \
             | sort -u)

(Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 22:01:10 +00:00
Scott Rifenbark
82b92b03f5 bitbake: bitbake-user-manual: Fixed section head typo
Unseting -> Unsetting

(Bitbake rev: 8b18ebb96e9f66649125f7cd1c8c8ffd1d691398)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 21:44:04 +00:00
Richard Purdie
65a7662c5b bitbake: server/process: Add missing exception raise
The intent of the code was to catch one kind of error, it was actually swallowing
all exceptions and looping indefinitely. Fix it to work as intended.

This explains some mystery hangs we've been seeing.

(Bitbake rev: d73dbc3580faa1225d95ae4cefac4879ca3c1b2f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:24:50 +00:00
Robert Yang
2c15d8000d bitbake: bitbake: ConfHandler: Don't strip leading spaces
Fixed:
- Add the following lines to conf/local.conf:
  FOO = "BAR1"
  FOO_append = "\
      BAR2"

  $ bitbake -e | grep '^FOO'
  FOO="BAR1BAR2"

  The leading spaces in the second line have been removed.

- But if add the previous two lines to base.bbclass:
  $ bitbake -e | grep '^FOO'
  FOO="BAR1    BAR2"

  The leading spaces in the second line are preserved, this is inconsistent, now
  fix ConfHandler to preserve leading spaces.

[YOCTO #12380]

(Bitbake rev: 8c3bc15a7b5e0a81d7b6c9d3fe43fbff63207156)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:24:50 +00:00
David Reyna
07735d117d bitbake: toaster: correctly register the thud branch
There are typos in the fixture files that need to be fixed to correctly
check out the 'thud' branch.

[YOCTO #13064]

(Bitbake rev: 649a304b4d0a35553f665f0a748a7f21fcab51ba)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:24:50 +00:00
Mark Hatle
a7774aced0 bitbake: gitsm: The fetcher did not process some recursive submodules properly.
Move the submodule processing outside of the if statement to avoid any
optimizations that may happen.

Update the test cases to include the additional case, and split the other
test cases into individual tests to make it easier to figure out what
the failure may be.

(Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28 23:14:27 +00:00
Mark Hatle
1979d9162a bitbake: gitsmy.py: Fix unpack of submodules of submodules
If the submodule is in a subdirectory, it needs to have that structure
preserved.  This means the unpack path needs to be in the 'dirname' of the
final path -- since the unpack directory name is specified in the URI.

Additional specific test cases were added to ensure this is working properly
based on two recent error reports.

(Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Mark Hatle
9238a72589 bitbake: gitsm.py: Fix relative URLs
Prior code happened to work with relative URLs, when the code was recently
restructured it caused all relative urls to no longer work.  Restore the prior
code flow for relative support and better comment why that code is there.

(Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Christopher Larson
6fd870e6a1 bitbake: bb.tests.codeparser: add parameter expansion modifiers test
We don't want references including shell parameter expansion modifiers
(i.e. `:-`, `#`, `%%`, etc) to be added to our vardeps, so add a test to
ensure this.

YOCTO #12987

(Bitbake rev: be022085fe1ea1b9a9d519f0455883e2da363d2c)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-21 23:44:34 +00:00
Christopher Larson
7178c69ea2 bitbake: bb.data_smart: only try to expand refs to valid variable names
This aligns the behavior of expansion with the recipe parser, only
attempting to expand references to valid variable names. This avoids
adding references for things like `${foo#${TOPDIR}}` to our vardeps
without imposing much additional processing overhead beyond the change
to the expansion regexp.

YOCTO #12987

(Bitbake rev: df2ac65370aa86cdbc1574fdede25e3519410e45)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-21 23:44:34 +00:00
Mark Hatle
7d715ae071 bitbake: gitsm.py: Refactor the functions and simplify the class
The update_submodules and unpack_submodules functions were nearly indentical,
so we made a common function where the different behavior could be passed
in by the download and unpack users.  The new function is process_submodules.

Moved the parse_gitmodules function under the new process_submodules, since
there are no external callers.

Refactor the file relative path processing to the URL translation code.
We also add a warning to the translation if a relative ssh URL has been
detected.  Since this can cause a problem.

In the case of a relative URL that does not work after being translated,
it should be possible to use the MIRROR functions to manual translate the
generated relative URL into one that works properly.

Remove 'git config' processing on download contents.  It turns out this is not
necessary since all of the later components work using the git fetcher.

Limit the 'git submodule update' call to only when unpacking a non-bare
repository.  Submodules are always loaded as bare, so this prevents
intermediate unpacks from being attempted.

Finally, the test cases were updated and the new commit ids in the test
repository were updates as well.

(Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00