Commit Graph

5361 Commits

Author SHA1 Message Date
Richard Purdie
9186078cf6 bitbake: bitbake: Update to version 1.44.0
(Bitbake rev: 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-09 14:10:48 +01:00
David Reyna
704c3e623e bitbake: toaster: Enable Zeus branch in place of Thud
Toaster directly supports the last two stable branches of Yocto
Project. With "Zeus" being released, it is time to replace "Thud".

[YOCTO #13579]

(Bitbake rev: 29374386fd7fcfac9d4070584dff76327845595e)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-08 20:49:36 +01:00
Richard Purdie
03d4d9d68f bitbake: bitbake: Bump verison 1.43.1 -> 1.43.2
This allows metadata to depend on SignatureGeneratorUniHashMixIn which was recently added.

(Bitbake rev: f0f814407fdd2fffa7071c36c011b489bfcd53da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 20:22:25 +01:00
David Reyna
97a5762be1 bitbake: toaster: improve warnings when adding dependency to packages
Some of the objects that bitbake reports to Toaster as dependencies to packages
are known objects that are not packages, for example library files and kernel
modules. In the Toaster logs, mark these as "Info" instead of "Warning".

[YOCTO #13386]

(Bitbake rev: 0d66f644d647900e8f5afa526a6d9cee687c41cc)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 14:42:13 +01:00
David Reyna
1ca006c7b6 bitbake: toaster: issues in import layer when clicking 'add layer'
There were three issues in this one bug.
  1) The Add Layer button allows empty layers
  2) The internal XHR URL was wrong, which caused a hidden AJAX error
     and did not correctly complete the action nor disable the button
     after an add.
  3) There was a race condition between typing in the dependent layer
     select text box (which would normally disable the add button), and
     the typeahead pull-down selection (which would normally enable the
     add button). This forced the user to select the typedahead layer twice.

[YOCTO #13385]

(Bitbake rev: c4ccf3a792ae7e8549b879ba77ff7f7edb0e665a)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 14:42:13 +01:00
Richard Purdie
562de41ce7 bitbake: tests/runqueue: Fix hashserve shutdown race
The hashserve can delete its socket whilst the cleanup us happening leading to
backtraces and test failures.

Add code to avoid this race condition.

[YOCTO #13542]

(Bitbake rev: efd7b025cee25d0ee668c09476395d08fcf5ae1a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 14:18:59 +01:00
Richard Purdie
a73cbe649a bitbake: siggen: Remove full path from unitaskhashes keys
The full paths make the cache useless in the sdk. They also bloat the
cache size. They're for human debugging benefit only so compromise and
reduce this to the filename.

(Bitbake rev: 3b275c4083eae1d3781f0862919af9de83932b0f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
9254d537aa bitbake: runqueue: Small performance optimisation
A minor performance optmisation to keep lists smaller when running large
builds. We can do this since once a task has been built, we don't need
to worry about it. This improves a major bottleneck that shows up on
performance profile charts in dryruns.

(Bitbake rev: cd6b89230823707c3c9bb9e6883bf5a971916581)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
2f8cd1d021 bitbake: runqueue: Save unihashes more frequently
There are some runqueue code paths where the unihash cache would not be
saved where for example only parsing or an occurred. Save the cache at the
end of runqueue generation to ensure entries are cached.

(Bitbake rev: 9eee0d36870c11dd303894a6151c33a83bd3a1bc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
ba0ff38cab bitbake: siggen: Avoid writing misleading sigdata files
Use the unihash in the output filename of sigdata files else the contents
of stamp directories is misleading. Write the unihash into the singature to
make it clear what happened.

(Bitbake rev: feb01ee54d3706fe93768f332054c7532f7209e4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
7ab4808e0a bitbake: siggen/runqueue: Fix signature mismatch issues
We need to set the setscene tasklist before we call into the
taskhash/unihash code else the behaviour is inconsistent.

Avoid reporting hashes for non setscene tasks since we'd never
query that.

(Bitbake rev: 419a7840b8627278db694029c25df00214d01d96)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
ff872fdda5 bitbake: runqueue: Change task migration behaviour for rerunning setscene tasks
Currently runqueue will rerun setscene tasks multiple times as hashes
change. This has caused numerous problems since a setscene task may
become "unavailable" for some future signature combination and the code
then can't easily "unskip" tasks its already passed into the execution
queue.

At least for now, only run setscene once and assume they're equivalent
at that point. In practise that has been much more stable in testing.

Tweak the test to match the change in behaviour.

(Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
155249b1db bitbake: siggen: Ensure setscenetasks list is available to worker context
The setscenetasks list needs to be available in the worker contexts
else the signature behaviour there mismatches what the server does.

Add the data to get/set_taskdata to ensure this happens.

(Bitbake rev: 632980ef90fe126b7ba3d138f4d574ae05914779)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
d0b7471ba8 bitbake: runqueue: Fix task migration problems
Tasks were not migrating consistently, particularly:

* if a task was rehashed which had already run
* if a task which was valid became invalid due to a rehash

We need to always run the migration code for rehashed tasks and then
reprocess them for hash validity. This means rearranging the code.

It also means several tests are no longer correct and can't be written
correctly to work on all possible workflows so those are removed.

(Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Joshua Watt
2ac500a609 bitbake: hashserv: Don't daemonize server process
The hash server process is terminated and waited on with join(), so it
should not be a daemon. Daemonizing it cause races with the server
cleanup, especially in the selftest because the process may not have
terminated and cleanup up its socket before the test cleanup runs and
tries to do it.

[YOCTO #13542]

(Bitbake rev: 7c829675581818f92d57056b57fbd3880829b6bd)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Joshua Watt
1805574559 bitbake: siggen: Fix attribute error when hashserver fails
The HashConnectionError class was moved to the client module and needs
to be updated.

[YOCTO #13537]

(Bitbake rev: 9fb862685e5e5a2aa534bc25cab1e4158d708b40)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Joshua Watt
6f6b41e642 bitbake: hashserve: Add missing import
The os module is required to connect to a unix domain socket

(Bitbake rev: 31a5111bcd0080a583d0d95fad3e09ae78bdf0fa)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Joshua Watt
f971d6ae2e bitbake: cookerdata: Add mc conffiles hashes to cache hash
The variable values that result from parsing multiconfig should be
included in the cooker data hash, otherwise changes to these files won't
be detected, which will allow the parsing cache to be loaded with the
old values for the multiconfigs. This can either manifest as the
variable values simply not updating, or getting basehash changed errors
when building.

This bug was previously undetected because all of the multiconfig base
files were a direct file dependency in all parsed recipes. This was
fixed in 34137a00f60 ("bitbake: bitbake: cooker: Rename __depends in all
multiconfigs"), exposing this bug.

[YOCTO #13541]

(Bitbake rev: c74481aa15226e1bff9d53e4ee4b702ebfa1ad32)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Jacob Kroon
daa6dcfc39 bitbake: tests/data: Test combinations of _append together with override
(Bitbake rev: f31f35e8527c60a95931a4a8311a4cd237770b42)

Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Ross Burton
a44d119456 bitbake: tests/fetch: add test case for git-lfs handling
Add a test case to exercise the detection of git-lfs repositories and the
behaviour of the lfs parameter.

(Bitbake rev: a7cf4fc72cce357c425084dc2c5f35b5ed1a4b7b)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Ross Burton
560358e922 bitbake: fetch2/git: refactor check for git-lfs command
Refactor the git-lfs checking: this means both clearer code in the download()
function and allows unit testing to monkeypatch the functionality.

(Bitbake rev: 33cf9172ded50a869f7201ba463ab9ecc69b8252)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Richard Purdie
886a71438f bitbake: utils: Add ionice option to prunedir
Autobuilder type infrastructure can benefit from deletion of certain files as
background IO due to the way Linux filesystem priority works.

We have problems where build directories as part of oe-selftest being
delete starves the running tasks of IO to the point builds take much
longer to compelte.

Having this option of running the deletion at "idle" helps a lot with
that.

(Bitbake rev: 797354d285f6d624d9adb52bab65823572da0e39)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Robert Yang
2af45501fd bitbake: runqueue: validate_hashes(): currentcount should be a number
According to sstate_checkhashes which is defined in sstate.bbclass, the
currentcoun should be a number (0, not None).

Fixed:
$ bitbake base-files -Sprintdiff
    >    bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete))

TypeError: %d format: a number is required, not NoneType

(Bitbake rev: 45cb73e2846eaffe8964a573875f54808e8f3633)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Ross Burton
e35be7aa7d bitbake: fetch2/git: add git-lfs toggle option
Add a new 'lfs' option to the git fetcher so that we can optionally not fetch
git-lfs content, for repositories that contain LFS data that we don't
actually need for building.

By default lfs is set to 1, so if the repository has LFS content then git-lfs is
required.  Setting lfs to 0 will mean that git-lfs won't be required to fetch,
and some files will be missing.

(Bitbake rev: be0b78ccfc5ede98041bc0545a15092494b12b26)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Yeoh Ee Peng
0a46b9dd9c bitbake: bitbake-layers: show-recipes: Enable bare output
Currently, show-recipes will append "(skipped)" marker to recipes which
were skipped due these recipes does not satisfied the configurations.

Example: $ bitbake-layers show-recipes -r
   ace
   backport-iwlwifi
   core-image-rt (skipped)
   core-image-rt-sdk (skipped)
   core-image-tiny

Add -b/--bare to enable output names without "(skipped)" marker.

Example: $ bitbake-layers show-recipes -r -b
   ace
   backport-iwlwifi
   core-image-rt
   core-image-rt-sdk
   core-image-tiny

(Bitbake rev: 87796e580cd160a535eb5fb9e31846a7cf1a249e)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Yeoh Ee Peng
7f2803d1aa bitbake: bitbake-layers: show-recipes: Select recipes from selected layer
Currently, show-recipes will show recipes from all configured layers.
Assume, meta-intel layer was added to conf/bblayers.conf.

Example of default $ bitbake-layers show-recipes:
core-image-rt:
  meta-intel           unknown (skipped)
  meta                 unknown (skipped)

Add -l/--layer to enable showing recipes from user selected layer.

Example: $ bitbake-layers show-recipes -l meta-intel
core-image-rt:
  meta-intel           unknown (skipped)

(Bitbake rev: 8c38d95c4474ea171cb55b0e336d9090451e89ce)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Yeoh Ee Peng
5808f455f0 bitbake: bitbake-layers: show-recipes: Show recipes only
Currently, show-recipes will show all recipes available (both
recipes with different version and recipes provided by more
than one layer).

Example of default $ bitbake-layers show-recipes:
core-image-rt:
  meta-intel           unknown (skipped)
  meta                 unknown (skipped)

yajl:
  meta-oe              2.1.0
  meta-oe              1.0.12

Add -r/--recipes-only to enable showing recipes only. This
provide a focus view on unique recipes available.

Example of $ bitbake-layers show-recipes -r:
core-image-rt (skipped)
yajl

(Bitbake rev: 048bd051a9b422a38c181f57bb5090a05684a5c3)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Mikko Rapeli
330bd591ec bitbake: svn fetcher: allow "svn propget svn:externals" to fail
Not all servers and repositories have this property set
which results in failures like this when actual svn checkout
command succeeded:

svn: warning: W200017: Property 'svn:externals' not found on ''
svn: E200000: A problem occurred; see other errors for details

(Bitbake rev: 238636f033cbf18e5741f0ea0e64db40e84f5838)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 17:52:01 +01:00
Joshua Watt
20f032338f bitbake: bitbake: Rework hash equivalence
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:

1) The server protocol is no longer HTTP based. Instead, it uses a
   simpler JSON over a streaming protocol link. This protocol has much
   lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
   domain socket. Unix domain sockets are more efficient for local
   communication, and so are preferred if the user enables hash
   equivalence only for the local build. The arguments to the
   'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
   equivalence server is changed to "auto" instead of "localhost:0". The
   latter didn't make sense when the local server was using a Unix
   domain socket.
4) Clients are expected to keep a persistent connection to the server
   instead of creating a new connection each time a request is made for
   optimal performance.
5) Most of the client logic has been moved to the hashserve module in
   bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
   This command can be used to query a hash equivalence server, including
   fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
   optimize hash lookups. This change is backward compatible, as the
   database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
   performance with persistently connected clients. This requires Python
   3.5 or later.

(Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 17:52:01 +01:00
Richard Purdie
f025010ee0 bitbake: runqueue/siggen: Optimise hash equiv queries
We only have hash equivalence for setscene tasks so only query the server
for those, reducing the number of connections needed.

(Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:45 +01:00
Martin Jansa
85f8e4c04a bitbake: Revert "bitbake: cooker: Ensure bbappends are found in stable order"
This reverts commit 94c0c7f15c7a6244a8576ed948ffc21afb96ba82.

This ignores the layer priority, making the issue much worse.
E.g. I'm seeing a lot of failures caused by missing users, because
base-passwd bbappends applied in unexpected order caused different
passwd.master to be found in re-ordered FILESPATH.

(Bitbake rev: 2dc862237dba82da37c8ac9289e0a21409b1305c)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:45 +01:00
Wes Lindauer
f73db27088 bitbake: bitbake: cooker: Ensure bbappends are found in stable order
Thanks to wildcards in bbappend filenames, it's possible to have
multiple bbappends that apply to the same recipe in the same directory.
In order to get sstate hits between different workspaces, we want to
apply those bbappend files in a consistent order.  Since readdir()
returns files in a non-deterministic order between workspaces (based on
inode number and/or time of creation), we'll need to sort its result in
order to have any consistency.

(Bitbake rev: 94c0c7f15c7a6244a8576ed948ffc21afb96ba82)

Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07 13:08:54 +01:00
Martin Jansa
bf2e0b039a bitbake: fetch2: show warning when renaming the archive with bad checksum failed
* noticed on read-only sshfs premirror
* it was showing the warning about renaming the file:
  WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441

  and then failed because of movefile() issue with python3 (fixed in previous commit):
  ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:

  with movefile() fixed, it let do_fetch continue and re-fetch locally with the right
  checksum, but still the renamed file didn't exist, because of movefile failure - add
  another warning when the movefile fails - for whatever reason - unfortunately movefile
  prints error messages with just print() so the real error is hidden only in log.do_fetch
  in this case:
  movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441'

(Bitbake rev: 9a1bf4ba9ec00c2a222d820f8f83d1f056b021d6)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-03 10:18:58 +01:00
Martin Jansa
3e21648723 bitbake: utils: Fix movefile() exception handling with python3
* with python3 this fails with:
  File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile
       0795:        try:
       0796:            os.rename(src, destpath)
       0797:            renamefailed = 0
       0798:        except Exception as e:
   *** 0799:            if e[0] != errno.EXDEV:
       0800:                # Some random error.
       0801:                print("movefile: Failed to move", src, "to", dest, e)
       0802:                return None
       0803:            # Invalid cross-device-link 'bind' mounted or actually Cross-Device
  Exception: TypeError: 'OSError' object is not subscriptable

(Bitbake rev: d6e43c443ddbbe467c4380c48d2bc28ae18504a1)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-03 10:18:58 +01:00
Armin Kuster
963faf7a7f bitbake: tests/fetch: Resolve fetch error in bitbake-selftest
FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring
      self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
      AssertionError: '' is not true : Could not find upstream version for db

[YOCTO #13496]

The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget.

Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the
recipe. Also change the version being checked.

(Bitbake rev: 4cf5bb761c561ddea86f2875be35d05abc8486e1)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-01 22:30:42 +01:00
Chen Qi
1af3e4bea6 bitbake: cooker.py: remove generation of recipe-depends.dot
The information of recipe-depends.dot is misleading.

e.g.
$ grep xz recipe-depends.dot | grep bzip2
"bzip2" -> "xz"
"xz" -> "bzip2"

Users would wonder why they get some circular dependency.

The information is derived from removing the task names
of task-depends.dot. It's not giving people any additonal
information, and it's misleading.

So we remove the generation of this file.

(Bitbake rev: 4c484cc01e3eee7ab2ab0359fd680b4dbd31dc30)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-28 16:44:09 +01:00
Joshua Watt
46a71949de bitbake: bitbake: respect force flag in runall and runonly
Specifying the force flag will now cause runall and runonly to
invalidate the tasks before running them. This allows a --runall or
--runonly to force the tasks to run, even if they would have otherwise
been skipped, e.g.:

 bitbake -f --runall fetch

Will run all do_fetch tasks even if they wouldn't be necessary (for
example, skipped by setscene)

(Bitbake rev: 71e52d3822016027106f2a2e74b8dfdf20f5dc1e)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-28 16:44:09 +01:00
Richard Purdie
bb1e701fb1 bitbake: runqueue: Optimise build_taskdepdata slightly
Rather than repeatedly calling mc_from_tid() do this in the parent,
removing around a million function calls. Takes time spent in this
function from 40s to 36s.

(Bitbake rev: 28b3f0d8867804799420689c314ac4a8f01efb8c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:30:13 +01:00
Richard Purdie
4cc3c0daed bitbake: runqueue: Further optimise holdoff tasks
There are other data structures which can be reprocessed at the same
time as holdoff_tasks, further improving build efficiency in various
places.

(Bitbake rev: 02090b3456b7a2de12e72dfeaabfd3b631609924)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:30:13 +01:00
Richard Purdie
fece8f4737 bitbake: runqueue: Optimise holdoff task handling
We don't need to process the holdoff task list until we're executing tasks
which saves some data manipulation, at the cost of some data structures
not being correct at all times. This saves significant amounts of time
in various profile charts of larger builds.

(Bitbake rev: 270f076111b12eab358417b0c4cf9c70d7cc787a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:30:13 +01:00
Richard Purdie
3afc48c38f bitbake: runqueue: Small but critical fix
We've observed do_package and do_package_setscene running in parallel. The
reason is that holdoff_tasks wasn't getting updated. Looking at the code, it
would seem the reason is that the task was in pending_migrations and hence
changed wasn't set and holdoff_tasks wasn't updated.

Fix this. It only affects builds with rehashing enabled.

(Bitbake rev: e26e61e84575669bd223f6ab316798097ed95ec8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:30:13 +01:00
Richard Purdie
e087855d48 bitbake: cookerdata: Delay the setup of the siggen slightly to allow metadata defined siggens
If we define a metadata siggen it can fail due to the early init here. Move
slightly later to avoid those failures which allows fixes in OE to the
check-layer script related to the hash equiv siggen.

(Bitbake rev: fdf5c341f3393173876a753c46c9bd067eb2b353)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:30:13 +01:00
Richard Purdie
20946c63c2 bitbake: runqueue: Ensure target_tids is filtered
bitbake <target> --runonly=fetch

failed as the target_tids list included entries which were no longer targeted
task ids. Fix this.

(Bitbake rev: 94e848ae6544e628a19cb97115279b0b1678967c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-16 10:05:33 +01:00
Richard Purdie
0b38a37e59 bitbake: hashserv: Ensure we don't accumulate sockets in TIME_WAIT state
This can cause a huge backlog of closing sockets on the server and
in our case we don't really want/need the protection TCP is trying to
give us so work around it.

(Bitbake rev: 7bc79fdf60519231da7c0c7b5b6143ce090ed830)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-16 10:05:32 +01:00
Robert Yang
2cb6a75b5a bitbake: bitbake: server/process: Handle BBHandledException to avoid unexpected exceptions
The parseBaseConfiguration() raises bb.BBHandledException(), but
BitBakeServer() didn't handle it, so we always got unexpected exceptions when
there were errors. For example:

=== Case 1:
* Add "print "hello"' in base.bbclass' def oe_import() function
def oe_import(d):
    print "hello"
[snip]

$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Last 60 lines of server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
  File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 21
    print "hello"
                ^
SyntaxError: Missing parentheses in call to 'print'

<The first exception>
During handling of the above exception, another exception occurred:
<Tracebacks>

<The second exception>
During handling of the above exception, another exception occurred:
<Tracebacks>

<The third exception>
During handling of the above exception, another exception occurred:
<Tracebacks>
[snip]

Now it looks like:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 21:

The code lines resulting in this error were:
     0001:def oe_import(d):
 *** 0002:    print "hello"
     0003:    import sys
     0004:
     0005:    bbpath = d.getVar("BBPATH").split(":")
     0006:    sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
SyntaxError: Missing parentheses in call to 'print' (base.bbclass, line 21)

=== Case 2:
* Add 'HOSTTOOLS += "hello"' to conf/local.conf:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
<Tracebacks>
[snip]
During handling of the above exception, another exception occurred:
[snip]
<Tracebacks>
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

The error message is printed by bb.fatal() which raises bb.BBHandledException(),
but BitBakeServer() doesn't handle it, so we got it.

Now it looks like:
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

No unexpected exceptions anymore.

[YOCTO #13267]

(Bitbake rev: 6e6865e6371dbd31a136eae64cc5b1fa5f5bee33)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-16 10:05:32 +01:00
Richard Purdie
5f0d31ce46 bitbake: runqueue: Ensure data is handled correctly
This doesn't appear to have ill effects right now but there is a correctness
issue which this so fix it.

(Bitbake rev: a5e084a266f63c2fd370122327615e49beaeb94e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-15 13:32:06 +01:00
Richard Purdie
2fa5d98f52 bitbake: runqueue: Fix data corruption problem
This was overwriting data in the parent which caused all kinds of
odd/weird failures.

(Bitbake rev: 4c5aeb424247a9d0c907524ffacd9c61fcdc0852)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-15 13:32:06 +01:00
Richard Purdie
a3c53bff11 bitbake: tests/runqueue: Fix tests
There were paths being accidentally included in some of the hashserv tests. Remove
that and update the hashes so the tests work independently of paths.

(Bitbake rev: 6ddb9f09cb60c2354fa6a67cce412c4dc1e7dc2d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-15 08:43:22 +01:00
Richard Purdie
93e4dc5f11 bitbake: runqueue: Ensure we clear the stamp cache
When the task hashes change we need to ensure the stampcache is cleared out
else tasks don't rerun when they should as we're basing decisions on stale
cache data.

(Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
64d0493d97 bitbake: runqueue: Add missing setscene task corner case
We weren't marking this special case of setscene task as buildable leading to
runqueue task failures.

(Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00