Commit Graph

153 Commits

Author SHA1 Message Date
Richard Purdie
c204cc47d0 bitbake: cooker/siggen: Empty siggen cache during parsing
When parsing recipes its apparent the memory usage of bitbake rises linearly
with number of recipes parsed. It shouldn't.

Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
dependency information left behind in siggen was the culprit. Add a new
method to allow us to drop this information. We don't need it after the recipe
has been parsed and hashes calculated (at runtime its different but only the
currently executing task would be in memory).

This should give signficant memory usage improvements for bitbake and that
in turn should help speed on more constrained systems, as well as when used in
multiconfig environments.

(Bitbake rev: ef29309d0b512b64d024e383e7baff22c727711c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-18 13:59:06 +00:00
Richard Purdie
6eee202b9a bitbake: siggen: Avoid cache mismatch issues with locked sigs
If locked sigs are in use this function makes little sense, need to
avoid generating mismatch warnings.

(Bitbake rev: 95687be83e716220eb3893b67428f97fd59fc2c5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 27ad9c1d468fba858a4adeb56b605227b415ae0f)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-11 23:02:14 +00:00
Richard Purdie
57a4bcc6bf bitbake: siggen: Cache unihash values to avoid cache lookup
Add unihash cache of values to speed up cache lookup.

This avoids the overhead of the disk based check functions.

(Bitbake rev: 1f5a662176fd174c4793e3e28d1b4e95616b1715)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5c9cc45b60904a1c355db9bf9c4495f1b25aca37)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-11 23:02:14 +00:00
Richard Purdie
7412d9acf7 bitbake: siggen: Optimise get_unihash disk based cache handling
Currently the cache can grow huge since any previously used hash is
retained in the cache. This change moves to use one hash per task
which improves the speed of the functions considerably. Currently
performance is an issue, as are very large cache files and cache
load time.

By moving to a single hash per task, the shorted filename as a key
is no longer usable as the same recipe has multiple variants for
the same filename so this has to change.

(Bitbake rev: 8f4f6c2f9acab23bc795ffe389c4cd74711d10ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ed764e7fcf04b6d0ba6b4cac7415b1ee8f492865)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-11 23:02:14 +00:00
Richard Purdie
5a7ede639f bitbake: siggen: Test extra cross/native hashserv method
Hack the hashserv to allow extra data to be injected into the hashserv
method. This allows OE-Core to handle cases where there are multiple
sstate objects for the same taskhash, e.g. native/cross objects based
upon BUILD_ARCH or the host distro (when uninative isn't used).

This has been tested and proven to be very effective. We will likely
rework the code to improve how this is handled but for now this
improves automated builds until we can get to that refactoring and
more invasive changes.

(Bitbake rev: e21bf572cfe18e805d3f849777189685f7391a67)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0a09b0fa03d1afc08037964dc63a18ef7cff9c78)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11 11:06:23 +00:00
Aníbal Limón
f619b13403 bitbake: lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.

Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"

(Bitbake rev: 4c7d689341f471efdf8ab7c2a7c6a19d2d370f5c)

Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 923aff060d8aba8456979c35b16d300ba7c13ff9)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11 11:06:22 +00:00
Richard Purdie
56631225aa bitbake: siggen: Split get_tashhash for performance
There are two operations happening in get_taskhash, the building of the
underlying data and the calculation of the hash.

Split these into two funtions since the preparation part doesn't need
to rerun when unihash changes, only the calculation does.

This split allows sigificant performance improvements for hashequiv
in builds where many hashes are equivalent and many hashes are changing.

(Bitbake rev: c4ce216b4c5a1626764752edd21005750d05a0c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6a32af2808d748819f4af55c443578c8a63062b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11 11:06:22 +00:00
Richard Purdie
c37d0c4aa5 bitbake: siggen: Fix performance issue in get_unihash
There is a significant performance issue in get_unihash(). The issue turns out
to be the lookups of setscene tasks. We can fix this by using a set() instead of
the current list.

(Bitbake rev: 5afad266f2ce55db2038c36f2e49a3c80be9bbfc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1e561672d039ebfb8cd0e0654a44dcf48513317c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11 11:06:22 +00:00
Richard Purdie
7448f9a180 bitbake: siggen: Ensure new unihash propagates through the system
Its possible the new unihash may not exist in sstate. Currently the code
would create an sstate object with the old hash however this updates it to
create the object with the new unihash.

(Bitbake rev: 0aee83e4e31dff7f4354e4eb4cbd35dd592e9f06)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit abcaa1398031fa5338a43859c661e6d4a9ce863d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11 11:06:22 +00:00
Richard Purdie
376f8eaeb9 bitbake: siggen: Avoid taskhash mismatch errors for nostamp tasks when dependencies rehash
An example:

NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started
ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded

Is caused by a rehash in a dependency happening somewhere earlier in the build
and the taint being reset.

Change the code so that nostamp taints are preserved to avoid the issue.

(Bitbake rev: c42d00ff293d0538cad1b84c108bf7f5f49d4d84)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 61624a3fc38e8546e01356d5ce7a09f21e7094ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11 11:06:22 +00:00
Richard Purdie
b8f326a778 bitbake: runqueue/siggen: Allow handling of equivalent hashes
Based on the hashserv's new ability to accept hash mappings, update runqueue
to use this through a helper function in siggen.

This addresses problems with meta-extsdk-toolchain and its dependency on
gdb-cross which caused errors when building eSDK. See the previous commit
for more details.

(Bitbake rev: 222df6d6b832868c6e87334f8acdd74b730a91d6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11 11:06:22 +00:00
Richard Purdie
456f5e0d23 bitbake: siggen: Fix hashequiv bug where new hash wasn't referenced correctly
If a hash is reported to the hash server, the stamp written out by the
current task didn't account for any new hash the server may have provided.
Fix this so the correct stamp is written. This means "bitbake X; bitbake X"
no longer rebuilds lots of things when hashequiv is active.

(Bitbake rev: 4299afdd290f9d1c5616598f5fe83c195a64b63c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-25 21:26:15 +00:00
Richard Purdie
a73cbe649a bitbake: siggen: Remove full path from unitaskhashes keys
The full paths make the cache useless in the sdk. They also bloat the
cache size. They're for human debugging benefit only so compromise and
reduce this to the filename.

(Bitbake rev: 3b275c4083eae1d3781f0862919af9de83932b0f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
ba0ff38cab bitbake: siggen: Avoid writing misleading sigdata files
Use the unihash in the output filename of sigdata files else the contents
of stamp directories is misleading. Write the unihash into the singature to
make it clear what happened.

(Bitbake rev: feb01ee54d3706fe93768f332054c7532f7209e4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
7ab4808e0a bitbake: siggen/runqueue: Fix signature mismatch issues
We need to set the setscene tasklist before we call into the
taskhash/unihash code else the behaviour is inconsistent.

Avoid reporting hashes for non setscene tasks since we'd never
query that.

(Bitbake rev: 419a7840b8627278db694029c25df00214d01d96)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Richard Purdie
155249b1db bitbake: siggen: Ensure setscenetasks list is available to worker context
The setscenetasks list needs to be available in the worker contexts
else the signature behaviour there mismatches what the server does.

Add the data to get/set_taskdata to ensure this happens.

(Bitbake rev: 632980ef90fe126b7ba3d138f4d574ae05914779)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 17:23:35 +01:00
Joshua Watt
1805574559 bitbake: siggen: Fix attribute error when hashserver fails
The HashConnectionError class was moved to the client module and needs
to be updated.

[YOCTO #13537]

(Bitbake rev: 9fb862685e5e5a2aa534bc25cab1e4158d708b40)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Joshua Watt
20f032338f bitbake: bitbake: Rework hash equivalence
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:

1) The server protocol is no longer HTTP based. Instead, it uses a
   simpler JSON over a streaming protocol link. This protocol has much
   lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
   domain socket. Unix domain sockets are more efficient for local
   communication, and so are preferred if the user enables hash
   equivalence only for the local build. The arguments to the
   'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
   equivalence server is changed to "auto" instead of "localhost:0". The
   latter didn't make sense when the local server was using a Unix
   domain socket.
4) Clients are expected to keep a persistent connection to the server
   instead of creating a new connection each time a request is made for
   optimal performance.
5) Most of the client logic has been moved to the hashserve module in
   bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
   This command can be used to query a hash equivalence server, including
   fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
   optimize hash lookups. This change is backward compatible, as the
   database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
   performance with persistently connected clients. This requires Python
   3.5 or later.

(Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 17:52:01 +01:00
Richard Purdie
f025010ee0 bitbake: runqueue/siggen: Optimise hash equiv queries
We only have hash equivalence for setscene tasks so only query the server
for those, reducing the number of connections needed.

(Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:45 +01:00
Richard Purdie
fd79638046 bitbake: tests/runqueue: Add hashserv+runqueue test
Add a test which tests the runqueue adaptations for hash equivalency.

(Bitbake rev: 477321d0780df177c1582db119c2bb6795912fc6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
4fbb862cdc bitbake: siggen: Clean up task reference formats
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.

This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.

Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.

(Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
40eb5b344b bitbake: siggen: Convert to use self.unitaskhashes
Rather than metadata driven sqlite databases for communication, use
bitbake's unitaskhashes variable instead.

(Bitbake rev: a0d941c787cf3ef030d190903279d311bc05d752)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
de143e0db6 bitbake: siggen: Add new unitaskhashes data variable which is cached
We need to preserve unihash task hashes between runs. Use the new SimpleCache
class to create such a class within the signature generator which is loaded
at init time and saved when builds complete. The default is unpopulated but
metadata sig handlers can populate this cache.

(Bitbake rev: 1f326f2c29c2664a5daaeeb0c1fd332630efbdba)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
db31374efa bitbake: siggen: Import unihash code from OE-Core
This code is closely tied with the hash server in bitbake and also means
we can't relibably test the hashserv runqueue functionality without OE
metadata. Moving this to bitbake as a MixIn class makes most sense
and encourages code collaboration and reuse as well as enabling easier
and more accurate testing of the APIs.

(Bitbake rev: 7bb79099a6c1b463d6ae9226c4cab5e76a965675)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
ae5615000b bitbake: siggen: Fix handling of tainted sig files
The addition of some debugging code meant that comparisions between sig
files with a taint and without a taint weren't working. Tweak the logic
to avoid tracebacks if one side doesn't have a taint.

(Bitbake rev: f5ea06fc2b6713c9f8e85ecf7cb981ae9a84d896)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-19 16:20:28 +01:00
Richard Purdie
8a2914fc98 bitbake: siggen: Use unique hashes for tasks
Now that runqueue optimises based on task hash, we need to ensure
tasks have unique hashes even in the simplest siggen mode. Use
the task name to calculate a unique hash.

This fixes runqueue tests when hash optimisations are added.

(Bitbake rev: 8ede873ef4ef492fbaf01474685c1ca8b34d80d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15 09:31:48 +01:00
Richard Purdie
eb3ae4a132 bitbake: siggen: Fix default handler
After the unihash changes the default signature handler didn't work. Tweak it
to adapt to those changes (allowing the runqueue tests to work).

(Bitbake rev: 7d486d3fb7176a3486f3f2484457724d7185df58)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15 09:31:48 +01:00
Richard Purdie
1f68b0bd98 bitbake: multiconfig: Switch from 'multiconfig' -> 'mc'
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.

mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.

[YOCTO #11168]

(Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Robert Yang
7f7a7cba4d bitbake: bb: siggen: Print more info when basehash are mis-matched
This is useful for debugging.

(Bitbake rev: 07b6054b32b37375e2de4b1276bf6fe3f9ad0b04)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-23 23:30:45 +01:00
Robert Yang
3c06699da3 bitbake: bb: siggen: Make dump_sigfile and compare_sigfiles print uuid4
This can make people easier to understand bitbake-diffsigs/dumpsig's output,
otherwise, it's hard to know it is a random uuid unless look into the code.

E.g.:
$ bitbake bc-native -ccleansstate -Snone
$ bitbake bc-native -ccleansstate -Snone
$ bitbake-diffsigs tmp/stamps/x86_64-linux/bc-native/1.07.1-r0.do_cleansstate.sigdata.*

* Before:
  Taint (by forced/invalidated task) changed from nostamp:fe79d162-c4a8-4174-8007-f6d4aa09abdc to nostamp:28192187-5021-40c1-9e21-45483b62c910

* Now:
  Taint (by forced/invalidated task) changed from nostamp(uuid4):fe79d162-c4a8-4174-8007-f6d4aa09abdc to nostamp(uuid4):28192187-5021-40c1-9e21-45483b62c910

(Bitbake rev: 724b4a5cec8c611d53350f3e5a3988ec3222684b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-23 23:30:45 +01:00
Richard Purdie
3bf0602713 bitbake: siggen: Fix multiconfig corner case
There was already a fix to ignore some multiconfig dependencies but its
'opposite' case wasn't covered. Cover that combination to so as to avoid
tracebacks in multiconfig builds.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: c883dfe378af9dfc192a8e392e84325d68648806)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
e5455662a9 bitbake: bitbake: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

(Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Richard Purdie
f008c24dab bitbake: data/siggen: Switch md5 -> sha256
Similarly to the codeparser change, change to sha256 hashes due to worries
over collisions. The main impact of this change is slightly slower parsing
time as well as longer sstate file names.

(Bitbake rev: 66f1b766997d53b4375fdd25719b1175f3828903)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08 11:16:03 +00:00
Joshua Watt
960fb3ed02 bitbake: siggen: Split out task unique hash
Abstracts the function to get the unique hash for a task. This hash is
used as in place of the taskhash for the purpose of determine how other
tasks depend on this one. Unless overridden, the taskhash is the same as
the unique hash, preserving the original behavior.

[YOCTO #13030]

(Bitbake rev: d0065b34cea81fcadff14b0383779e9559d94508)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-03 22:47:11 +00:00
Joshua Watt
e089fb1ae0 bitbake: siggen: Split out stampfile hash fetch
The mechanism used to get the hash for a stamp file is split out so that
it can be overridden by derived classes

[YOCTO #13030]

(Bitbake rev: ce241534d19b2f1c51dbdb3b92419676d234e464)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-07 12:38:58 +00:00
Peter Kjellerstedt
07e1a65f07 bitbake: siggen: Adapt colors used by bitbake-diffsigs to support light themes
The colors specified for use with bitbake-diffsigs were adapted for a
dark theme, e.g., by setting the background color to black, which made
it look very bad when used with a light theme.

To make it look good both with a dark or a light theme, it is better
to drop the background color. It is also better to leave out the color
altogether for the title and just use bold. Finally, dropping bold for
the red and green texts indicating removed/added values better matches
other colorized diff implementations as, e.g., git diff.

(Bitbake rev: f1a2c23520832ee91e85338c1ad8af1fec0d0b19)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-19 17:03:24 +00:00
Mike Crowe
b8e08c6519 bitbake: siggen: Correct format argument substitution
Use % operator to ensure that the dependency is substituted into the error
message correctly.

(Bitbake rev: 8e587386b0b3a0ed11edf71dfbe9fb508f60d0b3)

Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-07 23:08:55 +00:00
Richard Purdie
7e92127128 bitbake: data/siggen: Extract task hash generation code into a function
By creating a standalone function, we can add better functional testing
of this code.

(Bitbake rev: 796a20d24dc18479de1975a37b9e52a5ac75c73f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:27 +01:00
Alejandro Enedino Hernandez Samaniego
b8730f68d2 bitbake: bitbake: Add support for multiconfig dependencies
This patch adds the capability for tasks from different
multiconfigs to depend on one another.

These dependencies can be enabled using the following format:

task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"

For the sake of simplicity consider the following example:

Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.

Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"

Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.

This patch makes modifications to:
- cooker: To glue both multiconfigs in one place and make sure
  the dependencies can be provided.
- taskdata: To parse and add a new kind of dependency (mcdepends) to
  the taskdata object.
- runqueue: To differentiate tasks from different multiconfigs,
  add the specified dependencies to the corresponding tasks, and
  create a working runqueue that contains tasks from both multiconfigs.
- siggen: To avoid looking for tasks from different multiconfigs on
  objects where they dont belong.

The taskdata objects are still not aware of the concept of multiconfig,
so each object doesnt know which multiconfig its building, hence why
the mcdepends are added to all taskdata objects equally (we really
dont expect many of these), but the actual dependencies are added only
to the required tasks by the runqueue.

(Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-01 10:07:22 +01:00
Joshua Watt
6b66f22686 bitbake: siggen: Delete trailing whitespace
(Bitbake rev: b2fd783fa23403e8f08d998951b05bcabb458af1)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Paul Eggleton
f16cab3e74 bitbake: siggen: move reset() definition to base SignatureGenerator class
If we're implementing reset() in SignatureGenerator at all (and we need
to for a basic non-OE BitBake setup where that is the default signature
generator), then we need it to be clearing out the internal values
properly.

(Bitbake rev: 13f52d38fdbcb84c2a0c46f85baa44b22d53fdc1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-05 15:00:49 +01:00
Caio Marcelo de Oliveira Filho
49f530c218 bitbake: siggen: Add reset to SignatureGenerator
Fix failure after commit "cooker/siggen: Reset siggen when reparsing"
(e4c6ca9440f63761560b49bbe12654441f54687e) when executing without
specifying a BB_SIGNATURE_HANDLER.

(Bitbake rev: 2a78c2d09aea0323632bbc927f370f1d3c9c249e)

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:04 +01:00
Richard Purdie
ea9fff4bd0 bitbake: cooker/siggen: Reset siggen when reparsing
If we don't do this, we get basehash mismatch errors occurring from the reparse
which would then set bitbake's error exit code.

This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order
to fail with a non-zero BB_SERVER_TIMEOUT.

(Bitbake rev: e4c6ca9440f63761560b49bbe12654441f54687e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Richard Purdie
5f711bc63e bitbake: siggen: Make calc_taskhash match get_taskhash for file checksums
The code in these two functions is meant to be equivlanet in behaviour
but isn't. Add in code to ensure files that don't exist are handled
consistently by both functions. Users did report being able to generate
tracebacks otherwise.

(Bitbake rev: c11b4832879354b387a8c9c6a3096f2d621afeac)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Nathan Rossi
bf0fc2da66 bitbake: lib/bb/{data, siggen}: Don't warn on SkipRecipe exceptions
Recipes/variables that raise a SkipRecipe exception are intentionally
skipped, and should not generate warnings.

[YOCTO #11319]

(Bitbake rev: edf6e6094a9f7ad4b2ba06eef8fd34756edbedce)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:16:05 +01:00
Paul Eggleton
b372e85d9b bitbake: bitbake-diffsigs: colourise output
If the output is a TTY, add colour to the output in order to make it
easier to read. At the moment this is fairly basic, just add colour to
the "titles" of each change and to the diff output.

I tried to introduce this without changing the code too much - rather
than moving everything over to the new python formatting style, I've
introduced a color_format() function which takes care of the colour
formatting, either accepting additional format arguments or
alternatively leaving the caller to use the old-style formatting (%) to
insert values.

(Bitbake rev: 04a023c8fdea1e1812fcdcaf00345aab59f9abe1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
5f7bf1f66d bitbake: lib/bb/siggen: show word-diff for single-line values containing spaces
If a variable value has changed and either the new or old value contains
spaces, a word diff should be appropriate and may be a bit more readable.
Import the "simplediff" module and use it to show a word diff (in the
style of GNU wdiff and git diff --word-diff).

Also use a similar style diff to show changes in the runtaskhashes list.
I didn't use an actual word-diff here since it's a little different - we
can be sure that the list is a list and not simply a free-format string.

(Bitbake rev: 20db6b6553c80e18afc4f43dc2495435f7477822)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
5d8b89fc0b bitbake: lib/bb/siggen: add collapsed mode to compare_sigfiles()
If we just want to drill down to the actual differences then we don't
need to see certain things in the output, e.g. basehash changing or the
signature of dependent tasks. This will be used for comparing signatures
within buildhistory-diff in OE-Core; the default mode as used by
bitbake-diffsigs and bitbake -S printdiff remains unchanged for the
moment.

(Bitbake rev: 6543a59b1ebd3194a7c6421cffc66ebe31a67c62)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton
8c015e45ff bitbake: lib/bb/siggen: don't show unchanged runtaskdeps list
If the runtaskdeps list hasn't actually changed (but the signatures of
some of the tasks did) then it doesn't make sense to print out the old
and new lists as they are both the same and may be very long, e.g. for
do_rootfs in OE.

(Bitbake rev: cb170543605288b3e8badfac3a54c588f4c95413)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00