Commit Graph

90 Commits

Author SHA1 Message Date
Joshua Watt
ba491952ee classes/buildhistory: Add option to strip path prefix
Adds an option to strip a prefix from the paths reported in
buildhistory. This makes it easier to compare task signatures in the
build history when the builds were done from different directories.

(From OE-Core rev: 194e7a29212c4a29222730f47d3133dfe92447c1)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-03 23:09:41 +01:00
Martin Jansa
95bca793e4 sstatesig.py: make it fatal error when sstate manifest isn't found
* all known issues in this area were fixed, make it fatal that it
  cannot be overlooked if someone triggers this issue again

(From OE-Core rev: 64b89f3c8fc31842256c482a3039d90d3f12c1cc)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-21 15:18:23 +01:00
Martin Jansa
3cffbb7261 sstatesig.py: show an error instead of warning when sstate manifest isn't found
* not sure if there are some valid use-cases for missing manifest, but
  recently I'm seeing increasing number of build failures where something
  from native recipe is missing (seen it with pseudo, autoconf, nodejs
  recently) and the only indication that something is wrong (before showing
  sometimes misleading error like:
  recipe-sysroot-native/usr/bin/node: No such file or directory
  is this warning:
  NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
  NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded

  if I rebuild that native dependency, then it gets fixed and I don't
  see these failures in clean builds (as without sstate and with empty
  TMPDIR), only in incremental builds

* but if there isn't valid reason for missing manifest file, then I think
  it would be better to error early (or even bb.fatal())

(From OE-Core rev: 95607a26854d873399d2b9d7e5fcbffc0cbdba4c)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 15:21:20 +00:00
Tomasz Dziendzielski
899514c477 sstatesig: Add descriptive error message to getpwuid/getgrgid "uid/gid not found" KeyError
If path is not owned by any user installed on target it gives
insufficient error "getpwuid(): uid not found" which may be misleading.
This exception occurs if uid/gid of path was not found in PSEUDO_PASSWD
files, which simply means the path is owned by host user and there is
host user contamination.

Add more information to the exception message to make it easier for user
to debug.

[YOCTO #14031]

(From OE-Core rev: 38540b59ed4ec8632e30a5fd6364b010d9da8470)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-03 21:45:49 +00:00
Richard Purdie
3ecf5d9692 uninative: Don't use single sstate for pseudo-native
pseudo-native is a bit special. It conditionally compiles in support for
xattr, statx and statvfs amongst other options. If a pseudo-native binary is
used on a system where these functions are present but it wasn't compiled in
we see hard to debug permissions problems.

An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target
oe-selftest which shows a cryptic error:

  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target
    self.assertEqual(filelist1, filelist2)
  File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual
    self.fail(msg)
  File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail
    raise self.failureException(msg)

AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8']

First differing element 0:
'-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor'
'-rwxr-xr-x root root /etc/init.d/mdmonitor'

This is due to a version of pseudo without statx being used on a system where
ls uses statx, hence the files are displayed as 6000.6000 instead of root.root.

Avoid this by always building pseudo-native for the specific distro in question rather
than using a universal sstate feed.

This hopefully fixes one of the mysterious AB-INT issues.

(From OE-Core rev: 6e3785a3f1f3cf68f5fe101cd6bebe91db165973)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:53:07 +00:00
Richard Purdie
235101a662 sstatesig: Remove workaround for bitbake taskhash bug
When trying to lock an individual signature, we see the checksum calculations
of dependent tasks failing. The fix is to remove a bad optimisation within
bitbake but with the removed, we need to remove some bogus code with
OE-Core's sstatesig code too.

(From OE-Core rev: 9923392539b1ce6d70f713527373d6bbc03f3021)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-03 08:21:10 +00:00
Richard Purdie
cc840dfc9d sstatesig: Log timestamps for hashequiv in reprodubile builds for do_package
Currently if a task generates the same output with different timestamps,
hasequiv won't detect it but reproducibile builds will fail tests due
to the different timestamps.

Add do_package timestamps to the hash when reproducibile builds are enabled
to avoid this.

(From OE-Core rev: 11e8200ccec765ff6a4263e06512e5751eca261a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-30 13:04:39 +00:00
Richard Purdie
a2db4fa127 base/bitbake.conf: Enable pseudo path filtering
This is a pretty big change to the way pseudo operates when used in OpenEmbedded.
Normally, pseudo monitors and logs (adds to its database) any file created or
modified whilst in a fakeroot environment. There are large numbers of files
we simply don't care about the permissions of whilst in that fakeroot context,
for example ${S}, ${B}, ${T}, ${SSTATE_DIR}, the central sstate control
directories,

This change uses new functionality in pseudo to ignore these directory trees,
resulting in a cleaner database with less chance of "stray" mismatches if files
are modified outside pseudo context. It also should reduce some overhead from
pseudo as the interprocess round trip to the server is avoided.

There is a possible complication where some existing recipe may break, for
example, we found a recipe which was writing to "${B}/install" for
"make install" in do_install and since we listed ${B} as not to be tracked,
there were errors trying to chown root for files in this location.

This patch fixes a few corner cases in OE-Core when used with this new
ignore list:

* The archiver directory matched a "${WORKDIR}/deploy*" pattern so was renamed
  to something else since that directory does need its root permissions
* The ${S} and ${B} ignoring is conditional on them being different to ${WORKDIR}
* package_write_* task output (the debs/rpms/ipks) are now owned by the build
  user so we don't want the file ownership information in the hashequiv outhash
  calculation even if they are built under pseudo.
* The fontcache postinstall intercept is run under qemu outside of pseudo context
  so delete files it may delete up front where pseudo can see this.
* SSTATE_DIR is in PSEUDO_PATHS_IGNORE, which is in FAKEROOTENV which is cached
  by bitbake. We therefore need to trigger reparsing if this changes, which means
  SSTATE_DIR can be in BB_HASHBASE_WHITELIST but not BB_HASHCONFIG_WHITELIST.
  Rework the variables to handle this. This otherwise breaks some of our sstate
  tests in oe-selftest.
* Ignore the temp directory wic uses for rebuilding rootfs.

(From OE-Core rev: ad8f5532ffaead9a5ad13e1034fe9e5e1b7979f4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-07 11:18:38 +01:00
Joshua Watt
48a883bb6e sstatesig: Account for all dataCaches being passed
Bitbake now passes all the dataCaches to the taskhash API, so use this
to correctly filter mcdepends.

[YOCTO #13724]

(From OE-Core rev: 749731a420fb905b6af97ce2909f06b2bcd14fe2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-15 14:53:46 +01:00
Richard Purdie
fee9be0f65 abi_version/sstatesig: Introduce HASHEQUIV_HASH_VERSION
We've found we need a way to cause a change in signatures and move
to a new hash 'namespace' with hashequiv. This introduces a variable
which allows us to do this.

(From OE-Core rev: 1c2d5aeae2fa2dfc16bc60b6cf0b310f826337fd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-28 23:11:28 +00:00
Richard Purdie
ecbf203e34 sstatesig: Improve debug output if getpwuid() fails
If getpwduid fails, we don't see which file it failed on which is key information
to aid debugging. Print this information when exceptions are raised.

(From OE-Core rev: 9d7a5219713af8117eda145052c6d9abdf577d8f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-16 22:33:09 +00:00
Richard Purdie
76af397213 sstatesig: Avoid resetting taskhash within siggen for locked sigs
Since get_unihash uses taskhash as a key internally, changing it means
different bebahour when locked sigs are active verses not active. Under
corner cases this leads to a signature mismatch.

Avoid this by by adding a wrapper for the place its externally exposed
and then not changing the internals.

(From OE-Core rev: 23b7484d326cf7fac013384598d7745a042eaa75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-03 22:35:48 +00:00
Richard Purdie
737c5b27bb sstatesig: Fix locked signature handling with unihashes
get_taskhash will call get_unihash internally in the parent class. We
need to disable our filter of it whilst this runs else incorrect hashes
can be calculated.

This is believed to be causing the locked signatures test to fail under
some circumstances (depending on whether earlier hashes are being
remapped).

[YOCTO #13605]

(From OE-Core rev: 523c093a882f6831ba75b5c4513837554d7e2414)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-03 22:35:48 +00:00
Richard Purdie
7538be0a50 sstatesig: Test cross/native hashserv method extension
We can have one taskhash which represents multiple native/cross sstate objects
since they're stored by BUILD_ARCH or possibly host distro (or host gcc version).
We need to put these into separate namespaces on hashserv since their outhashes
will never match and we need deterministic lookups for the different namespaces.

Use this extramethod option to handle this. This fixes several problematic
failures on the autobuilder.

(From OE-Core rev: 2a76082363d189880613765ad339718e3614049d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-02 16:43:01 +00:00
Richard Purdie
ba4dbacedd sstatesig: Further optimise
Optimise the call into the parent function to be only when needed.

(From OE-Core rev: 198bacaa1068fd68075d9182318fe64e9809dd36)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:25:49 +00:00
Richard Purdie
58726c4a56 sstatesig: Optimise get_taskhash for hashequiv
With hashequiv the get_taskhash function is called much more regularly
and contains expensive operations. This these don't change based upon
hash in a given build, improve the caching within the function to
reduce overhead.

(From OE-Core rev: de98cfe3cde4b8d5f4b163b5fba3f129651ef06a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15 09:10:46 +00:00
Richard Purdie
d80dd1c3f9 lib/sstatesig: Fix class inheritance problems
The locked sigs class needs to be inherited after the hashequiv mixin so
that get_unihash can correctly wrap the underlying hashequiv function.

To do this turn the locked sigs class into a second mixin, then the order
can be correctly handled. Tweak the get/set_taskdata to match.

(From OE-Core rev: 73d3414ecf594e426e6024b71931038a201eca52)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Richard Purdie
c92aca546e sstatesig: Fix hash equivlanency locked signature issues
Using locked signatures with the hash equivalency server ran into
problems. We need to:

a) Ensure the lockedhashes data object is passed from the core to
   any individual tasks using the get/set_taskdata methods

b) Return a locked singature instead of a unihash

c) Write the unihash being used to locked signature lists rather than
   the calculated taskhash

d) Skip warnings of hash mismatch if the hash is a unihash

These changes fix esdk builds (which use locked sigs) when a hash equivalence
server is in use.

(From OE-Core rev: 25dc3d78de01dffa77a3a2452d6a97d741b446d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Joshua Watt
34923e4f77 Remove SSTATE_HASHEQUIV_SERVER
Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This
variable is redundant now that BB_HASHSERVE is present.

(From OE-Core rev: 54b3adb6bc90a8e4b9e92952688772ee074d36e5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 17:52:01 +01:00
Joshua Watt
f12e41c1bb sstatesig: Update server URI
The server no longer uses a "http://" URI, since it has been updated to
use a different protocol.

(From OE-Core rev: 519561172c48bc7f7a61a3d02edd418fc0895b7b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 17:52:01 +01:00
Richard Purdie
359d61067a sstatesig: Fix leftover splitting issue from siggen change
(From OE-Core rev: 8b76048da99ec3be7d763b7ccc81f767c7015bc7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Richard Purdie
a0542ed3ff sstatesig: Updates to match bitbake siggen changes
Update the metadata to correspond to the bitbake siggen task specification
format change. This standardises on "<fn>:<task>" everywhere rather than
the "." delimiter that was being used in some places.

This is an API breaking change but means we now have a consistent format
being used throughout the codebase without compatibility APIs.

(From OE-Core rev: 23db236a054ee7a989cdbbcb42ad5c6eefd4a6ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
bcc9f1b5f1 sstate/sstatesig: Update to new form of BB_HASHCHECK_FUNCTION
Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION,
adapt to this cleanup. This is an API breaking change.

(From OE-Core rev: a4d413d8d809132b0e0a5dd673a36e2bd0e0be4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
8d7f663136 sstatesig: Update to handle BB_HASHSERVE
Update the code to handle BB_HASHSERV, an autostarted bitbake internal hash
equivalency server suited to local developer usage.

(From OE-Core rev: 9b3d7b0b0ff27281391c2b8f2511d312d6995ed6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
56b178ca2f sstatesig: Adpat to recent bitbake hash equiv runqueue changes
Upstream bitbake now hanes preserving the unihash data itself, drop
this usage of persist_data which was extremely problematic due to
concurrent task access issues, particulary on loaded systems.

(From OE-Core rev: 034d91c2c94b201797a7830b0af6141132f9bad1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
00f40114b4 sstatesig: Add debug for incorrect hash server settings
If the hash server settings are incorrect, show the user useful
error messages instead of tracebacks.

(From OE-Core rev: 2df5d95b9b63e30ddaa7c72a4173b9a05b3f15f9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
62c8b8c4d9 sstatesig: Move unihash siggen code to bitbake
This code is closely tied with the hash server in bitbake and also means
we can't relibably test the hashserv runqueue functionality without OE
metadata. Moving this to bitbake as a MixIn class makes most sense
and encourages code collaboration and reuse as well as enabling easier
and more accurate testing of the APIs.

(From OE-Core rev: a2a9c6092d4dde706ed071b08a972d1d87184295)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
32df42cff5 multiconfig: Adapt to bitbake switch 'multiconfig' -> 'mc'
(From OE-Core rev: 8a6f7c1e455156966f467008645fef14db679ccf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Joshua Watt
1d86f65ff5 classes/sstate: Update output hash
Updates the output hash calculation for determining if tasks are
equivalent. The new algorithm does the following based on feedback:
 1) The output hash function was moved to the OE library.
 2) All files are printed in a single line tabular format
 3) Prints the file type and mode in a user-friendly ls-like format
 4) Includes the file owner and group (by name, not ID). These are only
    included if the task is run under pseudo since that is the only time
    they can be consistently determined.
 5) File size is included for regular files

(From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 14:35:58 +00:00
Richard Purdie
f08808340e lib/sstatesig: Update to longer sha256 hash length
Update the code to match the recent switch to sha256 hashes

(From OE-Core rev: 910a8f3e8e350ff5d316faf5e07fb40c6c51f45e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:35:56 +00:00
Joshua Watt
adc37721a8 sstate: Implement hash equivalence sstate
Converts sstate so that it can use a hash equivalence server to
determine if a task really needs to be rebuilt, or if it can be restored
from a different (equivalent) sstate object.

The unique hashes are cached persistently using persist_data. This has
a number of advantages:
 1) Unique hashes can be cached between invocations of bitbake to
    prevent needing to contact the server every time (which is slow)
 2) The value of each tasks unique hash can easily be synchronized
    between different threads, which will be useful if bitbake is
    updated to do on the fly task re-hashing.

[YOCTO #13030]

(From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08 11:16:44 +00:00
Alejandro Enedino Hernandez Samaniego
b7c2c78580 multiconfig: Enable multiconfig dependencies on oe-core
This patch enables multiconfig dependencies (mcdepends) to be used on
recipes using the following format:

task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"

For the sake of simplicity consider the following example:

Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.

Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"

Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.

This patch makes modifications to bitbake.conf to enable mcdepends, and
to sstatesig and staging.bbclass to avoid conflicts between packages from
different multiconfigs.

[YOCTO #10681]

(From OE-Core rev: f71bfe833c657244d2fd07b3b71e86081d7d1c04)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-01 10:07:22 +01:00
Joshua Watt
c07d1447a9 sstatesig: Remove trailing whitespace
(From OE-Core rev: 2067172e2e5f90cb321ca124f4f9065ae8ad680f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-06 22:59:33 +01:00
Richard Purdie
522d422498 lib/oe/sstatesig: Fix task mappings from multilib<->non-multilib contexts
If we're in a multilib context already and want a non-multilib context
this function returned incorrect values.

Try and retain optimisations for the common case not needing to request
a datastore but allow the different multilib/non-multilib combinations
to work too.

This fixes bugs where rootfs generation of a multilib image would
write into incorrect locations, or be unable to find sstate manifest
files due to incorrect data stores being used to expand data.

(From OE-Core rev: af7cd7823a8cc95c9f849498b2416cdb0d4d103e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-02 21:37:47 +01:00
Joshua Watt
4118603777 sstatesig: Move hardcoded native tools
Now that SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS supports the wildcard syntax,
these exclusions no longer need to be hardcoded

(From OE-Core rev: 891ca3c891f2ed5476442442bd9784ba10b9a94d)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Joshua Watt
c3582d3669 sstatesig: Add recipe wildcard
The special string "*" on the left hand side of the dependency
specification matches all recipes except the one on the right hand side.

(From OE-Core rev: 526855f6573317ced913f39e878beac1d5d294de)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Joshua Watt
629619dce7 sstatesig: Ignore icecc-create-env in task sigs
Changes to the icecc-create-env recipe should not cause all recipes to
rebuild just because the have inherited icecc.

(From OE-Core rev: b088a9c34c34c4bda54d93907e04972d3f8ba5a1)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Richard Purdie
3eff72cc1f sstatesig/staging/package_manager: Create common sstate manifest code
Create a common function for locating task manifest files rather than
several implementations with missing pieces.

(From OE-Core rev: 68150bac7444f089f19c789e9f6602d59f605d7a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:55 +00:00
Paul Eggleton
d4fd7e2aa1 lib/oe/sstatesig: fix wildcard matching wrong task signature files
With a '*' as a wildcard for the signature here we can also match a
portion of the task name with the result that we may match a sigdata
file for the wrong task. Luckily the signature is always the same
length - 32 characters - so we can simply use 32 '?' characters instead.
(A regex would have been another alternative, but the wildcard should be
effective and I felt like a regex would complicate the code more than
this solution).

Fixes [YOCTO #11763].

(From OE-Core rev: 73c6f85457361cd82d44838946aa81ba9bd1662a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-05 22:33:23 +00:00
Ross Burton
173760abc6 lib/oe/sstatesig: exclude ccache-native from signature hashes
Enabling ccache should not cause a complete rebuild, so filter out ccache-native
from the dependencies the same way we do for quilt-native (so the world doesn't
repatch if quilt changes).

This doesn't effect the actual dependencies, just the dependencies that impact
the hash.

[ YOCTO #11417 ]

(From OE-Core rev: ffeca7dcc1abe82cad8374a31bf72b36a472fa1b)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Patrick Ohly
23ecae6d67 archiver.bbclass: do not cause kernel rebuilds
Adding or removing archiver.bbclass from a build configuration causes
rebuilds of linux-yocto-based kernels because of the
do_kernel_configme->do_unpack_and_patch task dependency.

This particular dependency can be ignored for the do_kernel_configme
sstate signature calculcation. Idea for the fix from Richard Purdie.

Note that building the kernel and adding archiver.bbclass later to
archive sources leads to do_unpack_and_patch running after
do_kernel_configme (because that already ran in the first build),
which might be problematic. This is independent of the change here.

The use case in YOCTO #11441 is to removed archiver.bbclass between a
production build with archiving enabled and builds via oe-selftests
without archiving. That direction is fine.

Fixes: YOCTO #11441

(From OE-Core rev: fed0ed82928e6a7846fbad233ac657bd17bcefc7)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-18 14:01:46 +01:00
Paul Eggleton
9049c09793 buildhistory-diff: add option to compare task signature list
Having added writing out of the task signature list to buildhistory
(when BUILDHISTORY_FEATURES includes "task"), we now need a way to
compare the list. This just shows which tasks have been added / changed
signature / removed.

(From OE-Core rev: 63bd7e9f780a98dda458d612877495756bcc5463)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Paul Eggleton
6774995322 classes/buildhistory: write out task signatures on every build
If we want to determine what changed since the last build, one angle
from which to look at it is to check the signatures. However, if we
don't actually have the signatures from the last build we don't have
anywhere to start. Save the signatures on each build in order to give us
the starting point.

NOTE: you need to set your BUILDHISTORY_FEATURES value to include
"task" to enable collection of these signatures as it is is disabled by
default.

(From OE-Core rev: 11f68f65c46c5bc26ddeeade3021e83b3a7f895a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Paul Eggleton
23095a6b01 lib/oe/sstatesig: avoid reporting duplicate siginfo files from sstate
In find_siginfo(), which is used by bitbake-diffsigs among other things,
avoid adding a siginfo file from the sstate-cache where we've already
collected a sigdata file from the stamps directory with the same hash.
This avoids the possibility that the top two files (as picked by default
using the bitbake-diffsigs -t option) are for the same signature and
thus the tool would report no differences. In order to do that, just use
the hashfiles dict that we already have - we just need to change the
code to populate that even if we're collecting matching files without
looking for a fixed set of hashes (i.e. taskhashlist isn't set).

This replaces previous code in bitbake-diffsigs that attempted to filter
these out with limited success.

(From OE-Core rev: 5b69eef40868180c59400624096d7ebbbbea446b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Paul Eggleton
27b7de94f0 lib/oe/sstatesig: fix finding native siginfo files in sstate-cache
When comparing signatures with bitbake-diffsigs -t or bitbake -S
printdiff, we use this find_siginfo() function implemented in this
module to find the siginfo/sigdata files corresponding to the tasks
we're looking for. However, native sstate files go into a
NATIVELSBSTRING subdirectory and there was no handling for this when
asking about native recipes.

I'm not even sure why we were walking SSTATE_DIR in order to find
this - we don't need to, we just need to run glob.glob() on the filespec
we calculate, which should be a little bit more efficient.

(From OE-Core rev: 8cb472e4ed25e56ec0d9cf6d8d101d1ab6687a5b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Jianxun Zhang
c782cb40d7 lib/oe.sstatesig: make locked sig file consistent
Sort keys of dict 'types' prior to dumping, in order to have
identical output every time. This could make it a little easier
to diff these human-readable dumps.

(From OE-Core rev: 8abbaba1931e2cb2b87aa733aa9a3e8eb359b500)

Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Joshua Lock
3c59b1bf93 meta: remove True option to getVarFlag calls
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.

Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)

(From OE-Core rev: 2dea9e490a98377010b3d4118d054814c317a735)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Joshua Lock
c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Paul Eggleton
af4d7258b1 lib/oe/sstatesig: print locked sigs file message only when explicitly called
If we're building the extensible SDK we don't need to see the "Writing
locked sigs" message; it's only necessary when the user explicitly runs
bitbake -S none <target>.

(From OE-Core rev: 440a351ee13920045c9d3e51882908f7b3f01d35)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:23 +01:00