* I don't have any real evidence or good statistics for this, but when
comparing signature dumps from my big bitbake world builds I usually
see a lot of rebuilds caused by changes in .bbclasses and only very
rare would be the case where oe-core upgrade brings changes in -native
recipes and no change in .bbclasses used from target recipes
* changing the default to include them shouldn't cause significant
increase in rebuilds and sstate reuse a bit safer
* people working on toolchain (e.g. using gcc from AUTOREV) can easily
extend sstate_rundepfilter to ignore them again (it's easier than
removing existing filter), example how add own signature handler in
your layer is here:
9ac3a7c803
(From OE-Core rev: 336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, all GNOME terminals share a single process,
reducing memory usage. This can be disabled by starting gnome-terminal
with the --disable-factory option
However, gnome-terminal in Fedora 20 does no longer support the
'--disable-factory' flag, so remove it. As the support for 'mate' terminals was
added as a copy of the gnome code in 8cc078a9c679845464c59028f584d7aba098cc1f,
remove the flag here aswell.
(From OE-Core rev: e8dca725ed8211a874472300a3ed50e494039ab9)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like fetch, unpack and patch, populate_lic doesn't vary between different
archs so we should mark it as such. This means better sstate cache reuse
with fewer duplicate files as well as less confusing sstate debugging.
sstatesig also needs to account for the fact BPN is used for sstate files
in these cases.
(From OE-Core rev: 3d59d0bed756f64d0092caa3892239c779c4a341)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By resetting filesdates at this point, we lose matches from stamps which
may not have been in sstatedir. When we don't have hashes specicifed,
its better to return all matches and have the caller decide which are
relavent and which are not since this function has no ability to
decide. There will almost always be one match from stamps we need
to keep and refer to.
(From OE-Core rev: f4c1c9ad2c7e944d4926d0629611da97f9df6a9a)
(From OE-Core rev: 8629844640a5a4c6a9d9375e7b637bcbf3451c4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've dropped sstate-name so we can remove this code. The fallback was
incorrect since we use taskname without the do_ prefix so this patch
updates to account for that too.
(From OE-Core rev: 72ff58124081333d46d37f31f2d1bf40d715e3bd)
(From OE-Core rev: dbc1426ee75bcf2f6d8b18312522a9b823e23173)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the hashes to find isn't specified we need to return matches from both
the sstate cache and the local stamps directory regardless of how many we've
found so far. If we don't do this, we can miss stamps and the comparison is
less accurate/incorrect.
(From OE-Core rev: 08a074e11e2d517b81ca71fd9bda65297bb015a7)
(From OE-Core rev: 04fc682af7c1657b1c9f37b66a78c4ffc8066e24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add PACKAGE_EXCLUDE and NO_RECOMMENDATIONS to the info we track for
images, since these can change what ends up in the image.
(From OE-Core rev: a10189366f180b87f5be20b66834b7e7a9bb8c12)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the "ABI safe" recipes, we've been excluding those from signatures. This
is fine in the general case but in the specific case of image recipes it breaks.
A good test case is the interfaces file. Editting this causes init-ifupdown
to rebuild but not an image containing it (e.g. core-image-minimal).
We need to ensure the checksums are added to the image recipes and this change
does that.
(From OE-Core rev: fd085f15e7cd093953f974f69277e130174d551d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OE-Core commit b7de1eaac9eed559b2d68058f5de67de74a6cb58 added an extra
argument to the compare_dict_blobs() function but missed adding the
argument to one call to compare two versions of the image-info.txt file.
(From OE-Core rev: 24a45d752c3e3d0d8b59c040355e4fe7de22b041)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Files named .* in the top level of directories handled by this function
were getting lost after the directory copying command was fixed. Rather
than complicate the function further, use cpio instead.
(From OE-Core rev: 59612905e9bb865762667aa7b3cb06e53c3a4071)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.
Resolve the issue by telling tar not to recurse. This gives a significant
performance boost to various parts of the system (do_package for linux-yocto
256s -> 178s for example).
(From OE-Core rev: 8b90ed084d59b4e07aa547255d327b25cfb2ee2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
packagegroups are allarch and shouldn't change depending on the target
or machine selected. In general they should have good stable namespaces
for their dependencies. As such we can exclude them from rebuilding when
dependency checksums change.
(From OE-Core rev: 80b065ff46322ec0cad039dfd9eb2d010168dba6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a -v/--report-ver option to report changes in PKGE/PKGV/PKGR even
if the value is the same as the default from PE/PV/PR.
Also add a -a/--report-all option to report all changes instead of just
the default significant ones.
Addresses [YOCTO #5263].
(From OE-Core rev: b7de1eaac9eed559b2d68058f5de67de74a6cb58)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
tar version 1.27 returns:
tar: --same-order option cannot be used with -c
with the commandlines we have been using. We can remove the -s option (which
is --same-order) to remove the error.
(From OE-Core rev: 69c26e795c117aabfaf313abbfd10e70ede633d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we have a hierarchy of pkgdata directories and the code has to put together
a search path and look through each in turn until it finds the data it needs.
This has lead to a number of hardcoded paths and file globing which
is unpredictable and undesirable. Worse, certain tricks that should be
easy like a GL specific package architecture become problematic with the
curretn search paths.
With the modern sstate code, we can do better and construct a single pkgdata
directory for each machine in just the same way as we do for the sysroot. This
is already tried and well tested. With such a single directory, all the code that
iterated through multiple pkgdata directories and simply be removed and give
a significant simplification of the code. Even existing build directories adapt
to the change well since the package contents doesn't change, just the location
they're installed to and the stamp for them.
The only complication is the we need a different shlibs directory for each
multilib. These are only used by package.bbclass and the simple fix is to
add MLPREFIX to the shlib directory name. This means the multilib packages will
repackage and the sstate checksum will change but an existing build directory
will adapt to the changes safely.
It is close to release however I believe the benefits this patch give us
are worth consideration for inclusion and give us more options for dealing
with problems like the GL one. It also sets the ground work well for
shlibs improvements in 1.6.
(From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These have been deprecated for a long time, convert the remaining
references to the correct modules and prepare for removal of the
compatibility support from bitbake.
(From OE-Core rev: 6a39835af2b2b3c7797fe05479341d71a3f3aaf6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A simple clone of the corresponding Gnome class. Without this, devshell
fails completely on a default installation of MATE desktop Linux Mint 15.
(From OE-Core rev: 8cc078a9c679845464c59028f584d7aba098cc1f)
Signed-off-by: Andre McCurdy <andre.mccurdy@entropic.com>
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just as in f8ed7446755eeb88191e16749350efa1e7e6197c, tmux wants a single
argument for its command. This applies to the "split-window" command as
well as "new."
Note that this alone is not enough to fix the TmuxRunning devshell when
using pseudo because tmux does not preserve the environment that pseudo
requires.
(From OE-Core rev: 36fb9799d6a449d86acca3be354af56ad87c3151)
Signed-off-by: Tyler Hall <tylerwhall@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new standard for host distribution identification [1] is
/etc/os-release, and a number of newer distributions provide this file,
so add support for this in order to pick up more distributions.
Additionally, handle "rolling release" style distributions that don't
report a version number, e.g. Arch Linux.
With this change we can identify the most common distributions, so this
should satisfy [YOCTO #4271]. Note that this doesn't imply support for
these distros as build hosts, just that we can identify them.
[1] http://www.freedesktop.org/software/systemd/man/os-release.html
(From OE-Core rev: bff50b747cde04007ead65dde4207b16a8e1bf08)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to make a devshell using tmux can fail because "tmux new"
expects a single command, not a series of arguments. It does, however,
split strings in a suitable way. So you can quote the command.
The failure mode is particularly arcane, in that you end up
with a message like:
ERROR: Unable to spawn terminal auto: \
Execution of 'pseudo /bin/bash' failed with exit code 1:
usage: new-session [-d] [-n window-name] [-s session-name] \
[-t target-session] [command]
which is confusing because there's no "new-session" anywhere in
sight (that's actually "tmux new"), and because what failed to execute
wasn't either pseudo or bash.
(From OE-Core rev: f8ed7446755eeb88191e16749350efa1e7e6197c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of reporting an error when bb cannot be imported, skip the test
instead. This makes it a lot easier to iterate a test suite when we don't care
about this particular test.
(From OE-Core rev: c4a5bd810ca92d57c334113c528bd1d233b3eac4)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a helper function that returns just the first <num_parts> of <version>,
split by periods. For example, trim_version("1.2.3", 2) will return "1.2".
This should help reduce the spread of numerous copies of this idea across
classes and recipes.
(From OE-Core rev: 17a12e3c62807a7d60fcbf0aa4fd9cf4a739a204)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The error is not global since we don't use "from os import *", so it
should be os.error.
[YOCTO #4489]
(From OE-Core rev: ac0e2781c2723257bd380cac170d4c8b97e36747)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BUILDHISTORY_IMAGE_FILES variable specifying a space-separated
list of files within an image to copy into buildhistory, so that changes
to them can be tracked. Typically this would be used for configuration
files, and by default this includes /etc/passwd and /etc/group, but the
user is free to extend this list by setting the variable in local.conf.
Implements [YOCTO #4154].
(From OE-Core rev: ed6bb6e3db518082d3a9c45d548bc1339be2c5ca)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are various bits of cruft that have built up around our file accesses. This patch
cleans some of them up, specifically:
* Remove pointless "from __builtin__ import file"
* Use open(), not file()
* Wrap file usage in a with container to ensure files are closed
* Add missing .close() calls in some cases
(From OE-Core rev: a43e0a8ecd0441131e929daf998c3cd454d9c8f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I mistakenly thought subprocess had getcmdstatus in python 2. It doesn't so lets
add a wrapper and have this work in both worlds.
(From OE-Core rev: 2253e9f12734c6e6aa489942b5e4628eca1fa29d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The has_key() attribute has been removed in python 3 since there is better
syntax available. Use the improved syntax.
(From OE-Core rev: 3dff13793e875ff58cc38c4a960caca9b6969843)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commands module is removed in python3. Use the subprocess module instead
and the pipes module to replace the mkargs usage.
(From OE-Core rev: e2e1dcd74bc45381baccf507c0309dd792229afe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
StandardError is removed in python3, replace with Exception class instead.
(From OE-Core rev: a37ae30b9766df346ca57755530024a0b7d5f86b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python 3 print is a function call. In some cases bb.note is a more
appropriate call to make.
(From OE-Core rev: 754874e68d1037c76fc4cc7d35d3fa12946d00f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some left over tab characters in the python functions. This
removes them and resolves python 3 errors.
(From OE-Core rev: fafeb381c48291fa65c634c01c244843c8d7fad3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update older code to use modern exception handling syntax which
is the form accepted by python 3.
(From OE-Core rev: b010501cd089e649a68f683be0cf4d0aac90fbe3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The change to use copyhardlinktree in some of the sstate code instead of
copytree exposed a race condition. This is due to cp failing if it finds
a directory doesn't exist yet some other process creates it while cp was
trying to create it itself. tar doesn't error in this case.
To fix this we need to create the directory structure with tar, then
use cp to hardlink the files. Messy but probably worth doing.
I also took the opportunity to remove src_bak since the code is neater
without it.
(From OE-Core rev: 2f954a9a6932f1e6c564e7e7aacaac628a75eed7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make hard links for staging files instead of copy to save the disk space
(3G will be saved for a core-image-sato build), and it doesn't affect
much on the build time.
The following directories are affected:
1) The sysroot
2) The DEPLOY_DIR
3) The pkgdata
[YOCTO #4372]
(From OE-Core rev: 5853e0f482b22258c909268fe71673a29e31989b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds two new Terminal classes. It's separated into two, so that opening
a split inside a tmux window is preferred to the other terminal types, but
opening a tmux session is prioritized only slightly higher than screen.
- tmuxrunning: Open a new pane in the current running tmux window. Requires
that the TMUX variable be added to the env whitelist to use it.
- tmux: Open a new tmux session
(From OE-Core rev: 10f64d202ceb230c3c79e09dce182ffce94d1117)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For multilib and other uses of classextend, we don't want any
dependencies on kernel packages to be extended since there should only
be one kernel variant.
Fixes [YOCTO #2918] (where kernel-dev was being extended.)
(From OE-Core rev: b684c0f0d5d93f5147dee79951647eb3ddf4c840)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example if PACKAGES is "foo foo-data foo-dev foo-doc", this will return
"foo-data".
(From OE-Core rev: 3115187e468398a8c1edaf3e5369a2d10fb112f4)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds two new Terminal classes. It's separated into two, so that opening
a split inside a tmux window is preferred to the other terminal types, but
opening a tmux session is prioritized only slightly higher than screen.
- tmuxrunning: Open a new pane in the current running tmux window. Requires
that the TMUX variable be added to the env whitelist to use it.
- tmux: Open a new tmux session
(From OE-Core rev: 31c58d584f838738a6b6258b87b1c7e6ca173086)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of running lsb_release -a, a lsb_release -ir will be run.
This will prevent issue with distros that don't have all the needed
info in /etc/lsb-release file, in which case lsb_release won't generate
an error code.
Partial fix for [YOCTO #4071]
(From OE-Core rev: 79a2252545ab50c79e00e02c328191c1163f917d)
Signed-off-by: Cristian Iorga <cristian.iorga@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases, /etc/lsb-release file is used to extract
info about poky build host machine. But the strings are
not stripped of end of line special characters. As such,
when this info is concatenated and used as a directory
entry in sstate_cache, this is an issue.
Usually, this issue is masked by the fact that distro
related info is extracted from the output of lsb_release
command. In case of Yocto Linux, running "lsb_release -a"
will give an error code because CODENAME info is not present.
As such, bitbake will extract the info from /etc/lsb-release,
running into the above issue.
Consequence is that building under BA will crash.
Partial fix for [YOCTO #4071]
(From OE-Core rev: 5d0839bef631dceb4395fcf204779a76966a1061)
Signed-off-by: Cristian Iorga <cristian.iorga@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, various standard library operations like os.walk(),
os.path.isdir() and os.path.islink() each call stat or lstat which
involves a syscall into the kernel. There is no caching since they could
conceivably have changed on disk. The result is that for something like
the do_package task of the kernel we're spending over two minutes making
868,000 individual stat calls for 23,000 files. This is suboptimal.
This patch adds lib/oe/cachedpath.py which are a set of replacement
functions for these operations which use cached stat data rather than
hitting the kernel each time. It gives a nice performance improvement
halving the build time of the kernel do_package.
(From OE-Core rev: 556dee0c4d6d8a87c0cddbd2f60fe5917d009f18)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If lsb_release is not installed (as it may not be on headless/minimal
installations on distros whose LSB package has a long list of
dependencies) we need to gather the information directly from files in
/etc.
Fixes [YOCTO #4012].
(From OE-Core rev: 106a7bcdb5ad9956f1d78f508408bfbcf7ff5120)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's too easy to cause rebuilds because the DISTRO_FEATURES have changed in
meaningless ways (such as re-ordering or duplicate items). Help stop this by
checking if the feature to be back-filled is already present.
(From OE-Core rev: 63c7192119d54b92d908441109ed4e4fff761cba)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some use cases in OE operate on symlinks which dangling path components.
Assume that these are directories instead of raising ENOENT.
(From OE-Core rev: a96e2c84f24c15b77ee1fbc1f998b8b4796b8664)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various parts of the buildsystem have to work with symlinks. Resolving
them is not trivial because they are always relative to a sysroot
directory.
Patch adds a function which returns the destination of a symlink by
assuming a given path as the / toplevel directory. A testsuite was
added too.
(From OE-Core rev: 76e0bd7f8e3a3bd052a6e329f88e2d8099e899c4)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Report changes to RPROVIDES, RREPLACES, and RCONFLICTS. As RSUGGESTS
isn't widely used and isn't of huge concern if it changes, it is not
reported by default.
Implements [YOCTO #3391].
(From OE-Core rev: d20011571db96da79a8a0e056c6cef8e4c083608)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
