Commit Graph

648 Commits

Author SHA1 Message Date
Richard Purdie
bfd0a39bdf classes: Drop now unneeded update_data calls
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.

(From OE-Core rev: 8de0c5d3bd01919e2bf0394f9c485936d6098cec)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 09:29:55 -08:00
Richard Purdie
eac8f9b986 lib/oe/path: Add replace_absolute_symlinks function
We need a function to walk a directory and replace absolute symlinks with
relative ones. Add such a function to the path module of lib/oe.

It does this relative to the directory being walked for files.

(From OE-Core rev: 15376e5ff35367c1b40941d10e7b19302058a53e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-09 10:52:03 +00:00
Ming Liu
3c89b61533 meta: remove remaining True option to getVarFlag calls
This is a complementary fix to commit 2dea9e49:
[ meta: remove True option to getVarFlag calls ]

it intended to remove all True option to getVarFlag calls, but there are
still some remaining.

Search made with the following regex: getVarFlag ?\((.*), True\)

(From OE-Core rev: 5b53b34dcb64dc65ee651cbf78cbbd9105dc4f4c)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:02 +00:00
Richard Purdie
6ab4c5033a classes/oeqa: Replace subprocess.check_call() with check_output()
If you use subprocess.check_output() the traceback will contain the output
when the command fails which is very useful for debugging. There is no
good reason not to use this everywhere.

(From OE-Core rev: ad750dd1cc9d789abe723daddd098ce41d8547f5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 23:30:57 +00:00
Chen Qi
2a64701acf package_manager: default to have scriptlet output captured in log
We need to have scriptlet output captured in log. If we don't do so,
some useful information from scriptlets (especially postinstall script)
would be missing. In case a script has a warning message but it does not
necessarily have to fail, the message should be captured.

Opkg has already done that. Change for rpm and dpkg so that scriptlet
output is captured and no warning message is missing.

(From OE-Core rev: 0e52e0c619e02327602d83999a61d978d3a9240e)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:22 +00:00
Aníbal Limón
4054b25d5e oe/data: Add export2json function
The export2json function export the variables contained in
the data store to JSON format, the main usage for now will be
to provide test data to QA framework.

(From OE-Core rev: 57c7bf68ed66a56601e1431bb2db750c5742b5ce)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Richard Purdie
eeb30764e9 lib/oe/utils: Add build_depends_string function
This is useful when manipulating depends strings for task [depends]
flags and is slightly easier to parse than some inline python.

(From OE-Core rev: 7b05ea65a8db8a27b2a5579675775ee34ceb63c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:49 +00:00
Mariano Lopez
55aa669750 lib/oe/package_manager.py: Fix extract for ipk and deb
With the move to use lists instead of strings in subprocess
calls, package extraction was broken for ipk and deb. This
fixes this issue.

(From OE-Core rev: 3e1d8e5c7ac3238eda85ee95dfef044bef2a6411)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:20 +00:00
André Draszik
08fd7c4fc2 lib/oe/rootfs: reliably handle alternative symlinks
When removing unneeded packages from a (read-only) rootfs
during rootfs creation, alternative symlinks from those
packages may or may not be removed.

The reason is as follows:

update-alternatives(-native) is used during package
installation as part of the image creation. It uses
a database which contains entries for all the
alternative symlinks possible, and the -native version
uses the target's database by means of $OPKG_OFFLINE_ROOT,
i.e. the rootfs we're in the process of creating.

Once the rootfs has been created, OE removes certain
packages because we have a read-only rootfs - in
particular ROOTFS_RO_UNNEEDED which includes
VIRTUAL-RUNTIME_update-alternatives, i.e. the
update-alternatives. Recently, a change was made in
OE, where uninstallation of update-alternatives from the
rootfs causes removal of its database, too, to save space
(700KiB (uncompressed) in a busybox system)
  b24a63d71b517af701dfedbc7f7b541d25af708f
  http://git.openembedded.org/openembedded-core/commit/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb?id=b24a63d71b517af701dfedbc7f7b541d25af708f

Following from that, if update-alternatives is removed
from the target file system, update-alternatives-native
has no database anymore, meaning it can't manage any of
the alternative symlinks anymore.

Because the order of packages to uninstall is
non-deterministic, and update-alternatives could well
be removed before any packages that use the mechanism
provided, sometimes the extra symlinks are removed,
sometimes not.

By sorting the list of packages to be removed such that
update-alternatives is removed last, we can ensure that
that tings work reliably. (Certainly opkg seems to
uninstall packages in the order given on the command
line.)

[YOCTO #10916]

(From OE-Core rev: 5263dd3eac9d9fbdb7ef654d0cd532c192baed16)

Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:14 +00:00
Ross Burton
1800b1ba7a rootfs: don't put /usr/lib/ssl and /etc into debugfs
The /etc and /usr/lib/ssl directories were only put into the opkg-generated
debugfs because of a bug in opkg which means that a conffile has to exist if
we're running 'opkg status'.  This is now fixed, so the workaround can be
reverted.

(From OE-Core rev: 7267b1f6fa25e290eac070263355aa7f30b2ebcb)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:13 +00:00
Jianxun Zhang
c782cb40d7 lib/oe.sstatesig: make locked sig file consistent
Sort keys of dict 'types' prior to dumping, in order to have
identical output every time. This could make it a little easier
to diff these human-readable dumps.

(From OE-Core rev: 8abbaba1931e2cb2b87aa733aa9a3e8eb359b500)

Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Paul Eggleton
fff7563799 lib/oe/path: add warning comment about oe.path.remove() with wildcarded filenames
Add a warning in the doc comment for oe.path.remove() about using that
function on paths that may contain wildcards in the actual
file/directory names.

(From OE-Core rev: 18cc0965741102bccc62dfb32ed7753cdacbadc7)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Richard Purdie
022a8b58c8 meta/scripts: Various getVar/getVarFlag expansion parameter fixes
There were a few straggling expansion parameter removals left for
getVar/getVarFlag where the odd whitespace meant they were missed
on previous passes. There were also some plain broken ussages such
as:

d.getVar('ALTERNATIVE_TARGET', old_name, True)
path = d.getVar('PATH', d, True)
d.getVar('IMAGE_ROOTFS', 'True')

which I've corrected (they happend to work by luck).

(From OE-Core rev: 688f7a64917a5ce5cbe12f8e5da4d47e265d240f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Richard Purdie
d77a135e18 lib/oe/utils: Drop python2 compatibility code
We've moved to python3, we don't need this compatibility code which just makes
the code less readable.

(From OE-Core rev: 425afe2484707640ac71194885fdb263e95e9950)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 12:36:40 +00:00
Ross Burton
de43650c29 lib/oe/package_manager: bail if createrepo can't be found
If createrepo isn't found then the errors later are mysterious, so explicitly
check and error out early if it isn't there.

(From OE-Core rev: e09636bbb3ea8ec58984197fd9c691bb908efe00)

(From OE-Core rev: c87361fc886432a9db584712bf3e41ecd0541960)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-17 09:57:04 +00:00
Ross Burton
1969854ec3 utils: Always use datastore's PATH for host_gcc_version
BUILD_CC may reference something like ccache and expect this to come from
ccache-native, we at least have some selftests which assume this. Modify the
code to use PATH when runnig BUILD_CC to ensure the tests continue to work
as expected.

(From OE-Core rev: f3e753372baac43d0921186340cf260df056de20)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Joshua Lock
3c59b1bf93 meta: remove True option to getVarFlag calls
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.

Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)

(From OE-Core rev: 2dea9e490a98377010b3d4118d054814c317a735)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Joshua Lock
c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ed Bartosh
46e787b248 buildhistory-diff: report directory renames
The script detects directory renaming if two different
directories with the same set of files are added and removed.

[YOCTO #10691]

(From OE-Core rev: 944db779a9f45cbeeebc976c00da37a517eea237)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ed Bartosh
c21cec8488 uninative: rebuild uninative for gcc 4.8 and 4.9
Some c++ libraries fail to build if uninative is built
with gcc 5.x and host gcc version is either 4.8 or 4.9.

The issue should be solved by making separate uninative sstate
directory structure sstate-cache/universal-<gcc version> for host gcc
versions 4.8 and 4.9. This causes rebuilds of uninative if host gcc
is either 4.8 or 4.9 and it doesn't match gcc version used to build
uninative.

[YOCTO #10441]

(From OE-Core rev: d36f41e5658bbbb6080ee833027879c119edf3e0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ross Burton
d535aba974 oeqa: move lib/oe tests to oe-selftest
These tests don't get ran often (as demonstrated by the fact that some were not
ported to Python 3), so move them to oeqa/selftest so they get executed
frequently and can be extended easily.

[ YOCTO #7376 ]

(From OE-Core rev: 2001979ad41e6fdd5a37b0f90a96708f39c9df07)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Paul Eggleton
078ef4361d lib/oe/recipeutils: drop parse_recipe_simple()
This was intended to be used with tinfoil, but tinfoil now has its own
parse_recipe() method to do this which works properly in the memres
case.

(From OE-Core rev: cdfc6173cb06ca374b7d927442a0fdde8373ba48)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton
275b3fe5f4 classes/patch: move several functions to oe.patch
Move patch_path(), src_patches() and should_apply() to oe.patch, making
them easier to call from elsewhere (particularly across the
UI/server boundary).

(From OE-Core rev: 2724511e18810cc8082c1b028e3b7c8a8b5def56)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton
557382ab42 lib/oe/recipeutils: use cooker function instead of bb.providers
We now have a function in cooker itself that can do this lookup;
additionally, the rewritten tinfoil's cooker adapter has its own
implementation that can work remotely, so if we use it then this
function can work in that scenario as well.

(From OE-Core rev: 0a6a4be99c1e4ef3c0da53d63f18ad579545d6a8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Paul Eggleton
af867199a5 classes/image: suppress log_check mechanism for warnings/errors logged through BitBake
If you printed a warning through bb.warn() / bbwarn or an error through
bb.error() / bberror, this was also being picked up by our log_check
mechanism that was designed to pick up warnings and errors printed by
other programs used during do_rootfs. This meant you saw not only the
warning or error itself, you saw it a second time through log_check,
which is a bit ugly. Use the just-added BB_TASK_LOGGER to access the
logger and add a handler that we can use to find out if any warning or
error we find in the logs is one we should ignore as it has already been
printed.

Fixes [YOCTO #8223].

(From OE-Core rev: fb37304d27857df3c53c0867e81fbc8899b48089)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Paul Eggleton
13916a4fab lib/oe/rootfs: fix log_check warnings being printed twice with RPM packaging
We were calling _log_check() in the RPM-specific rootfs class as well as
in the base class; this is unnecessary and resulted in any errors/warnings
generated during the actual package installation time triggering two warnings
instead of one. Drop the call from RpmRootfs._create() to fix this.

(From OE-Core rev: 541c56d755ba0354297673e857628026ad9e4df2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Stephano Cetola
d8fbaebc48 package_manager: remove strings and migrate to direct arrays
When using subprocess call and check_output, it is better to use arrays
rather than strings when possible to avoid whitespace and quoting
problems.

[ YOCTO #9342 ]

(From OE-Core rev: b12cec9a5ef14ecb02be7feec65508cf5d65c795)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08 10:31:30 +00:00
Martin Vuille
b7deb1f046 terminal.py: Pass string instead of bytes to ExecutionError to avoid exception
Based on run() in bitbake/lib/bb/process.py, ExecutionError() expects strings
not bytes. Passing bytes results in a "TypeError: Can't convert 'bytes' object
to str implicitly" exception.

Fixes Bug 10729

(From OE-Core rev: 063b63d4d324c23322ac1b6b7c7928e725d7b968)

Signed-off-by: Martin Vuille <jpmv27@yahoo.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:08 +00:00
Joshua Lock
333d300fba lib/oe/lsb: better handle missing fields
Some rolling release distros, such as Arch Linux, don't include a
VERSION_ID field in their os-release file.

Change release_dict_osr() to better handle this optional field
being absent.

Further improve the resilience of the release_dict_*() methods by
always returning a dict and using dict.get() in distro_identifier()
to supply a default, empty string, value when then key is missing.

(From OE-Core rev: e36066dcc3b56cac1c695370ea178b566c0ebfd6)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Stephano Cetola
93e8db2224 devshell: list commands when throwing NoSupportedTerminals
When attempting to run devshell, if no terminal is available, the
error being thrown was not very specific. This adds a list of
commands that failed, informing the user of what they can install to
fix the error.

[ YOCTO #10472]

(From OE-Core rev: c077f4aab2fc956408d4ad45c4e2e2ea6e480624)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton
e3193b76e0 lib/oe/recipeutils: ignore archives by default in get_recipe_local_files()
By default, have get_recipe_local_files() not return any archive
files. This prevents a local tarball from being erroneously removed
from SRC_URI if you run "devtool modify" on a recipe followed by
"devtool update-recipe". It doesn't actually help you to directly
update the contents of such tarballs, but at least now it won't break
the recipe.

(From OE-Core rev: e9c418d4704c1bed4c5880e176e5288485f4f5a6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton
55a157f4e6 devtool: update-recipe: fix handling of compressed local patches
It is possible to use gzip or bzip2 to compress patches and still refer
to them in compressed form in the SRC_URI value within a recipe. If you
run "devtool modify" on such a recipe, make changes to the commit for
the patch and then run devtool update-recipe, we need to correctly
associate the commit back to the compressed patch file and re-compress
the patch, neither of which we were doing previously.

Additionally, add an oe-selftest test to ensure this doesn't regress in
future.

Fixes [YOCTO #8278].

(From OE-Core rev: e47d21624dfec6f71742b837e91da553f18a28c5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton
d316363b7b lib/oe/patch: fix handling of patches with no header
If a patch applied by a recipe has no header and we turn the recipe's
source into a git tree (when PATCHTOOL = "git" or when using devtool
extract / modify / upgrade), the commit message ends up consisting only
of the original filename marker ("%% original patch: filename.patch").
When we come to do turn the commits back into a set of patches in
extractPatches(), this first line ends up in the "Subject: " part of
the file, but we were ignoring it because the line didn't start with the
marker text. The end result was we weren't able to get the original
patch name. Strip off any "Subject [PATCH x/y]" part before looking for
the marker text to fix.

This caused "devtool modify openssl" followed by "devtool update-recipe
openssl" (without any changes in-between) to remove version-script.patch
because that patch has no header and we weren't able to determine the
original filename.

(From OE-Core rev: d9971f5dc8eb7de551fd6f5e058fd24770ef5d78)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:12 +00:00
Linus Wallgren
b91e47fccb lib/oe/package_manager: .deb pre/postinst args
The debian policy manual and MaintainerScripts wiki page states that the
postinst script is supposed to be called with the `configure` argument
at first install, likewise the preinst script is supposed to be called
with the `install` argument on first install.

https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html
https://wiki.debian.org/MaintainerScripts

(From OE-Core rev: 3d9c3aae54589794ce3484fa1b21d1af2bd32661)

Signed-off-by: Linus Wallgren <linus.wallgren@scypho.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:11 +00:00
Joshua Lock
81afedc905 lib/oe/lsb: attempt to ensure consistent distro id regardless of source
The LSB Distributor ID and os-release NAME differ for most of the
distributions tested by the Yocto Project (CentOS, Debian, Fedora,
openSUSE and Ubuntu) however for all but openSUSE the os-release ID
matches the LSB Distributor ID when both are lowered before
comparison.

Therefore, in order to improve the consistency of identification of
a distribution, switch to using the os-release ID and converting
the ID value to lowercase.

Table showing comparison of LSB Distributor ID to os-release fields NAME
and ID for current Yocto Project supported host distributions:

Distribution | Version | Distributor ID   | NAME             | ID       |
-------------------------------------------------------------------------
CentOS       | 7       | CentOS           | CentOS Linux     | centos   |
Debian       | 8       | Debian           | Debian GNU/Linux | debian   |
Fedora       | 23      | Fedora           | Fedora           | fedora   |
Fedora       | 24      | Fedora           | Fedora           | fedora   |
openSUSE     | 13.2    | openSUSE project | openSUSE         | opensuse |
openSUSE     | 42.1    | SUSE LINUX       | openSUSE Leap    | opensuse |
Ubuntu       | 14.04   | Ubuntu           | Ubuntu           | ubuntu   |
Ubuntu       | 16.04   | Ubuntu           | Ubuntu           | ubuntu   |

[YOCTO #10591]

(From OE-Core rev: 8689e5618d45c2119134ea64754430c06a93ea09)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Joshua Lock
42441ea481 lib/oe/lsb: prefer /etc/os-release for distribution data
os-release(5) is an increasingly standard source of operating system
identification and more likely to be present on modern OS deployments, i.e.
many container variants of common distros include os-release and not the
lsb_release tool.

Therefore we should favour parsing /etc/os-release in distro_identifier(),
try lsb_release when that fails and finally fall back on various distro
specific sources of OS identification.

(From OE-Core rev: fc4eddecddec68d03a985086fa32db40ad0c7bfc)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Joshua Lock
545f5f96d9 lib/oe/lsb: make the release dict keys consistent regardless of source
Rather than have the distro_identifier method look for different keys in
the dict depending on the source ensure that each function for retrieving
release data uses the same key names in the returned dict.

(From OE-Core rev: 2ddd6ddaf0c5ba14ae83347eba877ac9ef179c76)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Ross Burton
025f917cec lib/oe/qa: handle binaries with segments outside the first 4kb
The ELF parser was assuming that the segment tables are in the first 4kb of the
binary.  Whilst this generally appears to be the case, there have been instances
where the segment table is elsewhere (offset 2MB, in this sample I have).  Solve
this problem by mmap()ing the file instead.

Also clean up the code a little whilst chasing the problem.

(From OE-Core rev: a66660aa5bb709547ce0b65a4563e4217c3c3d9f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Joshua Lock
5193dfdb1f lib/oe/path: remove duplicate import
There's no need to import glob inside copyhardlinktree() as it's
already imported for the entire path module.

(From OE-Core rev: 42dc4695da136a15bebb7525b1da5c2722b10a28)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Ross Burton
4e6c5d8769 distro_check: MeeGo is long dead, compare against Clear Linux instead
Instead of checking against a file that represents a distribution that hasn't
existed for years, fetch package names for Clear Linux instead.

[ YOCTO #10601 ]

(From OE-Core rev: 006c4db0974c42ff0f6950dd24e61c008f801679)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Ross Burton
1655d55764 distro_check: partial rewrite to make it work again
This library suffered as part of the Python 2 to Python 3 migration and stopped
working entirely.

Fix all the migration problems such as files being treated as strings but opened
in binary mode, insufficient use of with on files, and so on.

Rewrite large amounts to be Pythonic instead of C-in-Python.

Update OpenSuse and Fedora URLs.

Fedora now splits the archive alphabetically so handle that.

[ YOCTO #10562 ]

(From OE-Core rev: 58de12eaaac9c60bb8fc84a3a965ef86d2a39ae0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:36 +00:00
Paul Eggleton
385f108377 lib/oe/recipeutils: print just filename in bbappend_recipe() if in temp dir
If you use devtool update-recipe with the --append option, and a "local"
(in oe-local-files) has been modified we copy it into the specified
destination layer. With the way the devtool update-recipe code works now
the source is always a temp directory, and printing paths from within
that is just confusing, so if the path starts with the temp directory
then just print the file name alone.

(From OE-Core rev: 61475f0267d40c618ebf36023d0b6414a25975cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:36 +00:00
Samuli Piippo
b18901d9bc package_manager.py: correctly remove all dependent packages
Do not use --force-depends when trying to remove all dependent packages,
as it removes only the selected package and not the dependent packages.

(From OE-Core rev: a82e8725902086dab785a0b14305927dae1e4e8d)

Signed-off-by: Samuli Piippo <samuli.piippo@qt.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:33 +00:00
Robert Yang
d31d1ad4e5 oe/copy_buildsystem.py: dereference symlink
When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file

symA will be invalid fater copied, it would be invalid from build time
if it points to a relative path, and would be invalid after extracted
the sdk if it points to a absolute py. Dereference symlink when copy
will fix the problem.

Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
   https://bugs.python.org/issue21697
   And Ubunutu 1404 doesn't upgrade python3 to fix the problem.

2) shutil.copytree(symlinks=False) raises errors when there is a invalid
   symlink, and tar just prints a warning, tar is preferred here since
   the real world is unpredicatable

3) tar is faster than shutil.copytree() as said by oe.path.copytree()

So use tar to copy.

(From OE-Core rev: f4d70bb0882eec4fb46cd942f2796fad57c72982)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:32 +00:00
Davis, Michael
e66f5eabed terminal.py: Add compatiblity for konsole 16.08.1
Konsole has dropped support for the nofork flag.  It has been replaced with the seperate flag.

(From OE-Core rev: f0b193b63d4c468c3aa58e15ef5a991e04b9b9a2)

Signed-off-by: Michael Davis <michael.davis@essvote.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-28 16:15:18 +01:00
Ross Burton
84198174bf lib/oe/qa: add ELF machine to string function
Add a function (and test suite) to turn the ELF machine field (e_machine) into a
string, so we can tell the user "x86-64" instead of 0x3E.

(From OE-Core rev: 72336003741fb16a7ecdd6b753eae56310413ff7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-11 22:19:21 +01:00
Peter Kjellerstedt
d067b0e1e4 package_manager.py: Allow multiple regexps in PACKAGE_EXCLUDE_COMPLEMENTARY
The PACKAGE_EXCLUDE_COMPLEMENTARY variable can currently only contain
one regular expression. This makes it hard to add to it from different
configuration files and recipes.

Allowing it to contain multiple, whitespace separated regular
expressions should be backwards compatible as it is assumed that
whitespace is not used in package names and thus is not used in any
existing instances of the variable.

After this change, the following three examples should be equivalent:

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo|bar"

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo bar"

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo"
  PACKAGE_EXCLUDE_COMPLEMENTARY += "bar"

(From OE-Core rev: a5f7e98a94e96d40b1276c85249619aa8d7be847)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00
Peter Kjellerstedt
2334201960 package_manager.py: Allow a leading - in PACKAGE_EXCLUDE_COMPLEMENTARY
This allows a regular expression specified in
PACKAGE_EXCLUDE_COMPLEMENTARY to have a leading dash. Without this,
the dash was treated by oe-pkgdata-util as the beginning of a command
line argument. E.g., if PACKAGE_EXCLUDE_COMPLEMENTARY = "-foo$", it
resulted in an error like:

  ERROR: <imagename>-1.0-r0 do_populate_sdk: Could not compute
  complementary packages list. Command '<topdir>/scripts/oe-pkgdata-util -p
  <builddir>/tmp/sysroots/<machine>/pkgdata glob
  <workdir>/installed_pkgs.txt *-dev *-dbg -x -foo$' returned 2:
  ERROR: argument -x/--exclude: expected one argument
  usage: oe-pkgdata-util glob [-h] [-x EXCLUDE] pkglistfile glob [glob ...]

(From OE-Core rev: ac4ca41d3a27356d46c0c39053e74d3519b24c44)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00
Paul Eggleton
7caf628e83 lib/oe/patch: improve accuracy of patch header extraction
When PATCHTOOL = "git", if we need to manually apply a patch and then
commit it (i.e. when git am doesn't work) we try to extract the author /
date / shortlog from the patch header. Make the following improvements
to that extraction process:

* If there's no explicit Subject: but the first line is followed by a
  blank line, isn't an Upstream-Status: or Index: marker and isn't too
  long, then assume it's good enough to be the shortlog. This avoids
  having too many patches with "Upgrade to version x.y" as the shortlog
  (since that is often when patches get added).
* Add --follow to the command we use to find the commit that added the
  patch, so we mostly get the commit that added the patch rather than
  getting stuck on upgrade commits that last moved/renamed the patch
* Populate the date from the commit that added the patch if we were able
  to get the author but not the date from the patch (otherwise you get
  today's date which is less useful).

(From OE-Core rev: 896cfb10ec166a677cbb3b4f8643719cabeb7663)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-24 07:30:09 +01:00
Paul Eggleton
f6928407ed lib/oe/patch: exclude "From <hash>" from commit message when PATCHTOOL is "git"
If you leave "From <hash>" lines in the commit message it can actually
break git rebase because it tries to interpret the line in the context
of the current repository, and if the hash is invalid then a rebase
will blow up with:

  fatal: git cat-file: could not get object info

or in newer git versions:

  error: unable to find <hash>
  fatal: git cat-file <hash>: bad file

(I hit this when I tried to do a devtool upgrade on openssl to 1.0.2i
the first time I did "git rebase --skip")

(From OE-Core rev: 19a6b18ac23cb2d7bb89203f774b2bee7f0cb03c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-24 07:30:09 +01:00