If we're to completely replace the standard SDK with the extensible SDK,
we need to be able to provide the standard toolchain on install without
doing anything other than installing it, so that you can install the SDK
and then point your IDE at it. This is particularly applicable to the
minimal SDK which normally installs nothing by default.
NOTE: enabling this option currently adds ~280MB to the size of the
minimal eSDK installer. If we need to reduce this further we would have
to look at adjusting the dependencies and/or the sstate_depvalid()
function in sstate.bbclass which eliminates dependencies, or look at
reducing the size of the artifacts themselves.
Implements [YOCTO #9751].
(From OE-Core rev: ed0d8ed72370df694f720cc13897493478dc1de9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
tar's --no-recursion flag only applies to files mentioned after the
flag, which made it a no-op in this invocation of tar, because it was at
the end of the command line.
This is simple to verify with GNU tar 1.29:
| $ mkdir foo
| $ mkdir foo/dir
| $ touch foo/dir/file
| $ tar -cf - foo --no-recursion | tar t
| foo/
| foo/dir/
| foo/dir/file
| $ tar -cf - --no-recursion foo | tar t
| foo/
Modify the code so that it actually does what the comment says by moving
the flag in front of the --files-from argument.
(From OE-Core rev: d45f5e71fef5ffbd4408f69c5c179dc71a3eb452)
Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oe.utils.getstatusoutput() is a wrapper for subprocess.getstatusoutput() which
uses Universal Newlines, so the output is a str() not bytes().
(From OE-Core rev: ce24d4c3632b71939ad198268a900ee823a89b27)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is possible in an attempt only install, that everything listed is not
available to be installed. This will have the effect of clearing the
package list. However, we only check for an empty package list at
the beginning of the function. We need to also check before running the
install, otherwise we can fail due to 'error: no package(s) given".
(From OE-Core rev: 9ae6a2830dacb3c335754a6da91bd5cc30546b31)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using PR service the buildhistory-diff output contains a lot of
PKGR changes: In practice the mass of PKGR updates hide other important
changes as they often account for 80% of all changes.
Skipped incremental and decremental changes of PKGR versions to reduce
amount of the script output. All changes are still included in the
output if script is run with -a/--report-all command line option.
[YOCTO #9755]
(From OE-Core rev: a343788b11f6c4f92ae8d2035fe8cb54f922227e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This task runs all functions in IMAGE_QA_COMMANDS after the image
construction has completed in order to validate the resulting image.
Image sanity checks should either be Python functions which raise
bb.build.FuncFailed on failure or shell functions with return a
non-zero exit code.
Python functions may instead raise an oe.utils.ImageQAFailed
Exception which takes an extra argument, a description of the
failure.
python image_check_python_ok () {
if True:
raise bb.build.FuncFailed('This check always fails')
else:
bb.note("Nothing to see here")
}
image_check_shell_ok () {
if true
exit 1
else
exit 0
fi
}
[YOCTO #9448]
(From OE-Core rev: c9bef2ecf1a30159d11781184829f41844a58c13)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a subcommand which will "finish" the work on a recipe. This is
effectively the same as update-recipe followed by reset, except that the
destination layer is required and it will do the right thing depending
on the situation - if the recipe file itself is in the workspace (e.g.
as a result of devtool add), the recipe file and any associated files
will be moved to the destination layer; or if the destination layer is
the one containing the original recipe, the recipe will be overwritten;
otherwise a bbappend will be created to apply the changes. In all cases
the layer path can be loosely specified - it could be a layer name, or
a partial path into a recipe. In the case of upgrades, devtool finish
will also take care of deleting the old recipe.
This avoids the user having to figure out the correct actions when
they're done - they just do "devtool finish recipename layername" and
it saves their work and then removes the recipe from the workspace.
Addresses [YOCTO #8594].
(From OE-Core rev: fa550fcb9333d59b28fc0e4aebde888831410f5c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This provides us with the information we need to remove the original
version recipe and associated files when running "devtool finish" after
"devtool upgrade".
(From OE-Core rev: 92eb42c347af919cd9f8739515fdf806c12b5ba8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Allow the function to be called with the base layer path (in which
case it will just return the same path)
* Ensure that the function doesn't recurse indefinitely if it's called
on a file that's not inside a layer
* Correct the doc comment for accuracy
(From OE-Core rev: 912026d85c2f535be2f60c45979162ea25c7f356)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During a parallel build it's possible for unrelated shlib files to be removed if
the recipe they came from is about to be rebuilt. They can't be involved in the
dependency chains as otherwise they wouldn't be removed, so just silently handle
files disappearing.
[ YOCTO #8555 ]
(From OE-Core rev: 1e355da3fda742c78d99ddd2ee5caa9df52f92e1)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With Python 3, the encoding of a file is significant; several recipes in
OE-Core have patches which are not fully utf-8 decodable e.g. man,
lrzsz, and gstreamer1.0-libav, leading to errors when using devtool's
modify, upgrade or extract subcommands on these recipes. To work around
this, try reading the patch file as utf-8 first and if that fails try
latin-1 before giving up.
(From OE-Core rev: 7f4d7a6f51569954e204f110827a8ce256bcdc68)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the new pid monitoring code we have for recent versions of
gnome-terminal we can just drop the --disable-factory code now since
the other solution handles this case as well.
(From OE-Core rev: ed4957c444a2982c19e2f1f96d9afb2a992c1daf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently gnome-terminal just returns straight away, opening a terminal in a new
separate process we have no insight into. For patch resolution, this leads to
spawning many different terminal windows, for pydevshell, it just flashes a window
up and then closes.
We need to block until the command completes but gnome-terminal gives us no way
to do this. We therefore write the pid to a file using a "phonehome" wrapper
script, then monitor the pid until it exits.
[YOCTO #7254]
(also fixing do_devpyshell)
(From OE-Core rev: 76e8ab47c936674b8bb9bf1c48de53b30f5bf74a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was a copy-and-paste of subprocess.check_output() from when we supported
Python <2.7, so simply delete it and use subprocess.check_output() instead.
(From OE-Core rev: b1f2d9ed8d4dc89c9e669f43f546463ccc2a76b9)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're building the extensible SDK we don't need to see the "Writing
locked sigs" message; it's only necessary when the user explicitly runs
bitbake -S none <target>.
(From OE-Core rev: 440a351ee13920045c9d3e51882908f7b3f01d35)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new task progress functionality to report progress during
do_rootfs. This is a little coarse and ideally we would have some
progress within the installation section, but it's better than
nothing.
(From OE-Core rev: 370f08d434480c1790950e40db8f7687da78cb14)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In RpmPM:insert_feeds_uris, the paths are kept in sets, which are unordered,
but they are later used to set the priority for the Smart channels, so
unexpected results could occur. Change the sets to lists and use the same
code as in create_configs() to add items to the list, rather than the set
operators.
[YOCTO #9717]
(From OE-Core rev: ce4137f4bb955207fede0c4ef338835d9a461f59)
Signed-off-by: Bill Randle <william.c.randle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Specify tmpdir for opkg via '-t' option so that opkg does not use
the default 'TMPDIR' which usually is '/tmp' on build host.
This would solve race problems like below.
sh: /tmp/opkg-rOG6Tl/opkg-intercept-iPoEp5/depmod: Permission denied
(From OE-Core rev: 5665e284b7e7ee071201664dc50a3412bd4ac8cd)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Order is not preserved in dict() and this code depends on the order of
these lists of package architectures used when multilibs are enabled.
This caused 'random' breakage where sometimes the correct order was present
and sometimes it wasn't.
Use collections.OrderedDict() to avoid this problem.
Kudos to Bill Randle and Alejandro Hernandez who did most of the work debugging
this, I simply took the problem they identified and wrote a patch to fix it.
This unblocks the M1 build but this code needs auditing as there are clearly
other ordering issues (e.g. the set() usage).
[YOCTO #9717]
(From OE-Core rev: 61a33582dfc964d612d20d34734a787d873e312c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
create_socket: Use urllib because urllib2 is now urllib in python3
and proxies as argument are deprecated so export them in the environ
instead.
get_links_from_url: Change usage of sgmllib for parsing HTML because
is deprecated in python 3, use instead bs4 that is already imported
in the bitbake tree.
[YOCTO #9744]
(From OE-Core rev: ee26ecf58277560459dd01992bb3f486f92c1531)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On python 3, bytes variable types must be decoded if these are intended to be
used as strings, otherwise we get the following error exception:
TypeError: Type str doesn't support the buffer API
(From OE-Core rev: b950539c911b7945d652b05616164828e711ac7f)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
string.rstrip() doesn't exist in Python 3, so use the .rstrip method on the
object itself instead.
(From OE-Core rev: 554719e3f65e468f89a96d4869766d3ff7a8012e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The read method of the data_stream File object now returns bytes,
not a str, so we must decode it.
(From OE-Core rev: cfae302c4996c49a8754497ea9f13f8331d6975d)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bytes type variables are threat different as strings variables in python3,
so convert bytes to strings. This was found when using the
script buildhistory-diff, where the interpreter was yielding this error
TypeError: Type str doesn't support the buffer API
(From OE-Core rev: 3064d36186b47954eb94095217f7bb37e3fce651)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.
(From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved call of decode('utf-8') as close as possible to
call of subprocess API to avoid calling it in a lot of
other places.
Decoded binary data to utf-8 where appropriate to fix devtool
and recipetool tests in python 3 environment.
(From OE-Core rev: 30d02e2aa2d42fdf76271234b2dc9f37bc46b250)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
xrange() no longer exists in python 3, use range()
(From OE-Core rev: d022b4335100612d6596cc4c4956cb98ed5873cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.
(From OE-Core rev: caebd862bac7eed725e0f0321bf50793671b5312)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.
(From OE-Core rev: 2476bdcbef591e951d11d57d53f1315848758571)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.
(From OE-Core rev: bb4685af1bffe17b3aa92a6d21398f38a44ea874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The syntax for octal values changed in python3, adapt to it.
(From OE-Core rev: 737a095fcde773a36e0fee1f27b74aaa88062386)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some more variables in appropriate places in recipe_progression such
that the patch_recipe_* functions are able to insert variables in the
right place within a recipe.
(From OE-Core rev: c33ba6cc5d14b1da96f6d906836c50e0346dcf06)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function was assuming that what you wanted was that output lines
had trailing newline characters. If you're just outputting each line
verbatim to a text file then that's fine, but sometimes you start with
the assumption that the lines don't have trailing newlines; thus we
shouldn't allow for the possibility that the caller doesn't want them
and add a parameter to control it.
(From OE-Core rev: fb2bb509ff5c7bd71b41a1dcba3b1bff1d18cf5d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Split out a function from patch_recipe_file() that takes just the lines
as input so we can edit recipe lines in memory. This will be used within
recipetool to ensure we insert new values in the right place.
(From OE-Core rev: d780642f950fb3a9699f466a405a2710d870dd08)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having the package path with all the other package info allows to
reuse more code and have this information outside the package manager,
without additional processing.
[YOCTO #8536]
(From OE-Core rev: 343f762792cbfccffaf3aa901289f9bb0f8cef3d)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update older exception syntax to modern one required by python 3.
Compatible with python 2.7.
(From OE-Core rev: d13f0ac614f1d1e2ef2c8ddc71cbfcf76a8dc3f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
file() API doesn't exist in python 3, convert to open(). Also handle
some cases where files aren't closed. Compatible with python 2.7.
[Contributions from Ed and Richard]
(From OE-Core rev: 0f4ec13e11bb8abe21aba2a28547dfb9372bc377)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used print function instead of print statement to make
the code work in python 3.
[Changes from both Ed and Richard]
(From OE-Core rev: ced1995694c394d92cb82fb9c25a33dc027a3b69)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 20ea6d274bb0a9a5addb111f32793de49b907865, debugfs generation
for images using opkg, which included openssl was fixed.
However, that broke the generation of the opkg-based images, that lacks
openssl. The error is a python stack trace, showing that shutil.copytree
tries to copy a non-existing directory.
This relates to [YOCTO #9040].
(From OE-Core rev: f6b0b260ce18a30d04edfb0afb7942b9f9a5480b)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a common _log_check_common() function (based on the old
_log_check_warn() function) to implement the logic for both
_log_check_warn() and _log_check_error().
The main benefit of this is that now all error messages will be
reported again, not just the first one found. Additionally the output
will now look the same for both error and warning messages.
This removes the context for the error messages. However, since there
was no indication in the output that some of the lines were context,
they were more confusing than helping.
(From OE-Core rev: dd73dcac36b80b3b886a2e9bf575b91c4f60d039)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For each warning found in the log, _log_check_warn() would output a
line stating that it had found a warning, then the actual warning and
finally an empty line. This is quite excessive when there are many
warnings in the log.
With this change the output is instead a line stating how many
warnings were found, followed by the warnings. This makes the output
much more compact and actually much more readable.
(From OE-Core rev: d6e3477749b1d09d40a773e0ac857a24d5851984)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will make _log_check_warn() exclude the same lines as
_log_check_error() does.
(From OE-Core rev: 85f64c68278f797c6f73f002f63d7f46fe80aef4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before there were three different ways to exclude a line from being
searched for error messages in _log_check_error(). Now there is only
one: an array of regular expressions. This should make it easy to add
more excludes if nedded.
(From OE-Core rev: 321df88088fbfa657b61b2bae32751f03daec46f)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 0387d095 lines with "NOTE:" in them were excluded from the
log check for warnings. However, those lines were only there in the
first place since the regular expression that is used to find warning
messages explicitly included those lines...
(From OE-Core rev: 865ab39f18a52ed84217df56d0e65113e2894d02)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fact that this function was overridden in the RpmRootfs class
seems to have led to a number of misstakes when changes have been made
to the base function in the Rootfs class. E.g., this change will
properly solve ticket 7789, which was supposedly solved in 38871dc0,
but that change had no effect in practice as the log_check_regex that
was modified for RpmRootfs class was not used by the RpmRootfs version
of _log_check_error()...
The only thing _log_check_error() in RpmRootfs did that the base
function in Rootfs did not do was to skip lines in the log that begin
with a + sign. This has now been moved to the base function instead.
[YOCTO #7789]
(From OE-Core rev: 1eb0a46502fca4b2ee30ccd2508f4e21a40c25ca)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The reverted commit added a warning with the log line that triggered
the log check for error messages before the warning that states that
an error has been found in the log. However, the error line is output
by the call to bb.fatal() that follows immediately after the original
warning, which makes it redundant. Additionaly, having two warnings
contradicts the intent of commit 8dfdd329 where the log warnings were
tidied up.
This reverts commit f9cf31525fc885e1a0f65bd55654631257f87078.
(From OE-Core rev: e509a83c8eb5a572aad7e7451d019fa2469d70c3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
