This patch makes the necessary changes for using the targetcontrol.py module
so that one can run the same tests on a qemu instance or a remote machine
based on the value of TEST_TARGET variable: "qemu" or "simpleremote".
The default value is "qemu" which starts a qemu instance and it's the
with what we currently have.
With "simpleremote", the remote machine must be up with network and ssh
and you need to set TEST_TARGET_IP with the IP address of the remote machine
(it can still be a qemu instance that was manually started).
Basically testimage.bbclass now does something along the lines of:
- load tests -> deploy (prepare) / start target -> run tests.
There were a couple of changes necessary for tests and
also some cleanups/renames that were needed to adjust this change. (use
ip everywhere when refering to target and server_ip when refering to host/build machine)
Also two unnecessary and unsed methods were dropped from sshcontrol.
[ YOCTO #5554 ]
(From OE-Core rev: a7820350fa3271d78ed7476e02f4aef593be1125)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't normally perform any operations (such as "git pull") that
trigger "git gc --auto", thus garbage collection never happens which
means performance of accessing the repository degrades noticeably over
time. Add an explicit "git gc --auto" to clean things up when needed.
Thanks to Elijah Newren and Ross Burton for suggesting this.
(From OE-Core rev: 4a45a999e0ad2e99581428a5a6d34f483c00544f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Disable several python functions if not parsing within the worker
context. This avoids executing expensive operations while parsing
recipes (which is unnecessary).
(Thanks to Richard Purdie for pointing out the issue and suggesting the
workaround.)
(From OE-Core rev: 540a2a30be21c3eca4323efbe91e7dcfc31a4c97)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a function added to SSTATEPOSTINSTFUNCS and read the necessary
information out of pkgdata, instead of using a function executed
during do_package that reads the data directly. This has two benefits:
* The package info collection will now work when the package content is
restored from shared state
* Adding/removing the inherit of buildhistory will no longer change the
do_package signatures and force re-execution of that function for
every recipe.
Fixes [YOCTO #5358]
(From OE-Core rev: cd7f7efcd5f297d876823b8f579ecefb9542b089)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add PACKAGE_EXCLUDE and NO_RECOMMENDATIONS to the info we track for
images, since these can change what ends up in the image.
(From OE-Core rev: a10189366f180b87f5be20b66834b7e7a9bb8c12)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should have been removed when the implementation was rewritten in
OE-Core commit 2179db89436d719635f858c87d1e098696bead2a. The collected
values weren't being used anywhere since then.
(From OE-Core rev: cbc23a87c1897b7fda40f452dd36acb0bca3d197)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These are important parts of the version for every package, so we should
include them in PKGDATA just as we include PV/PR/PKGV/PKGR.
(From OE-Core rev: 5ceed97ba02a698f1c260c3f56cdf2cc156e6d8b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were using "du -sk" to collect the total size of all files in each
package for writing out to PKGSIZE in each pkgdata file; however this
reports the total space used on disk not the total size of all files,
which means it is dependent on the block size and filesystem being used
for TMPDIR on the build host. Instead, take the total of the size
reported by lstat() for each packaged file, which we are already
collecting for FILES_INFO in any case.
Note: this changes PKGSIZE to be reported in bytes instead of kilobytes
since this is what lstat reports, but this is really what we should be
storing anyway so that we have the precision if we need it.
Fixes [YOCTO #5334]
(From OE-Core rev: 29615b36fca696822a715ece2afbe0bf9a43ed61)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The FILES_INFO entry in each pkgdata file stores the list of files for
each package. Make the following improvements to how this is stored:
* Store paths as they would be seen on the target rather than
erroneously including the full path to PKGDEST (which is specific to
the build host the package was built on)
* For simplicity when loading the data, store complete paths for each
entry instead of trying to break off the first part and use it as the
dict key
* Record sizes for each file (as needed by Toaster)
* Serialise the value explicitly using json rather than just passing it
through str().
Fixes [YOCTO #5443].
(From OE-Core rev: ca86603607a69a17cc5540d69de0e242b33382d3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at the code, it should be ICECC_USER_PACKAGE_BL and not
ICECC_PACKAGE_BL here, and also fix "localy" -> "locally".
Thanks to Scott Rifenbark for pointing this out.
(From OE-Core rev: b325e46059efe6974abf01131dca3f2094a6bf90)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that none of the packagegroups depend on virtual/kernel, we have the problem
that MACHINE=qemumips bitbake core-image-minimal doesn't put a kernel
into the deploy directory. This breaks many common usecases and
user expectations.
To avoid this, add a dependency on the kernel deploy to image do_build tasks.
This should avoid any circular dependency issues but equally ensure users
have their expectations met.
[YOCTO #5581]
(From OE-Core rev: fe26b2379ecdbdb56acde8592bc0c2d95092a207)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gcc has cross and target components with a shared workdir. The unpack umask
settings need to match for all of these. We need to use strings in each
case to ensure the sstate code matches them correctly.
This patch tweaks various things to ensure the change adding the unpack umask
change doesn't break the compiler builds.
(From OE-Core rev: 67162438ee9c402b23c32853af9d313949eb6e4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* when git checkouts files from fetched clone it respects system umask
and creates files with different permissions, if such files are copied
to packages, resulting target images have also different permissions
on them.
* we need reproducible builds across different builders with different
system umask, so set 022 umask
[YOCTO #5590]
(From OE-Core rev: c9289c506633ffe5c482000d8d225e45454c064d)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The comment was originally written for module.bbclass and is now
slightly misleading. This updates it to match the current code.
(From OE-Core rev: 434277ed156c1685283f6fd681062d265f4fa6d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 6a6735cb98.
The module class already ensures the scripts are rebuilt correctly. Running
this at sstate installation time is problematic since it can require the
cross compiler. Adding such a dependency would cause issues of its own.
(From OE-Core rev: b2c948d56241ff7cdea2e9e68b740f305c72f5ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current code is a little bit overcomplicated, deficient and also
possibly broken.
Issues include:
a) Not maximally optisming rpaths (e.g. a lib in usr/lib might get an
rpath of $ORIGIN/../../usr/lib)
b) The return in the middle of the for loop look suspiciously like
it might break on some binaries
c) The depth function, loops of "../" prepending and so on can
be replaced with a call to os.path.relpath
This patch cleans up the above issues.
Running binaries should result in less "../" resolutions which can't
hurt performance either.
[YOCTO #3989]
(From OE-Core rev: feea54df6768036649ca6c57524e2a1f480ad249)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix icecc.bbclass failing with:
ERROR: recipe-name NULL prefix
when it's used with empty TARGET_PREFIX.
* Allarch recipes cannot use compiler at all (even the local one)
so there is no point of using icecc for them.
(From OE-Core rev: a956f9d91c8128e43b55c6bc01337472e47fe43a)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Write a list of installed packages to a .manifest file next to the
image, so we can find out what went into the image after it has been
constructed without necessarily having to have buildhistory enabled
(although that will provide more detail.) We can make use of this for
example in the testimage class associated code that checks for installed
packages for determining whether or not to run specific tests.
Note: this replaces the previous ipk-specific manifest code with
something that works for ipk, rpm and deb, and instead of a pruned
status file, packages are listed one per line, in the following format:
<packagename> <packagearch> <version>
Tests for all three backends have shown that the performance impact of
this change is negligible (about 1.5s max).
Implements [YOCTO #5410]
(From OE-Core rev: 2978d1f2617a33e2e3a77e249d73e998d79b4ec9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
check_create_long_filename used a fixed filename for its test files. This
meant that os.remove(testfile) could fail with ENOENT if two instances were
running at the same time against the same sstate directory. Using a
randomly generated filename stops this from happening.
(Although it might seem unlikely, this race did appear to occur multiple
times with Jenkins - presumably because the matrix jobs were all kicked off
at the same time.)
(From OE-Core rev: bc28e3f26e7f85af82f403924c0ae29e1ad34a87)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Although ENAMETOOLONG is 36 on Linux x86 and x86_64 it does isn't on other
architectures so the value shouldn't be hard coded.
(From OE-Core rev: 11a9cf5ee0daf82097fb2f36b58016f20a5968f3)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids crashing on newer svn layouts where the entries files
don't contain three lines. If someone wants to fix this to
get the right version on newer subversion checkouts, patches
welcome but this at least stops things crashing.
[YOCTO #5363]
(From OE-Core rev: e850c53d4d8cb877a704a23f9ce02d6185ba3ffa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new do_bundle_initramfs task introduced in
609d5a9ab9e58bb1c2bcc2145399fbc8b701b85a defeats using the sstate
cache. The kernel is resurrected from the sstate cache but ends up being
built again since do_bundle_initramfs depends on do_compile.
The task is no longer nostamp to avoid causing unnecessary rebuilds. The
sstate checksum stamps should know when to rebuild.
The task now runs before do_deploy and part of the work has been moved to
do_deploy where it now writes to ${DEPLOYDIR} rather than
${DEPLOY_DIR_IMAGE} so that the files end up in sstate.
The task can also race against do_install since both call into the kernel
build system. This is fixed by making do_bundle_initramfs run after
do_install (which therefore also fixes the problem that
3baa63b4d588c3262254528b406ede265dd117bf was addressing.)
(From OE-Core rev: 55989cb509340bd265d0ce0d8bfe849681be4616)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 3baa63b4d588c3262254528b406ede265dd117bf. It broke
builds that aren't using kernel-yocto.
(From OE-Core rev: 81831db1c32afa3346f3ed9f4325ad280e5bb005)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When preping a read-only rootfs and finding some post-install
scripts that can not be run, list the names of said scripts to
avoid having to look around the rootfs to find a list.
(From OE-Core rev: 0188120691f433fdccf71b92618115195278c0af)
Signed-off-by: Jeffrey C Honig <jeffrey.honig@windriver.com>
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On a multilib system when one of the multibs has a different OS then
other multilibs a failure can occur during the install process because
RPM assumes all systems have the same OS.
When an n32 platform is selected as an alternative multilib, it shows
up as mips64_n32-.*-linux-gnun32 in /etc/rpm/platform. This causes
problems when the smart tool tries to add a channel for the multilib.
RPM archScore call always returns zero for arch "mips64_n32" -
after appending default vendor and os, it finds "mips64_n32-wrs-linux"
doesn't match any predefined platforms. Fix this by removing the
restriction of -gnun32 suffix in platform file.
(From OE-Core rev: d9489c44ee4f195ae1b09f340b9545cddba58145)
Signed-off-by: Lei Liu <lei.liu2@windriver.com>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PREFERRED_PROVIDER_virtual_pkg has been incorrectly set with more
than one multilib prefixes. For example, if we have two alternative
multilibs lib64 and lib32, PREFERRED_PROVIDER_virtual_pkg will be
set to lib32-lib64-pkg or lib64-lib32-pkg, depending on which
multilib shows up first in the list.
(From OE-Core rev: 17a432dc059e24ba10d4baec988828c0025a5e46)
Signed-off-by: Lei Liu <lei.liu2@windriver.com>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildhistory code has fallback paths for older bitbakes for now. The
distrodata class is much less used and it can be assumed a recent bitbake
is used in that case rather than adding fallback code.
(From OE-Core rev: 570cc145029fd9d5528aef5c27cb65164265c799)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The license_create_manifest function contains bashism, this will lead
to unexpected results on ubuntu build host, as sh is linked to dash on
ubuntu. Even if COPY_LIC_MANIFEST and COPY_LIC_DIRS are enabled, the
license files will still be missing on target.
This patch fixes the above problem.
[YOCTO #5549]
(From OE-Core rev: 4df9daee5c732c0a20dabe8515577238a1508512)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
${KERNEL_OUTPUT} is being renamed/restored in bundle_initramfs task, so we
must ensure bundle_initramfs run after kernel_link_vmlinux where the link
of vmlinux is created as the bootable image.
(From OE-Core rev: 3baa63b4d588c3262254528b406ede265dd117bf)
Signed-off-by: Ming Liu <ming.liu@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If kernel_do_compile() encounters a .lzma image for the initramfs, it
attempts to decompress it using the little-known "lzmash" utility. This
may not be present, which will cause the build to fail.
Fortunately, it appears that the more mainstream "lzma" tool is
compatible so we can use that one instead.
(From OE-Core rev: f7a04cbdee67675ff6a8787709f3a312dc25bfec)
Signed-off-by: Phil Blundell <philb@gnu.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The icecc class often calls 'which' for determining paths. This leads
to many messages on stderr in case 'which' doesn't find the
executable. Using bb.utils.which is more appropriate here and doesn't
pollute stderr.
(From OE-Core rev: 7ed7aa38182ce8c7300a08e3aefcc65da2b524a8)
Signed-off-by: Tobias Henkel <tobias.henkel@bmw-carit.de>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code in checkpkg from distrodata.bbclass had similar
functionality with fetch when searching for latest package version.
For packages that use svn protocol that part was rewrote in order
to use fetcher API.
It now calls latest_revision method from Svn class in fetch2 so
that it gets latest version.
[ YOCTO #1813 ]
(From OE-Core rev: 2686b92bbab88cc777fdc0e4dded5aeabca7ac77)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, it supported the situation that /boot area with separate boot
partition:
...
menuentry "Update bzImage-3.10.10-WR6.0.0.0_standard-3.10" {
set root=(hd0,1)
linux /bzImage-3.10.10-WR6.0.0.0_standard root=/dev/sdb1 rw ip=dhcp
}
...
But didn't consider the situation that /boot within root partition:
...
menuentry "Update bzImage-3.10.10-WR6.0.0.0_standard-3.10" {
set root=(hd0,1)
linux /boot/bzImage-3.10.10-WR6.0.0.0_standard root=/dev/sdb1 rw ip=dhcp
}
...
This fix supported them both.
[YOCTO #5514]
(From OE-Core rev: 53d342db9f9995564573715f28c3e4c9c8c68bf9)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit [kernel.bbclass: test for hardlinks before installing] doesn't
work on all build machines without a space between the ! and the test for
hardlink equivalance. The build continues, while the test fails and no
copy is made into the sysroot. Eventually tasks like build_hdimg will fail
with the missing kernel with the root cause being less than obvious.
(From OE-Core rev: 0704648e9dfd2897f6f54773a0e7d2762117970c)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
commit df564c4 [kernel: Use hardlinks for do_populate_sysroot for speed]
does indeed speed up the build, but it also means that the KERNEL_OUPUT
and KERNEL_IMAGETYPE may be hardlinks.
If they are hardlinks, install complains that they are actually the same
file and stops the build. We can easily test and avoid the copy if the
file is already in place.
[YOCTO #5527]
(From OE-Core rev: 6a3b3e501c66733216ac43d3f020cfdb512dd640)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The expected usage of COPY_LIC_MANIFEST and COPY_LIC_DIRS appears to be
to set them to "1" to enable; however the test here is just testing
whether they have a value at all, so setting them to "0" would also
enable them which is somewhat disingenuous. Actually check if they are
set to "1" instead in order to fix this.
(From OE-Core rev: 2d8e124adcf27af524eeeae61daf1b21a1c2f27c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is very old, and we now have pkgdata which is much more complete.
Nobody appears to be using this class, and even OE-Classic had no
current references to it.
(From OE-Core rev: e67dbb638044b804738bdd589d64d45963a3297a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This appears to be a very old way of archiving recipes and associated
files. We have better ways of doing this now, and nobody appears to be
using this class - even OE-Classic had no current references to it.
(From OE-Core rev: e3141844ef84d8a48efa81e0c9c85821ced16f7c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SDKMACHINE is set then check that a configuration file matching it
actually exists, otherwise the user won't know that they've set it
incorrectly.
(From OE-Core rev: 8c984f92af821a4048c93f8e308c5f4a3fa39ca4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* If the md5 parameter is specified with no value, report that the
checksum is not specified instead of reporting that it has changed
* If the md5 checksum has changed, point directly to the license file in
a way that is easy to copy and paste and give the line numbers in an
easy to read form, as well as asking the user to verify that the new
contents matches the current LICENSE value.
(From OE-Core rev: ec8590aa81e201e28e500935d31cd7266114471f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't actually need this dummy class; "inherit" can be used with an
expression that evaluates to nothing with current BitBake.
(From OE-Core rev: f312eb2d2a2715e772ed9e8afc7aea326d8079ec)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This saves about 2GB on a core-image-sato build so is worth doing and is
consistent with our efforts to try and decrease our build footprint.
Build time in my local test seemed unaffected but on more IO bound
machines it should help.
(From OE-Core rev: 7136cbc64e5efb09f3fae3e2e35a181ca3d66dd4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.fetch.decodeurl() will throw if it doesn't like the look of the URL that
it's given. (Bitbake's idea of what constitutes a valid URL is somewhat
idiosyncratic so it is fairly easy to trip over this by mistake when writing
a recipe.)
If these exceptions are allowed to propagate all the way up to better_exec()
then we will get a large amount of python stack trace spew when they are
finally caught. Avoid that by catching them locally and throwing
bb.build.FuncFailed() with a suitable explanation instead.
(From OE-Core rev: ef35e164c62d89806367b822e3baeff482ec237f)
Signed-off-by: Phil Blundell <philb@gnu.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An empty distro value leads to OVERRIDES and FILESOVERRIDES containing
"::" entries which causes odd issues such as files being included when
they shouldn't be. We could put in anonymous python to guard against
empty entries but its messy and setting a default value for DISTRO to
something harmless is much easier.
This patch adds a weak default and ensures the sanity test doesn't
complain about it.
DISTRO_VERSION and SDK_VERSION are also updated to match.
(From OE-Core rev: b7279f99639774674da806d37d252f388f33055f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
