This patch implements custom target loading for testimage, currently
missing due to major changes to the test framework.
Custom targets can be defined in various meta-layers, so we
need an extra path information in order to find them.
Any other additional info is retrieved as usual via the variables
TEST_TARGET and TEST_SUITES
(From OE-Core rev: 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.
(From OE-Core rev: 8de0c5d3bd01919e2bf0394f9c485936d6098cec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current relocation code is broken, at least in the native case. Fixing it
would mean trying pass in new data on sstate tasks about the relative positioning
of symlinks compared to the sstate relocation paths. Whilst we could do this,
right now I'm favouring making this an error and fixing the small number of
problematic recipes we have in OE-Core (3).
(From OE-Core rev: cf94de4ddee3e5072da8608c9151301fcec02cd0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Absolute path symlinks are a bit of a pain for sstate and the native versions
of these recipes currently contain broken symlinks as a result. There are
only a small number of problematic recipes, at least in OE-Core, namely the
three here.
Rather than trying to make sstate handle this magically, which turns out to
be a harder problem than you'd first realise, simply make the symlinks relative
early in the process and avoid all the problems.
The alternative is adding new complexity to sstate which we could really
do without as without the complexity, you can't always tell where the
absolute symlink is relative to (due to prefixes used for native sstate).
(From OE-Core rev: e478550c8cd889f12e336e268e9e3b30827bf840)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If image type "foo" depends on image type "bar.xz", then dependencies
should be collected from the base image type (ie "IMAGE_DEPENDS_bar")
not from "IMAGE_DEPENDS_bar.xz".
(From OE-Core rev: 8a9f249a9166347cc0468191ce130003e3d306e1)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possible to restore the behaviour changed by commit 78cde87
"kernel-module-split: Append KERNEL_VERSION string to kernel module name".
(From OE-Core rev: 4aabdddf67859cd8518e5cb672fb391c859817e0)
Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In do_compile we set HOME so that ~/.npm* only get created in the work
directory; we need to do the same in do_install as well or they'll go
into the user's home directory which we do not want.
(From OE-Core rev: a667cb9ed2dceb804b85d1a024e2619bc34d2681)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Any custom tasks that were added on the image between do_image_complete
and do_build were not being taken into account. Use the newly added
bb.build.tasksbetween() function to take care of that.
(From OE-Core rev: c966dee6208d3b7a9fa3c1b8aa1266058dcb65ff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A generic version of the code to work out the tasks between two
tasks (based on the code here) has been added to bb.build, so use that
instead.
(From OE-Core rev: 1226e2eb2678407c315befd5700e653602e616a0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Up until recently it was possible to set SDK_TARGETS to include a native
recipe you wanted installed into the sysroot when installing the eSDK.
I'm not sure what happened but now when you try to add a native recipe
to SDK_TARGETS you get a missing task error because this recipe has no
do_package_write_* task. Of course such a task dependency is erroneous
and is apparently caused by setting SDK_RDEPENDS. I've checked and it
turns out that we no longer need to set SDK_RDEPENDS anyway (probably
because we explicitly set up task dependencies further down in the
class, which I don't think we were fully doing in early versions of the
eSDK). Thus, drop setting this variable to restore the functionality.
(From OE-Core rev: fe8268ede33a8c76492b85da0f41062f47a5279c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you abort a build mid execution of the staging extend_sysroot function
there are currently races and cleanup of that function may fail.
This change splits the code into separate phases so that the manifests are
manipulated before files are installed, meaning we should be able to reverse
actions if builds fail, crash or are interrupted.
(From OE-Core rev: fc504b6b8538453d2f058d7cc69be098f52f8091)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This wastes space/time as the ptest files aren't going to be used in the
sysroot so add them to the blacklist.
(From OE-Core rev: 61591fe2eea588ff92ff7f81f2ed8844617f5307)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make name of the wic image type class consistent with
existing naming scheme for image types.
(From OE-Core rev: 4aab1b77d5f9403cbb3fae790069ef54821491fb)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When debugging a kernel configuration issue, one of the first questions
is "what BSP was used". To answer this qusetion, we log the BSP .scc
file that was used to generate the kernel configuration in the kernel
source meta directory.
(From OE-Core rev: 1003615b866bfe196994bdc25d305ff5eace2715)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ability to merge two branches directly from a .scc file was
dropped during the streamlining of the tools.
As was pointed out by David Vincent <freesilicon@gmail.com>, there is
once again a valid use case for this functionality, so we restore the
capability.
(From OE-Core rev: a0059ebbb52c659282e355664bba1a2fa282170e)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we switched to recipe specific sysroots (rss), performance took a nose dive. Its
easy to blame rss but it turns out not to be entirely at fault.
Three configurations are compared here:
a) Pre-RSS (revision 45df694a9f)
b) Post-RSS (revision 226a508da9)
c) as b) with this change
Overall build times:
a) 22794.25user 2687.88system 30:32.84elapsed 1390%CPU (0avgtext+0avgdata 919056maxresident)k
b) 22677.25user 3238.79system 36:16.68elapsed 1190%CPU (0avgtext+0avgdata 918896maxresident)k
c) 23571.84user 3383.65system 31:36.83elapsed 1421%CPU (0avgtext+0avgdata 919068maxresident)k
For the overall build and sstate directories, du -s shows:
a)
3992588 build-pre-rss/sstate-cache
30804484 build-pre-rss/tmp
b)
4013272 build-with-rss/sstate-cache
36519084 build-with-rss/tmp
c)
4014744 build-with-rss2/sstate-cache
35336960 build-with-rss2/tmp
However more worryingly:
$ du -s build-pre-rss/tmp/sysroots/
2506092 build-pre-rss/tmp/sysroots/
$ du -s build-with-rss/tmp/sysroots-components/
3790712 build-with-rss/tmp/sysroots-components/
$ du -s build-with-rss2/tmp/sysroots-components/
2467544 build-with-rss2/tmp/sysroots-components/
These numbers *should* be equivalent but as you can see, b) is ~1.2GB larger. The reason turned out
to be patchelf. Taking a specific binary from a specific recipe, bc from bc-native, in a) its 82kb
(stripped) yet in b) its 2.17MB.
$ ./patchelf --set-interpreter /bin/rp bc
warning: working around a Linux kernel bug by creating a hole of 2084864 bytes in ‘bc’
https://github.com/NixOS/patchelf/blob/master/src/patchelf.cc#L710 shows that this "hole" is just
padded zeros using memset, its not a proper sparse hole.
This patch copies files with cp --sparse=always after modifying them with patchelf, then replacing
the original file. The better fix will be to fix this in patchself itself and seek() there
when writing the new file but that means new uninative tarballs and will take a bit of work
so I'm proposing this workaround in the meantime.
Also, this patch drops error handling since subprocess check_output() tracebacks will print this
information if the command fails so we can simplify the code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The number of mkdir calls was showing up high on the profile charts since
it was getting called once per file which is excessive. Each call results
in one or more syscalls which is bad for performance. Cache which
directories we've seen to reduce the calls to a more reasonable number
and speed up recipe specific sysroots.
(From OE-Core rev: 680fb343be5d0a7f9abbe9c75ca232abe5871663)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
According to profile data, repeated calls to bb.debug and bb.note in
the extend_recipe_sysroot() codepath were accounting for 75% of the time
(1.5s) in calls from tasks like do_image_complete.
This batches up the log messages into one call into the logging system
which gives similar behaviour to disabling the logging but retains the
debug information.
Since setscene_depvalid is also called from bitbake's setscene code,
we have to be a little creative with the function parameters and leave
the other debug output mechanism in place. This should hopefully
speed up recipe specific sysroots.
(From OE-Core rev: 3b0af8dc0f796345d1f1ba77ea35bbd090a5feb3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The native .pc files currently have hardcoded paths in them meaning each has
to be relocated at final install time. pkg-config has built in functionality
to avoid this, namely the pcfiledir variable.
This function translates .pc files to use the variable meaning further
relocation later is unnecessary.
(From OE-Core rev: a3f4e9ff55c7d9eba175e83eb6c99cf349a74432)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst the path to the executable is dynamically determined, the passed in
environment variables or parameters are not relocatable and rely on the sstate
remapping code. In the recipe specific sysroot case this has become more costly.
This patch translates such paths into relocatable entries which means that
a sed replacement at final installation isn't needed.
(From OE-Core rev: 3e02a7cda0ba70e3f9886c5cce37d71361280b11)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debian based distros has a builtin syslog module so when
try to load tests using unittest it references the builtin
module instead of runtime/cases.
[YOCTO #10964]
(From OE-Core rev: 9923e3cdb58c2b3c54ec5fe99b2cec4cdc9fff92)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some missing suites on sato-sdk and lsb-sdk images so add it.
The createrepo-native needs to be built before run testimage in
order to create a repository from install packages.
The DL_DIR data also needs to be updated from the current bitbake
environment instead of use the value from testdata file.
[YOCTO #10964]
(From OE-Core rev: 42f3026b39519ba98ae3153329622ce670d57c09)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The source makes no reference to insert_feed_uris any more, simply drop
the unused reference, to avoid confusing the developers.
(From OE-Core rev: 26dddb7d4a4945abaaefd460a342de6abd609594)
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a complementary fix to commit 7c552996:
[ meta: remove True option to getVar calls ]
it intended to remove all True option to getVar calls, but there are
still some remaining.
Search made with the following regex: getVar ?\((.*), True\)
(From OE-Core rev: 87d03ffe03d6f01e360bfd51714be96e62506e0a)
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to the new framework data store dependecy was
removed, instead a new file is generated and used in testimage.
When testing builds from the autobuilders the test data values
are from the autobuilder, including the paths.
Some tests require paths to current environment in order to run,
this commit will update such paths and fix the error of running
images donwloaded from autobuilders.
[YOCTO #10964]
(From OE-Core rev: 26ad5105fc2ce03b7ee8ecc6911fd40a52bd573a)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add created symlinks to the exclude file. This will both make them
less distracting and hide them from the srctree_hash_files function.
(From OE-Core rev: c11fcd6fbde4a90913960b223451e0ce9e6b4b64)
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For this we move them out of the python section so they can be
overridden on a per-recipe basis.
The motivation for this change is that not all tool chains need the
path modifications provided by the command, and these will provide
alternative or empty commands. The Go compiler is such an example.
(From OE-Core rev: 7d2a2160336413736dd4640f5b84ba4d74bb00f5)
Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If ext sdk is built by gcc version higher than host gcc version
and host gcc version is 4.8 or 4.9 the installation is known to
fail due to the way uninative sstate package is built.
It's a known issue and we don't have a way to fix it for above
mentioned combinations of build and host gcc versions.
Detected non-installable combinations of gcc versions and
print an installation error.
[YOCTO #10881]
(From OE-Core rev: bfb9ea2604557804bd8c16adb57ccdd868a4020f)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was discovered that buildtools-tarball can't be built with Debian packaging:
E: Unable to locate package nativesdk-buildtools-perl-dummy
The package exists but dpkg doesn't see it because the Architecture field isn't
valid: for this package the Architecture field is 'allarch' which isn't in the
list of valid architectures we define.
package_deb already has a mapping of OE architectures (PACKAGE_ARCH) to dpkg
architectures (DPKG_ARCH), for example our x86_64 is amd64 in dpkg. However
allarch.bbclass sets an invalid TARGET_ARCH of 'allarch' (to break anything
which attempts to use the target architecture) and package_deb relied on the
PACKAGE_ARCH check later turning the Architecture field into 'all'.
This usually works, but nativesdk-buildtools-perl-dummy inherits allarch but
then changes PACKAGE_ARCH to buildtools-dummy-nativesdk to isolate the packages
in a separate feed. We can handle this by explictly checking for allarch in the
TARGET_ARCH check, as that everything that inherits allarch will want to have
Architecture: all in their packages.
[ YOCTO #10700 ]
(From OE-Core rev: 58a5771f20f7851816b5f39df0ada8f16438065c)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This task uses DATETIME so add it to vardepsexclude so bitbake doesn't refuse to
execute the task when the time changes between the controller and the worker
calculating the hash.
[ YOCTO #10960 ]
(From OE-Core rev: e6784ef0d56a1b297001bc59c0da3aecebd1c656)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
grub-efi recipe added 'grub-efi-' prefix to the file name of
efi binary. Changed grub-efi.bbclass accordingly.
(From OE-Core rev: 860c44435ea7cbbe3517d5ab9146213cc80fd0e7)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
isoimage-sihybrid plugin uses MACHINE_ARCH to get the name of initrd image.
It doesn't work for all machines, for example for quemux86-64 machine
MACHINE_ARCH is quemux86_64 and initrd name is
core-image-minimal-initramfs-qemux86-64.cpio.gz
Used TRANSLATED_TARGET_ARCH variable to get the initrd image name.
Replaced MACHINE_ARCH->TRANSLATED_TARGET_ARCH in WICVARS variable to
make it available from <image>.env file.
(From OE-Core rev: edf0830412de66eada9ef3f3947ca42e301c5377)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed HDDDIR as it's not used by wic anymore.
Stopped usage of HDDDIR in wic test suite.
(From OE-Core rev: b3ed3a79852f85651c8fa192a8118f9904e88ef8)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
INITRD variable is not set if hddimg is disabled.
isoimage-isohybrid can't get correct name for initrd if INITRD
variable is not set.
Added INITRD_LIVE to WICVARS and used it in isoimage-isohybrid
code to get initrd artifact name. Used INITRD if INITRD_LIVE is not set.
(From OE-Core rev: c672753be60550b15d5d593c251b61776d5b104d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a lot of wic code in image.bbclass and image_types.bbclass
Having all code separated in one file should make it more readable
and easier to maintain.
(From OE-Core rev: 786368568a9525212e69f5cbf6da236f0a6be013)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current location of .env files $STAGING_DIR/imagedata. It doesn't
depend on machine and be rewritten by the builds for different
machines.
Changed location to $STAGING_DIR/$MACHINE/imagedata to avoid .env
files to be rewritten.
(From OE-Core rev: 94245144f5cef344d90bc2a7b3267cdae9d192e4)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Wic put result images into <output dir>/build, which was confusing.
Now it's fixed in wic code and images are put into output directory.
Changed code in image_types to reflect this.
[YOCTO #10783]
(From OE-Core rev: 3983507bf859f6c3c29e9dea06ffe51f42c0759c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a stale sysroot object was found the complete stamp wasn't removed,
meaning later code could fail when the new link creation was attempted.
Ensure the stale complete stamp is also removed if present.
(From OE-Core rev: d7d26488f252d60628862ee114a4404e8d6cb6f1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recent changes exposed the fact this dependency was missing in nativesdk
cases, fix this.
(From OE-Core rev: 46bd7e48cb5ae654d915fa493420534219682e85)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The main sysroot components of unreachable build targets will be removed by the core
code. This currently doesn't trigger a removal in the individual workdirs. This
adds in symlinking between the complete stamps and the component sysroot meaning
we can detect when someting was removed and hence remove it from the sysroot.
This fixes cases where DISTRO_FEATURES like systemd are changed amongst other
things and makes builds more robust against configuration changes.
If a dependency is rebuild, that is caught by checksum comparision code elsewhere
in this function as before.
(From OE-Core rev: 7656347c556915dc9acc5a2311ec4e879de6f43d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Manifest files containing the same duplicated prefix are wasteful on space
and ultimately this costs build time. Drop the WORKDIR prefix from the manifest
files since this small change mounts up a lot.
(From OE-Core rev: 1cb245a99762e21a170b6a9beabb07e558424946)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Manifest files containing the same duplicated prefix are wasteful on space
and ultimately this costs build time. Add support for manifest files with
common prefixes removed and use the prefix if the path isn't absolute.
(From OE-Core rev: 71e8c98238d627f9665faee9d7c226b9c07bd256)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With recipe specific sysroots, the gzip-replacement-native dance/class
is obsolete, simplify the code accordingly.
(From OE-Core rev: 39865fdf3698a130f792d41853f9c9ca1901e335)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that STAGINGDIR_HOST doesn't contain MACHINE, we no longer need to expand
the value. Pre-expansion can mean components like PV can be expanded too early
and cause problems for certain use cases.
(From OE-Core rev: d04f0363a1e8ae641da3e80ffbd4c8803cb1c91a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We used to have issues removing tasks like do_fetch due to implications
for targets like world and universe. These have now been resolved.
Removing uneeded tasks has advantages compared to noexec since it means
that accidentally left in dependencies are no longer needed/processed
(e.g. do_patch depends on quilt-native).
This cleans up a number of cases which local analysis highlighted as
being unneeded leading to slightly cleaner task graphs.
(From OE-Core rev: 4e6ee37e09c60e83c0dfd844ba9cf8a07507f099)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
WKS_FILE(S) can refer to .wks.in files which get expanded during the
build by do_write_wks_template. The actual content of the .wks.in file
gets added to the recipe meta data during parsing, and thus we need to
ensure that the recipe gets re-parsed when the file changes.
This fixes two related problems:
- editing the .wks.in file and rebuilding an image did not recreate
the image unless something else changed or "bitbake -c clean" was
used explicitly
- when forcing a rebuild, the cached meta data and the actual one
do not match, leading to "ERROR: Taskhash mismatch ... for ....bb.do_write_wks_template"
(From OE-Core rev: d00b95b18237b276c221b16dfc511a6da150ef06)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
