Commit Graph

168 Commits

Author SHA1 Message Date
Martin Jansa
26a7a415eb populate_sdk_ext.bbclass: set METADATA_REVISION with an DISTRO override
* otherwise it ends '<unknown>' inside esdk, because of parsing order:
  # $METADATA_REVISION [3 operations]
  #   set /OE/build/test-D/conf/local.conf:43
  #     "f2da54ef432eac89b0f18eaad68e602b6990b5de"
  #   immediate /OE/build/test-D/layers/poky/meta/classes/metadata_scm.bbclass:9
  #     "${@oe.buildcfg.detect_revision(d)}"
  #   set /OE/build/test-D/layers/poky/meta/classes/metadata_scm.bbclass:10
  #     [vardepvalue] "${METADATA_REVISION}"
  # pre-expansion value:
  #   "<unknown>"
  METADATA_REVISION="<unknown>"

* This causes base-files.do_install and following tasks to have different
  signatures between esdk and the build directory where this esdk was created:

  bitbake-diffsigs {test-D,poky/build-uninative-disabled}/tmp/stamps/qemux86_64-poky-linux/base-files/*do_install*sigdata*
  NOTE: Starting bitbake server...
  basehash changed from 5b6981cf58bfd57d416b0e31611b73a26baae635dd1ac31c08d46f95064c3ffc to dbdce042da4d7813d632b6d1cc87a16f728ad20e55fecbc392830e6acf72babd
  Variable METADATA_REVISION value changed from '<unknown>' to 'f2da54ef432eac89b0f18eaad68e602b6990b5de'

  and an warning from "python3 /OE/build/test-D/ext-sdk-prepare.py" when eSDK is being prepared for use:
  WARNING: The base-files:do_install sig is computed to be 83b9c9a6ef1145baac5a1e0d08814b9156af239c58fc42df95c25a9cd8a7f201,
    but the sig is locked to 3dc22233059075978e5503691e98e79e7cc60db94259dfcd886bca2291c0add7 in SIGGEN_LOCKEDSIGS_t-qemux86-64

[RP: Add commit about why we need the override for future reference]
(From OE-Core rev: da6f6340f97e80cb1b21f6083ef5d0a9a856eef5)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 675ea7281c17f77bf5dea17cfd4d9da0928382a0)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-05-12 04:04:52 -10:00
Kenfe-Mickael Laventure
7a502f838a populate_sdk_ext: Handle spaces within user $PATH
The script generated by the sdk_ext_postinst function was not quoting
the user existing PATH when updating it causing the export command to
fail.

Add necessary double quotes around $PATH.

(From OE-Core rev: d2800e00bb4a1d169e0a91b758d3fb2ec5e35f74)

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 00e96bf250eaaded839caf465dbc0af5b604aed7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-23 22:45:34 +00:00
Niko Mauno
4abffe76eb Fix missing leading whitespace with ':append'
Mitigate occurences where ':append' operator is used and leading
whitespace character is obviously missing, risking inadvertent
string concatenation.

(From OE-Core rev: fcd340ec53ff8352b8cae0eb351810072b025a08)

(From OE-Core rev: cb64ace13db85e143d99627c8803fbb13ba18617)

Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 23:32:20 +00:00
Saul Wold
d9e500f83d meta/scripts: Improve internal variable naming
Update internal variable names to improve the terms used.

(From OE-Core rev: f408068e5d7998ae165f3002e51bc54b380b8099)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-10 08:00:28 +00:00
Richard Purdie
71ef319193 meta/scripts: Automated conversion of OE renamed variables
(From OE-Core rev: aa52af4518604b5bf13f3c5e885113bf868d6c81)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-21 23:37:27 +00:00
Richard Purdie
98b3ed1b2e meta/scripts: Handle bitbake variable renaming
After other variables were renamed in bitbake, update OE-Core to match.

(From OE-Core rev: 91812ba5a34598e03ad860745707c7cba1ae5d91)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-21 23:37:27 +00:00
Richard Purdie
492214104a meta/scripts: Change BB_ENV_EXTRA_WHITE -> BB_ENV_PASSTHROUGH_ADDITIONS
After the change to bitbake, update the references in OE-Core to match the updates.

(From OE-Core rev: 193affb9f28b0116c3fd619834f145326fee08c5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-21 23:37:26 +00:00
Ross Burton
c0583c6bfc meta: use ln -rs instead of lnr
lnr is a script in oe-core that creates relative symlinks, with the same
behaviour as `ln --relative --symlink`.  It was added back in 2014[1] as
not all of the supported host distributions at the time shipped
coreutils 8.16, the first release with --relative.

However the oldest coreutils release in the supported distributions is
now 8.22 in CentOS 7, so lnr can be deprecated and users switched to ln.

[1] 6ae3b85eaffd1b0b6914422e8de7c1230723157d

(From OE-Core rev: 1ca455a98de4c713f58df0a537d4c982d256cd68)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-10 19:27:29 +00:00
Richard Purdie
b85fd33e7d sstate: Fixes for eSDK generation after zstd switch
(From OE-Core rev: f5a90df08631117cd66bec68cb1851db96822bb8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-15 12:58:00 +01:00
Richard Purdie
825043ec77 populate_sdk_ext: Fix handling of TOOLCHAIN_HOST_TASK in the eSDK case
The current way TOOLCHAIN_HOST_TASK is handled within eSDK builds is
causing much confusion, even to people who should understand the code.

For a normal SDK, some layers append to TOOLCHAIN_HOST_TASK unconditionally
which is fine, until the eSDK tries to override the variable to it's own
values. It does not support or use packages installed using this variable
and would use native recipes instead, it is a very different approach.

In the referenced bug, binaries are added but not relocated leading to
confusing errors.

To fix this, add a new variables for the eSDK TOOLCHAIN_HOST_TASK
to be explict and force the eSDK code to use this instead. The setVar
in non-parsing context will clear out any appends resolving the current
issues. The patch also gets rid of some dubious task override use.

[YCOTO #14047]

(From OE-Core rev: 30912ba084aa8600156edddbe4f2db0b85e869d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:11 +01:00
Richard Purdie
65133015b1 meta: Manual override fixes
The automated conversion of OE-Core to use the new override sytax isn't
perfect. This patches some mis-converted lines and some lines which were missed
by the automation.

(From OE-Core rev: 4e9a06b64b43131b731fb59a0305f78a98e27fbd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:10 +01:00
Richard Purdie
bb6ddc3691 Convert to new override syntax
This is the result of automated script conversion:

scripts/contrib/convert-overrides.py <oe-core directory>

converting the metadata to use ":" as the override character instead of "_".

(From OE-Core rev: 42344347be29f0997cc2f7636d9603b1fe1875ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:10 +01:00
Mark Hatle
c1dcadb7f8 populate_sdk_ext: Error if trying to generate an eSDK from a mulitconfig
Running a build such as:

  bitbake mc:my_config:core-image-minimal -c populate_sdk-ext

will result in an error like:

ERROR: Task base-files.do_fetch attempted to execute unexpectedly
Task .../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_packagedata, unihash ec5ba0e6b31561daba005fb49c5239c8e46913465b51166b5905f3e5ffcf2741, taskhash ec5ba0e6b31561daba005fb49c5239c8e46913465b51166b5905f3e5ffcf2741
Task .../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package_write_rpm, unihash 1c7d7509c2ff6dcf11009fbec444726826214795d60474ec8d3262d89c40a955, taskhash 1c7d7509c2ff6dcf11009fbec444726826214795d60474ec8d3262d89c40a955
Task .../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_populate_sysroot, unihash 9cc3672f4fa62491f545b15cf617a64cd77d15a2cfd432b57d4b936bc415f40d, taskhash 9cc3672f4fa62491f545b15cf617a64cd77d15a2cfd432b57d4b936bc415f40d
Task .../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package_qa, unihash 8ada5f62092c971df8dda1d71c728e42994e1dcf2bbdab419de43867d77b64cc, taskhash 8ada5f62092c971df8dda1d71c728e42994e1dcf2bbdab419de43867d77b64cc
Task .../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/images/core-image-minimal.bb:do_image_qa, unihash 16656a339389e407a5fdca5d64983af845288f3b3cc5582398e5247efb393257, taskhash 16656a339389e407a5fdca5d64983af845288f3b3cc5582398e5247efb393257
Task .../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/images/core-image-minimal.bb:do_image_complete, unihash ef88c74a9f4ae4d252c421eb4e399773aa50cea7c51ffbeed9011e5198a16abb, taskhash ef88c74a9f4ae4d252c421eb4e399773aa50cea7c51ffbeed9011e5198a16abb
This is usually due to missing setscene tasks. Those missing in this build were: {'.../poky/build-mc/tmp-my_config-glibc/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package_qa',

Instead of letting the system error, we simply tell the user this is not supported.

As long as the eSDK is constructed based on the primary library, it works fine.

(From OE-Core rev: b359c60071585fa323124fc6febe652fe0128b52)

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-02 23:14:32 +01:00
Justin Bronder
2834c2f853 populate_sdk_ext: copy BBMULTICONFIG files
As the generated local.conf includes BBMULTICONFIG, the referenced files
in conf/multiconfig also need to be copied.  Otherwise with
BBMULTICONFIG="abc" for instance, building the esdk fails with:

ERROR: ParseError at tmp/build-glibc/work/qemux86_64-oe-linux/core-image-ssh/1.0-r0/sdk-ext/image/tmp-renamed-sdk/layers/openembedded-core/meta/conf/bitbake.conf:767: Could not include required file conf/multiconfig/abc.conf

(From OE-Core rev: 8bc339b83a45becc0c7edf016fcce187152669e3)

Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-22 15:18:11 +01:00
Devendra Tewari
b71375304f classes/lib/scripts: Use bb.utils.rename() instead of os.rename()
Incremental build in Docker fails with:

OSError: [Errno 18] Invalid cross-device link

when source and destination are on different overlay filesystems.

Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.

[YOCTO #14301]

(From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:16:23 +01:00
Mark Hatle
3edc458cb9 populate_sdk_ext: Add support for PR service
In the classes/populate_sdk_ext.bbclass the system already copies a number
of configurations, such as the hash equivalency data.  However, the PR
service was being handled.

The new code works by checking if PRSERV_HOST is defined, if it is, use
the existing export functions to write out a conf/prserv.inc file into
the eSDK.  On eSDK install, if a conf/prserv.inc file is present we then
import this file into the system.

This mechanism will work if the PRSERV_HOST is local or remote, as it pulls
the necessary data from the server and then imports it to a local database
on eSDK installation.

Note: the conf/prserv.inc file is not deleted at this time.  It was left
for possible debugging purposes, but removing it is something we could decide
to do in the future.

(From OE-Core rev: e207dabdfaa07cd5ebba1cd7dd58610f7185c7e2)

Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-31 00:06:23 +01:00
Mark Hatle
63831568bd populate_sdk_ext: Avoid copying and producing .pyc files
Since pyc cache files are really system specific, no real reason to copy or
generate them during the eSDK build process.  Also generating them has the
possibility of re-using inodes that pseudo may have been tracking, leading
a build failure.

(From OE-Core rev: ce8eba263647ae63a722122e28f26af46ae083a0)

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-28 22:28:27 +01:00
Chen Qi
f00dfafbaf populate_sdk_ext: record METADATA_REVISION
As we delete the .git/ directory, it's impossible to get METADATA_REVISION
inside eSDK. Because of this, we meet the following warning when installing eSDK.

  WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
  but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

So we record METADATA_REVISION in eSDK generation time to fix this problem.

(From OE-Core rev: ff2ad51b801fd62e2abbc573ba2c9ee8fdc7e012)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-14 16:33:59 +00:00
Chandana kalluri
a334a8c67f populate_sdk_ext: use SDK_CUSTOM_TEPLATECONF variable to enable custom templateconf.cfg
The current implementation will always pick an existing templateconf.cfg if
present else it will use the one from OE.  A user might not always want to pick
an existing tempalteconf.cfg even if its present.  Introduce
SDK_CUSTOM_TEMPLATECONF variable to provide an option for the user to specify if he
wants to use an existing custom templateconf.cfg or not

If SDK_CUSTOM_TEMPLATECONF=='1' then enable custom templateconf.cfg. By default
SDK_CUSTOM_TEMPLATECONF is set to '0'

(From OE-Core rev: d0f863a24d05bddeb21e181fb01fa0051c79d7d8)

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-09 12:20:33 +00:00
Chen Qi
23bb87bff7 populate_sdk_ext.bbclass: add ESDK_MANIFEST_EXCLUDES
Add ESDK_MANIFEST_EXCLUDES to enable excluding items in sdk-conf-manifest.

By default, files under conf/ are all added to sdk-conf-manifest, as the
manifest file is set to 'conf/*'.

However, there are situations where some configuration files under conf/
directory are not intended to be added to sdk-conf-manifest, thus adding
ESDK_MANIFEST_EXCLUDES to enable users to do this.

This variable takes the form of glob matching.
e.g.
ESDK_MANIFEST_EXCLUDES = "conf/autogen*"
This would exclude all files under conf/ starting with 'autogen' from
sdk-conf-manifest.

(From OE-Core rev: 2d71e427b530ec4ea5524efa951b6a87f21b8b22)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-30 15:01:51 +01:00
Khem Raj
31bbf06a13 populate_sdk_ext: Do not assume local.conf will always exist
Some distros may not have local.conf, and use auto.conf or site.conf
alone to describe distro config metadata, therefore make code robust
to not fail eSDK builds for such setups

(From OE-Core rev: 6379f951ba92b2b86a55de5fc267bacf8c6095d2)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-14 00:14:28 +01:00
Richard Purdie
4901fc6d27 populate_sdk_ext: Ensure buildtools doesn't corrupt OECORE_NATIVE_SYSROOT
buildtools is built as a nativesdk which needs to use OECORE_NATIVE_SYSROOT
for its own purposes and can reset it. Save and restore the value
within the eSDK so the two don't clash.

(From OE-Core rev: 513f4006bf563620ee063ba6d0e9b3f27334642e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-08 09:17:49 +01:00
Richard Purdie
48abe482dd populate_sdk_ext: Fix to use python3, not python
We should be using python3 here, it was missed in the conversion. Spotted on
autobuilder tests failing on systems with python missing.

(From OE-Core rev: db07b09196022078346aadd565760240b7da6a71)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-28 08:36:01 +01:00
Andrej Valek
75bebb3546 populate_sdk_ext: enable custom templateconf.cfg
Do not always override templateconf.cfg content. Add option to use
already existing file.

(From OE-Core rev: e524e49ef22fd69882d5d2d01cd84db790e9cb88)

Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-07 10:06:16 +00:00
Richard Purdie
f73bf58f05 populate_sdk_ext: We now require python3, not python
We no longer expect a "python" binary in PATH so update the eSDK's
expectations to match. This was the only failure on autobuilder test
systems with python missing.

(From OE-Core rev: 946ce21b10dcad506edcaadb4e4242c049e4c316)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-15 10:26:42 +00:00
Peter Kjellerstedt
e9e708f923 populate_sdk_ext.bbclass: No longer needed to clean away conf/sanity_info
Since the sanity_info file has moved from the conf directory to the
cache directory, there is no longer any need to clean it away
explicitly in clean_esdk_builddir() since the whole cache directory is
already cleaned away anyway.

(From OE-Core rev: 40c30990e1be72130819c040fe471e2bdc0c6e7d)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-10 21:18:22 +00:00
Richard Purdie
fd8d01134a populate_sdk_ext: Fix confusing failure message
This stack trace is horrible and obscurs the real failure message, avoid that.

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:copy_buildsystem(d)
     0003:
File: 'build/meta/classes/populate_sdk_ext.bbclass', lineno: 446, function: copy_buildsystem
     0442:    sdk_ext_type = d.getVar('SDK_EXT_TYPE')
     0443:    if (sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative) and not sdk_include_nativesdk:
     0444:        # Create the filtered task list used to generate the sstate cache shipped with the SDK
     0445:        tasklistfn = d.getVar('WORKDIR') + '/tasklist.txt'
 *** 0446:        create_filtered_tasklist(d, baseoutpath, tasklistfn, conf_initpath)
     0447:    else:
     0448:        tasklistfn = None
     0449:
     0450:    if os.path.exists(builddir + '/cache/bb_unihashes.dat'):
File: 'build/meta/classes/populate_sdk_ext.bbclass', lineno: 182, function: create_filtered_tasklist
     0178:        # Clean out residue of running bitbake, which check_sstate_task_list()
     0179:        # will effectively do
     0180:        clean_esdk_builddir(d, sdkbasepath)
     0181:    finally:
 *** 0182:        os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf')
     0183:
     0184:python copy_buildsystem () {
     0185:    import re
     0186:    import shutil
Exception: FileNotFoundError: [Errno 2] No such file or directory: 'tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/sdk-ext/image//opt/poky/3.0/conf/local.conf.bak' -> 'tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/sdk-ext/image//opt/poky/3.0/conf/local.conf'

(From OE-Core rev: 5115d2616b0847d521ccbecf1fbbf1f695a04fea)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-06 14:48:25 +00:00
Mark Hatle
10cdc3b27d populate_sdk_ext.bbclass: Make integrated buildtools optional
If the host system is expected to have enough capabilities that the
buildtools-tarball is not required, we don't need to bundle it.

This can save some significant space, especially when using with a minimal
eSDK.

minimal eSDK - core-image-minimal-qemux86-64

with buildtools-tarball     - 34 MB installer - 281 MB installed
without buildtoools-tarball - 11 MB installer -  48 MB installed

(From OE-Core rev: e2b215054d11527a05210066b9bfd7d9a2054e01)

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Richard Purdie
e25ee9fe28 populate_sdk_ext: Fix for hash equiv
Write out the hash equiv cache file into any eSDK so that it doesn't rely
on having to call the hash server for the basic data requests.

(From OE-Core rev: 05c6842e747261b3350d6325e238429cf8728ca0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Jaewon Lee
88db9e41c1 populate_sdk_ext: Introduce mechanism to keep nativesdk* sstate in esdk
When doing a devtool build-sdk from within an esdk all nativesdk
components would be rebuilt. This patch introduces SDK_INCLUDE_NATIVESDK
flag to toggle the inclusion of nativesdk packages when creating the
esdk sstate

Currently locked-sigs.inc is generated during do_sdk_depends which
doesn't pull in nativesdk packages. Generating another locked-sigs.inc
in do_populate_sdk_ext and pruning it to only nativesdk* packages by
using a modified version of the already existing function
prune_locked_sigs and merging it with the current locked-sigs.inc
Also adding SDK_INCLUDE_NATIVESDK tasklistfn to the logic surrounding
setting tasklist file to not prune esdk sstate during creation

[YOCTO #13261]

(From OE-Core rev: d046afd12e1c209b29dca6ba402b9aa14680c5ce)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 15:27:02 +01:00
Manjukumar Matha
5397a7acd8 populate_sdk_ext.bbclass: Fix the derivate sdk build error from an published eSDK
When you install the eSDK in publish mode and then try to build a
derivative sdk using devtool build-sdk, the following error happens

| NOTE: Generating sstate task list...
| NOTE: Generating sstate-cache...
| NOTE: Generating sstate-cache...
| DEBUG: Python function copy_buildsystem finished
| NOTE: Executing install_tools ...
| DEBUG: Executing shell function install_tools
| install: cannot stat /core/meta/files/ext-sdk-prepare.py':
No such file or directory

This patch will fix the error by installing ext-sdk-prepare.py in
publish mode, so that derivative sdk can be generated.

(From OE-Core rev: aab4089a57a655ca5a7792f05eb5d8dcdb934ae1)

Signed-off-by: Manjukumar Matha <manjukumar.harthikote-matha@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:06 +00:00
Khem Raj
801745d918 populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
Some distros use site.conf to emit certain variables which are important
for eSDK e.g. DISTRO with out which eSDK will not be able to ger right
metadata when it tries to build

(From OE-Core rev: 95659bed3f6f3216b346f70cfc9ffae9788c0fc1)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-20 14:53:55 +00:00
Jens Rehsack
2499589fb6 populate_sdk_ext.bbclass: avoid populating tools twice
When running `bitbake -c populate_sdk_ext ...` for an image
recipe inheriting populate_sdk_qt5, `nativesdk-qemu-helper` is
deployed via `nativesdk-packagegroup-sdk-host`, which already
contains some of the tools (by name: "oe-find-native-sysroot
runqemu*").

To avoid error like:
    Configuring nativesdk-packagegroup-qt5-toolchain-host.

    NOTE: Running intercept scripts:
    NOTE: > Executing update_gio_module_cache-nativesdk intercept ...
    NOTE: + [ True = False ]
    + qemu-x86_64 -r 3.2.0 -E LD_LIBRARY_PATH=/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/fischer-poky-linux/updatable-app-image/1.0-r0/sdk-ext/image/opt/mops/2.5+snapshot/sysroots/x86_64-pokysdk-linux/usr/lib:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/fischer-poky-linux/updatable-app-image/1.0-r0/sdk-ext/image/opt/mops/2.5+snapshot/sysroots/x86_64-pokysdk-linux/lib -L /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/fischer-poky-linux/updatable-app-image/1.0-r0/sdk-ext/image /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/fischer-poky-linux/updatable-app-image/1.0-r0/sdk-ext/image/opt/mops/2.5+snapshot/sysroots/x86_64-pokysdk-linux/usr/libexec/nativesdk-gio-querymodules /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/fischer-poky-linux/updatable-app-image/1.0-r0/sdk-ext/image/opt/mops/2.5+snapshot/sysroots/x86_64-pokysdk-linux/usr/lib/gio/modules/

    NOTE: Executing copy_buildsystem ...
    DEBUG: Executing python function copy_buildsystem
    NOTE: Generating sstate task list...
    NOTE: Generating sstate-cache...
    NOTE: Generating sstate-cache...
    DEBUG: Python function copy_buildsystem finished
    NOTE: Executing install_tools ...
    DEBUG: Executing shell function install_tools
    Traceback (most recent call last):
      File "/home/sno/gpw-community-bsp/sources/poky/scripts/lnr", line 21, in <module>
	os.symlink(target, linkname)
    FileExistsError: [Errno 17] File exists: '../../../../layers/poky/scripts/oe-find-native-sysroot' -> '/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/fischer-poky-linux/updatable-app-image/1.0-r0/sdk-ext/image//opt/mops/2.5+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/oe-find-native-sysroot'
    WARNING: exit code 1 from a shell command.
    DEBUG: Python function do_populate_sdk_ext finished

only add those tools unpackaged to the deployment which are
still missing.

(From OE-Core rev: 3274819ce3bb64316853c6f0bb7840c38635c26c)

Signed-off-by: Jens Rehsack <sno@netbsd.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-10 12:47:35 +01:00
Jeremy Puhlman
90d6d998ce Only add uninative and checksum if inherited uninative class
The checksum value is only calculated if the uninative class is
inherited, so check for inherit before adding it to local.conf

(From OE-Core rev: 3b5b832589d943700b273e3a4d83561be0c47f36)

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-18 11:07:57 +01:00
Damien Riegel
7d3ad1405c populate_sdk_ext.bbclass: fix corebase identification
When generating the extended SDK, there is a copy step where this class
goes through the layers and other stuff that have been copied to
generate the SDK. The corebase; ie. the folder that contains the core
layer 'meta' is treated in a special way. Unfortunately in our tree, we
have:

  sources/meta/meta
           |     `- core layer
           `------- corebase

In populate_sdk_ext's copy_buildsystem, the heuristic to determine which
element of the list returned by copy_bitbake_and_layers is corebase is
fooled by such layout.

In copy_bitbake_and_layers, corebase is already handled specifically and
reliably, so we should let that function tell us which folder is
corebase instead of trying to determine it.

To do so, change the return type of copy_bitbake_and_layers to a tuple
that contains (corebase, copied_layers). It also simplifies the code on
the caller side.

(From OE-Core rev: 5368bc5d0d3606198b93e877bcafcd77bb5f4fd1)

Signed-off-by: Damien Riegel <damien.riegel@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-18 11:07:57 +01:00
Peter Kjellerstedt
144faeb10c classes: Use ${MACHINE_ARCH} instead of ${MACHINE} for stamp-extra-info task flag
Without this change, there will be two sstate index files in
tmp/sstate-control for any machine that contains a dash in the
name.

(From OE-Core rev: 29e7799bdb3773c40492e01448e0c614ed44583d)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-10 09:10:36 +01:00
Ross Burton
2f56415059 populate_sdk: install UTF-8 locales in SDKs
As glibc 2.27 can't read older locale-archives, SDKs using glibc 2.27 on hosts
using glibc earlier than 2.27 won't be able to find any locales, so bitbake
won't start and Python can't use UTF-8.

So by default install all locales into the SDK.  Special-case Extensible SDKs by
installing no locales as they ship glibc in a buildtools, and that will have the
locales.

Locale installation requires cross-localedef, so add that to DEPENDS.

Also remove the explicit en_US addition in buildtools-tarball as it is now
redundant.

(From OE-Core rev: 96896568d197cd06302713c24c0f7d91bfaea6c1)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:18:47 +00:00
Richard Purdie
63c8c8fb60 populate_sdk_ext: Set cleandirs correctly
The current conflicting use of SDKDEPLOYDIR causes a race between do_populate_sdk
and do_populate_sdk_ext potentially causing the SDK to either go missing or the
build to fail.

(From OE-Core rev: 792cfbab488782a7bd610fc2078077d5497be4d1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 10:39:09 +00:00
Chang Rebecca Swee Fun
52c25b03ec classes/populate_sdk_ext: support wic in eSDK
Make 'wic' image creation tool/command available in eSDK
environment. This would allow eSDK users to manipulate
images within eSDK environment.

[YOCTO #12177]

(From OE-Core rev: 90df6758a9f8753c646b129aa912e3849bf4c987)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-13 10:15:20 +00:00
Robert Yang
eaad759ac8 populate_sdk_ext.bbclass: don't rename layers when failed
The previous code:
os.rename(sdkbasepath, temp_sdkbasepath)
try:
    foo
finally:
    os.rename(temp_sdkbasepath, sdkbasepath)

always renamed the path, it made the debug harder when error happened.
drop the "try: finally" makes the debug easier.

(From OE-Core rev: 32126512349d65f0dbc31196c4ec6e1a1147cf5e)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-06 22:32:22 +00:00
Richard Purdie
6c664a9713 populate_sdk_ext: Use prebuilt uninative tarball
For uninative to work, it relies on it being updated to new versions as
newer glibcs are built. This means the uninative generated by the current
build may not be as recent as the uninative that is being downloaded by
uninative.bbclass.

If this occurs, we can get symbol mismatch errors.

Ultimately, the sstate and the uninative versions need to match so we
should use the same tarball as uninative.bbclass is using, not the one
we built.

[YOCTO #12405]

(From OE-Core rev: a24c10b7bdab8aa960fdd3a58d2009f24344e579)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-11 21:46:38 +00:00
Paul Eggleton
3fde63363a devtool: ensure recipes devtool is working on are unlocked within the eSDK
Alongside reworking the way devtool extracts source, we now need to
ensure that within the extensible SDK where task signatures are locked,
the signatures of the tasks for the recipes being worked on get unlocked
at the right time or otherwise we'll now get taskhash mismatches when
running devtool modify on a recipe that was included in the eSDK such as
the kernel (due to a separate bug). The existing mechanism for
auto-unlocking recipes was a little weak and was happening too late, so
I've reimplemented it so that:
(a) it gets triggered immediately when the recipe/append is created
(b) we avoid writing to the unlocked signatures file unnecessarily
    (since it's a global configuration file) and
(c) within the eSDK configuration we whitelist SIGGEN_UNLOCKED_RECIPES
    to avoid unnecessary reparses every time we perform one of the
    devtool operations that does need to change this list.

Fixes [YOCTO #11883] (not the underlying cause, but this manifestation
of the issue).

(From OE-Core rev: 4e9a0be32fc30fb87d65da7cd1a4015c99533aff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:29 +01:00
Richard Purdie
d009306adb populate_sdk_ext: Add BB_SERVER_TIMEOUT to SDK_LOCAL_CONF_BLACKLIST
Whilst this should work we see failures in testsdkext at the moment when
this is set. Add this to the blacklist for now until we can fix these issues
meaning we can at least test BB_SERVER_TIMEOUT in other scenarios.

Bug 119733 has been opened to track this.

(From OE-Core rev: 88cb174680cd37b5afb2b79e8248979895c35312)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24 13:48:10 +01:00
Saul Wold
d98a2f4df0 populate_sdk_ext: Add variable to indicate running in eSDK
This allows for other scripts to know that they are being executed in
the context of the eSDK in order to provide different behaviour as
needed.

[YOCTO #11155]

(From OE-Core rev: ebdd9b5a19eb6fb6c3249be4f0e2480a3de2aa8f)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:37 +01:00
Richard Purdie
0ad65be8bf oe-init-build-env-memres: Drop it
With the new server structure we no longer need this separate
environment init script. Just set BB_SERVER_TIMEOUT to be greater
than zero and bitbake will remain in memory and the UI will auto-reconnect
to it.

Also clean out the old shutdown code from oe-init-build-env which
also doesn't make sense now.

(From OE-Core rev: a17724d687f9c04dc21f2f369da2b492f7119ba0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Francisco Pedraza
1b804d7944 meta/classes/populate_sdk: Adds support for generating eSDK manifest files
Add get_extra_sdk_info to reuse code in buildhistory

The functionalities to generate SDK and eSDK manifest files are different,
the SDK comes from package information and the eSDK comes from sstate artifacts.
Only execute write_sdk_{host, target}_manifest when is on populate_sdk class.

Adds new functions write_sdk{host, target}_ext_manifest to execute on postprocess
in populate_sdk_ext because at the end we have all the sstate artifacts to
generate the manifest.

[YOCTO #9038]

(From OE-Core rev: 25ad7ed6f7bb0c931b404bda09576323200d093d)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 23:01:23 +01:00
Richard Purdie
e4e8435886 populate_sdk_ext: Avoid build failures where sstate was used
If sstate was used to populate the build and one of the universal-4.8
or universal-4.9 mirror urls was used, the sstate checks during eSDK
construction could fail as it would zero out the SSTATE_MIRRORs
variable.

Use the same mirrors variable setting as the eSDK would end up using
to perform the checks to avoid this.

(From OE-Core rev: 6b9e8b780dcd8d5ffba3df35cfe41674413ee26d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-16 14:08:28 +01:00
Paul Eggleton
9e267ad173 classes/populate_sdk_ext: work around runqemu behaviour within the eSDK
Currently, in order to figure out variable values when run within the
eSDK, runqemu does not use the standard SDK method nor is it able to run
bitbake (since the eSDK environment isn't initialised like the normal
OE build environment). runqemu really ought to be fixed, but the quick
workaround is to set DEPLOY_DIR_IMAGE in the environment so that runqemu
can find image files.

Fixes [YOCTO #10447].

(From OE-Core rev: 1ef833b6393366a10f4bb65df89725ad65761386)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19 10:38:37 +01:00
Richard Purdie
045d01c63d populate_sdk_ext: Add do_addto_recipe_sysroot to BB_SETSCENE_ENFORCE_WHITELIST
Without this, eSDK builds are failing due to qemu-helper-native's dependency on this
task. It makes sense to allow this to execute in eSDK contexts (its a non-sstate task
intentionally).

(From OE-Core rev: 3e8ade8c0772c4492efd93824f78cb043281d235)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-14 09:47:09 +01:00
Mikko Ylinen
a84a323e4d classes/populate_sdk_ext: reset TCLIBCAPPEND to get consistent TMPDIR
populate_sdk_ext sets TMPDIR to a known static value with '/tmp' directory
name and that name is hard coded in a few places (e.g., in
meta-environment-extsdk.bb that writes the eSDK environment variables).

Distros that do not reset TCLIBCAPPEND (poky does) end up getting
TMPDIR = /tmp-${TCLIBCAPPEND} via defaultsetup.conf and that breaks
the functionality in eSDK that expects everything is in /tmp.

To get TMPDIR consistent, we also need to reset TCLIBCAPPEND in
populate_sdk_ext.bbclass.

Fixes: [YOCTO #11298]

(From OE-Core rev: 9ec29153f279bb9e1dbcddc2c66e00fdbe3fd6e9)

Signed-off-by: Mikko Ylinen <mikko.ylinen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-14 09:47:08 +01:00