Commit Graph

1228 Commits

Author SHA1 Message Date
Richard Purdie
a2db4fa127 base/bitbake.conf: Enable pseudo path filtering
This is a pretty big change to the way pseudo operates when used in OpenEmbedded.
Normally, pseudo monitors and logs (adds to its database) any file created or
modified whilst in a fakeroot environment. There are large numbers of files
we simply don't care about the permissions of whilst in that fakeroot context,
for example ${S}, ${B}, ${T}, ${SSTATE_DIR}, the central sstate control
directories,

This change uses new functionality in pseudo to ignore these directory trees,
resulting in a cleaner database with less chance of "stray" mismatches if files
are modified outside pseudo context. It also should reduce some overhead from
pseudo as the interprocess round trip to the server is avoided.

There is a possible complication where some existing recipe may break, for
example, we found a recipe which was writing to "${B}/install" for
"make install" in do_install and since we listed ${B} as not to be tracked,
there were errors trying to chown root for files in this location.

This patch fixes a few corner cases in OE-Core when used with this new
ignore list:

* The archiver directory matched a "${WORKDIR}/deploy*" pattern so was renamed
  to something else since that directory does need its root permissions
* The ${S} and ${B} ignoring is conditional on them being different to ${WORKDIR}
* package_write_* task output (the debs/rpms/ipks) are now owned by the build
  user so we don't want the file ownership information in the hashequiv outhash
  calculation even if they are built under pseudo.
* The fontcache postinstall intercept is run under qemu outside of pseudo context
  so delete files it may delete up front where pseudo can see this.
* SSTATE_DIR is in PSEUDO_PATHS_IGNORE, which is in FAKEROOTENV which is cached
  by bitbake. We therefore need to trigger reparsing if this changes, which means
  SSTATE_DIR can be in BB_HASHBASE_WHITELIST but not BB_HASHCONFIG_WHITELIST.
  Rework the variables to handle this. This otherwise breaks some of our sstate
  tests in oe-selftest.
* Ignore the temp directory wic uses for rebuilding rootfs.

(From OE-Core rev: ad8f5532ffaead9a5ad13e1034fe9e5e1b7979f4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-07 11:18:38 +01:00
Ross Burton
c2a11e18c8 utils: respect scheduler affinity in cpu_count()
cpu_count() returns multiprocessing.cpu_count() but that is simply returns
os.cpu_count() so we could use that directly.

However this returns the number of CPUs on the host, not the number of
usable CPUs on the host. If the user is using scheduler affinity then
the number of usable CPUs may be less, so when determining how many cores
we can use check the affinity instead.

(From OE-Core rev: e2e8ccbe410b2f38bcd9525982b2261cf71aab60)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-30 15:01:51 +01:00
Naveen Saini
3204982fd7 lib/oe/recipeutils.py: add support for BBFILES_DYNAMIC
Instead of relying on value of BBFILES from bitbake, devtool
parses the layer.conf because the layer might not be in bblayers.conf.
And it currently does not consider the value of BBFILES_DYNAMIC because
of which recipes, in paths defined by BBFILES_DYNAMIC, upgraded
using devtool end up in wrong location.

Include the code from bitbake to append values to BBFILES based on
what is in BBFILES_DYNAMIC too.

(From OE-Core rev: c7bbb98ea8ccd3568dd8bded6e404e2f781e6841)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-27 08:28:03 +01:00
Richard Purdie
b70c39767c lib/package_manager: Fix missing imports
The package_manager code rearranging had some issues with module imports that
were now missing. Fix all the ones I could spot from quick inspection.

(From OE-Core rev: 287eccd7af7d97604ca68d456c23655fd1b6c40b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-01 19:57:49 +01:00
Joshua Watt
20e9df5721 lib/oe/reproducible.py: Fix git HEAD check
The check for a git HEAD still wasn't quite correct because it was using
the .git directory as the current working directory. Instead, it should
be passed as the --git-dir argument when running git. Running `git
rev-parse HEAD` in a .git directory with no HEAD reports 'HEAD' and
exits with success but then 'git log' will fail, which is not what we
want.

(From OE-Core rev: cdbd47dd7e1657b91b65a0940b7cbf119764240f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-29 11:37:31 +01:00
Richard Purdie
4e3240220b Revert "lib/oe/log_colorizer.py: add LogColorizerProxyProgressHandler"
This reverts commit 312fb3c86a3d84e60867b132666c01859f73ceb2 as this
wasn't meant to merge as yet.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-29 11:35:51 +01:00
Chris Laplante
1739fee633 lib/oe/log_colorizer.py: add LogColorizerProxyProgressHandler
This progress handler intercepts log output, stripping any ANSII color
escape codes. Then the stripped output is fed to the underlying progress
handler which will render the progress bar as usual.

(From OE-Core rev: 312fb3c86a3d84e60867b132666c01859f73ceb2)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-29 10:12:31 +01:00
Fredrik Gustafsson
f5f6ae2986 deb: Move package manager to its own dir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

RP: Fixes to parse/build
(From OE-Core rev: 510d5c48c0496f23a3d7aede76ea8735da2d371d)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
5bc67f5502 ipk: Move package manager to its own dir
This is part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

RP: Fixes to parse/build
(From OE-Core rev: 3ef5a3c885e1010cddfe7eba1cd3728f15270d78)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
c5bea36b72 rpm: Move package manager to its own dir
This is part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

RP: Fixes to parse/build
(From OE-Core rev: 8b776ed9ed291dd8e112621561762449c7eb5ee2)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
635305fe27 deb: Move sdk to its own dir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: 2b305d5f48be84b6362be850d6e0b2bd7f4691ce)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
bfa8f4c098 ipk: Move sdk to its own dir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: 3f9cec50065eec5a02ffcc8ccc2986f2027b44b5)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
e9e2e32999 rpm: Move sdk to its own dir
This is part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: 75066e78d92a23516fd9c6d538c4f991d1504839)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
6fd8b17748 deb: Move rootfs to its own dir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: abadf053371ce863bf21b4a9474eb61761545de1)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
873aee855f ipk: Move rootfs to its own dir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

RP: Fixes to parse/build
(From OE-Core rev: 67fa086589bae484a9beca50b627b007766dcb93)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
b8d3d22ccd rpm: Move rootfs to its own dir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: e4f07444276a88f336d5fe5e60688fcfb2f22df7)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
9129dfe523 deb: Move deb manifest to its own subdir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: f8ee8bc737a982001b9fd0ad441495a52f12e9b4)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
59397b3a01 ipk: Move ipk manifest to its own subdir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: 405cd8560fed2e05fc82919d728c42516793cc0f)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
a0416ca06e rpm: Move manifest to its own subdir
This is a part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: 87a1c8ee406f73e53888df3b682e8a5f0f610c2f)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:10 +01:00
Fredrik Gustafsson
34dc8571d2 package_manager: Move to package_manager/__init__.py
This is part of a refactor that will split the package manager
code so that it's possible to use other package managers in other
layers.

(From OE-Core rev: 66ccc7a228bf73df0a4dd846bf2c8e99eaa79580)

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:09 +01:00
Naoto Yamaguchi
49e92f8d6d patch.py: Change to more strictly fuzz detection
When applying patch subject is including a " fuzz ", do_patch_qa detect
Fuzz.

After the patch is applied, the following log message appears.

  Applying: meson: treat all fuzz cases as unit tests

The current Fuzz detection checks for the presence of "fuzz" in this
log message.  The log in this example will be treated as Fuzz,
despite its success.

This patch change to more strictly fuzz detection.
if log message is including " fuzz " and "Hunk " in log message,
it will be treated as Fuzz.

(From OE-Core rev: a8605c66ef5afe7c3583366781dfd90fe3526398)

Signed-off-by: Naoto Yamaguchi <wata2ki@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 19:58:09 +01:00
Joshua Watt
5843460575 lib/oe/reproducible: Fix error when no git HEAD
Fixes an error that occurs when attempting to get the timestamp of the
latest commit when there is no HEAD in the git repository. The easiest
way to trigger this condition is to use the 'subdir=' option when
specifying a 'git://' SRC_URI.

(From OE-Core rev: a64caca5b5dbe4a76acd0b5709b2c3e75b245863)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-22 12:45:56 +01:00
Joshua Watt
ebb40b027c classes/reproducible: Move to library code
Moves most of the python code used for dealing with the source date
epoch to library code.

(From OE-Core rev: a7ede90955bc0c8bec1cbb3cab498ef2583b2f4e)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-22 12:45:56 +01:00
Tim Orling
c9ba6e7bfe lib/oe/recipeutils.py: add AUTHOR; BBCLASSEXTEND
If you try to create a plugin for recipetool that
adds the AUTHOR field, it is impossible to put it
in the recommended position [1] without adding to the
recipe_progression variable.

While we are at it, also add BBCLASSEXTEND at the
end, as also recommended by [1].

[1] http://www.openembedded.org/wiki/Styleguide

(From OE-Core rev: d687c5b7b10b3decdd80d5c2fd61072a87f061f2)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-12 22:39:41 +01:00
Damian Wrobel
061eb6f501 rootfs: do not let ldconfig to create symlinks
Adds -X option to ldconfig to not create symlinks
as part of the ldconfig cache creation process.

It is much better to keep all needed symlinks as part of
of the package then let ldconfig do the job.

In the case symlink was created by ldconfig and later on the
package got removed the dangling link will remain on the
filesystem. Not to mention that such a symlink is orphaned
(e.g. rpm -qf <file-path> will not show the package the symlink
is part of).

It also align the behaviour with libc-musl where the ldconfig
is not invoked by default.

(From OE-Core rev: ad4ff7c18cc6a5c15a0525149dbe2dbd5ae39129)

Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-03 11:38:24 +01:00
Joshua Watt
48a883bb6e sstatesig: Account for all dataCaches being passed
Bitbake now passes all the dataCaches to the taskhash API, so use this
to correctly filter mcdepends.

[YOCTO #13724]

(From OE-Core rev: 749731a420fb905b6af97ce2909f06b2bcd14fe2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-15 14:53:46 +01:00
Richard Purdie
eb226b897f buildhistory: Add simplistic file move detection
We'd like to use buildhistory more during patch review however its
proving hard, particularly where whole subtrees of files move,
such as a kernel version upgrade, or where a software module moves
include directory.

This adds file rename matching which covers our common case of library
moves, kernel upgrades and more.

A new test case is also added so that someone in the future can change
the code and test the logic is still doing the expected things.

(From OE-Core rev: 791ce304f5e066759874beac0feef5ee62a1c255)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-15 14:53:45 +01:00
Rasmus Villemoes
6710572f90 glibc: move ld.so.conf back to main package
There are cases where one doesn't want ldconfig on target (e.g. for
read-only root filesystems, it's rather pointless), yet one still
needs ld.so.conf to be present at image build time:

When some recipe installs libraries to a non-standard location, and
dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the
ld.so.conf containing the

  include /etc/ld.so.conf.d/*.conf

stanza to get those other locations picked up.

So change the packaging logic so that there's always an ld.so.conf
present when the build-time ldconfig runs.

The ld.so.conf and ld.so.conf.d/*.conf files don't take up much
room (at least not compared to the 700K binary ldconfig), and they
might be needed in case ldconfig is installable, so leave them
alone.

In case of a read-only rootfs, one could add some logic to remove them
if one really wants to shave those few dozens of bytes off.

While here, fix typos in the bb.note (add spaces) so one can just
copy-paste the line from the log-file and redo the command.

(From OE-Core rev: a4cdda012f613d8d80203b9f5fc737d8511d16ce)

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-09 13:13:59 +01:00
Alexander Kanavin
cae9a6c9c8 apt: update to 1.8.2.1
I took the opportunity to rewrite the recipe from scratch; there was just too much baggage in it.

(From OE-Core rev: f058272de9cba188d96940c8c921cf31727fe4d1)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:31 +01:00
Martin Jansa
41c8e1b706 Revert "lib/oe/patch: fix handling of patches with no header"
* This reverts commit d9971f5dc8eb7de551fd6f5e058fd24770ef5d78.

* With the missing Subject line fixed in GitApplyTree.prepareCommit()
  we should be able to revert, the fix which was trying to help it by
  parsing GitApplyTree.patch_line_prefix ("%% original patch:") also
  from Subject line, now GitApplyTree.patch_line_prefix should always
  end on separate line which is then skipped when copying the lines to
  resulting patch, see original commit message from Paul:

    lib/oe/patch: fix handling of patches with no header

    If a patch applied by a recipe has no header and we turn the recipe's
    source into a git tree (when PATCHTOOL = "git" or when using devtool
    extract / modify / upgrade), the commit message ends up consisting only
    of the original filename marker ("%% original patch: filename.patch").
    When we come to do turn the commits back into a set of patches in
    extractPatches(), this first line ends up in the "Subject: " part of
    the file, but we were ignoring it because the line didn't start with the
    marker text. The end result was we weren't able to get the original
    patch name. Strip off any "Subject [PATCH x/y]" part before looking for
    the marker text to fix.

    This caused "devtool modify openssl" followed by "devtool update-recipe
    openssl" (without any changes in-between) to remove version-script.patch
    because that patch has no header and we weren't able to determine the
    original filename.

(From OE-Core rev: d9e56db415d386447a299dd633b10f1eda0dd401)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:29 +01:00
Martin Jansa
71ef6bbb85 lib/oe/patch: GitApplyTree: save 1 echo in commit-msg hook
* also remove the extra blank lines which is often added to patches
  when refreshed with devtool (GitApplyTree.patch_line_prefix lines
  are ignored when refreshing .patch files, but newly added blank
  lines aren't - the leading blank line wasneeded for patches with
  just the subject line (to prevent the GitApplyTree.patch_line_prefix
  line ending appended to the commit summary), but we can add it
  in prepareCommit instead

(From OE-Core rev: c50c0d6144ad290168167ccef948c7b4ffc9665a)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:29 +01:00
Martin Jansa
15ee96a245 lib/oe/patch: prevent applying patches without any subject
* this was discovered with
  $ devtool finish --force-patch-refresh
  where it was removing some patches and replacing them with
  patch in filename called "patch:"

  e.g. this .patch file:
  311067d2d8/dynamic-layers/rust-layer/recipes-browser/firefox/firefox/fixes/fix-camera-permission-dialg-doesnot-close.patch
  confuses devtool which results to create new .patch file called "patch:"

  $ devtool finish --force-patch-refresh firefox meta-browser
  NOTE: Starting bitbake server...
  WARNING: Host distribution "ubuntu-20.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
  Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
  Loaded 2480 entries from dependency cache.
  Parsing recipes: 100% |#################################################################################################################################################################################################################################| Time: 0:00:00
  Parsing of 1718 .bb files complete (1717 cached, 1 parsed). 2480 targets, 68 skipped, 0 masked, 0 errors.

  Summary: There was 1 WARNING message shown.
  INFO: Updating patch 0001-Bug-1554949-Fix-WebRTC-build-failure-with-newer-linu.patch
  ...
  INFO: Updating patch pre-generated-old-configure.patch
  INFO: Adding new patch patch:
  INFO: Updating recipe firefox_68.0esr.bb
  INFO: Removing file /OE/build/test-oe-build-time/poky/meta-browser/dynamic-layers/rust-layer/recipes-browser/firefox/firefox/fixes/fix-camera-permission-dialg-doesnot-close.patch
  INFO: Cleaning sysroot for recipe firefox...
  INFO: Leaving source tree /OE/build/test-oe-build-time/poky/build/workspace/sources/firefox as-is; if you no longer need it then please delete it manually

  this looked like incorrect parsing of the git format-patch
  files exported from workspace/sources (the git format-patch
  version of fix-camera-permission-dialg-doesnot-close.patch
  starts like this:

  $ head 0008-original-patch-fix-camera-permission-dialg-doesnot-c.patch
  From 37dfa11961b48024bedcfb9336f49107c9535638 Mon Sep 17 00:00:00 2001
  From: Takuro Ashie <ashie@clear-code.com>
  Date: Mon, 20 Aug 2018 10:16:20 +0900
  Subject: [PATCH 08/34] %% original patch:
   fix-camera-permission-dialg-doesnot-close.patch

  so first I've modified GitApplyTree.extractPatches() to be able to
  parse the original patch name correctly even in this case where subject
  is wrapped, but then it still wasn't right, because we ended with
  correctly named .patch file, but all we could use for Subject line
  was the name of the original .patch file (instead of the Subject
  from metadata commit which introduced this .patch files as some other
  .patch files get when refreshed with devtool.

  In the end the issue happens even sooner in GitApplyTree.prepareCommit()
  where it correctly found the Subject from metadata commit, but then
  didn't apply it when there weren't any other outlines from patch headers.

(From OE-Core rev: 0a8252f0cb13e3dc16f70b984f9f98b845b163de)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:29 +01:00
Richard Purdie
97377e284d lib/classextend: Drop unneeded comment
This was leftover debug which can be removed.

(From OE-Core rev: 7a578c760aa32f62bf39be7289de1516ad97e92e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-30 12:32:47 +01:00
Joe Slater
e620119715 terminal.py: do not stop searching for auto
If a terminal fails to spawn() we should continue looking.
gnome-terminal, in particular can be present but not start.

(From OE-Core rev: 6e4babdeee38d32002a4c9129e77466ae4156dd7)

Signed-off-by: Joe Slater <joe.slater@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 11:04:47 +01:00
Richard Purdie
edbf8d3999 multilib/recipes: Use new RecipePostKeyExpansion event
There are issues with multilib due to the ordering of events where some
functions see the remapped multilib dependencies and some do not. A significant
problem is that the multilib class needs to make some changes before key expansion
and some afterwards but by using existing event handlers, some code sees things
in a partially translated state, leading to bugs.

This patch changes things to use a new event handler from bitbake which makes the
ordering of the changes explcit.

The challenge in doing this is that it breaks some existing anonymous python and
dyanmic assignments. In some cases these used to be translated and no longer are,
meaning MLPREFIX has to be added. In some cases these are now translated and the
MLPREFIX can be removed.

This change does now make it very clear when MLPREFIX is required and when it is
not, its just the migration path which is harder. The patch changes the small number
of cases where fixes are needed.

In particular, where a variable like RDEPENDS is conditionally extended (e.g.
with an override), MLPREFIX is now required.

This patch also reverts:
base: Revert 'base.bbclass: considering multilib when setting LICENSE_EXCLUSION'

This reverts 6597130256a1609c3e05ec5891aceaf549c37985 as the changes
to multilib datastore handling mean its no longer necessary.

(From OE-Core rev: b3fda056a674889cd9697e779de023d4f993d3ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 10:47:05 +01:00
Konrad Weihmann
9ac51e479d lib/oe/recipeutils.py: passthrough of FETCHCMD
let FETCHCMD_* settings be passed to upstream check functions from
get_recipe_upgrade_status. It enables using different values for
the fetching tools, as otherwise always the defaults are used.
E.g. use different timeout or retry values for wget.
This is especially useful for remote server that do request-limiting (429)
or that are simply unavailable (500, 504) and can speedup upgrade-check
for a larger set of recipe significantly

(From OE-Core rev: f69347744241afea19f155d9ff5115c675e9043d)

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-19 22:57:26 +01:00
Changqing Li
e9dd7659b4 lib/oe/package_manager: update default rpm config %_prefer_color
* %_prefer_color is used by rpm to determine which color's ELF file
is preferred to be installed.

Here are file colors:
0 is unknown or other
1 is Elf32
2 is Elf64
4 is MIPS64 n32 (this color is added by oe-core's patch)

if default value set to 7, all colors are preferred color, always
be last-in-wins.

For this scenario, when we have 64bits python3 installed first,
then install 32bits python3 later, 64bits python3 will be overwrited,
and sys.path will point to /usr/lib, not /usr/lib64, this may cause
some python3 modules not work. so fixed by remove setting of default
value 7, and use default value 2 of rpm

* other distro like fedora also use the default %_prefer_color 2

(From OE-Core rev: 56fa74497393a10f751d01c600c1936761e00294)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-09 18:57:21 +01:00
Richard Purdie
a98d9c5b71 image/packagegroup: Remove PACKAGE_GROUP_*, long since deprecated
This was deprecated in 2014 so we can safely remove the old code now.

(From OE-Core rev: fe1b79188cbe8159a0950f0c02d7f476a6694a04)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-29 15:02:19 +01:00
Martin Jansa
c23d6f7799 prservice.py: fix do_package with newer Python in Ubuntu 20.04
* with Ubuntu 20.04 which is using python 3.8 I'm seeing a lot of errors like:
  ERROR: libxml2-2.9.10-r0 do_package: Can NOT get PRAUTO, exception No module named '_sysconfigdata'
  not sure what caused this from python 3.8, but this seems to work
* PRserv is enabled with:
  PRSERV_HOST = "localhost:0"

(From OE-Core rev: 4b26eaf7152fb712aba47a0c746333578f58ee8d)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-05 11:46:37 +01:00
Jan Luebbe
f07c7dfc33 lib/oe/package_manager: don't try to rm /var/lib/opkg
As opkglibdir starts with a /, os.path.join will ignore
self.target_rootfs, leading to an attempt to remove /var/lib/opkg.

This only fails if it exists on the host, explaining why this remained
undiscovered for long.

(From OE-Core rev: fc974977cea389f54e7fc7de7b1c8fd3d8bafe58)

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-02 15:24:59 +01:00
Jan Luebbe
cb180dc45f lib/oe/package_manager: avoid installing provided packages via apt
If there already is a package providing (and conflicting against)
packages what should be installed, apt will try remove the conflicting
package (target-sdk-provides-dummy) and any that depend on it (like apt
and dpkg). This usually fails because of the protection of essential
packages. In that case, no -dev/-dbg packages are installed to the SDK.

Avoid this problem by checking which packages are already provided and
removing them from the list to be installed. Also sort the list to make
it easier to read when debugging.

(From OE-Core rev: 3ffb339dd55f8ca7c952fd3390608510f772e19f)

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-02 15:24:59 +01:00
Jan Luebbe
2864210236 lib/oe/package_manager: collect provided package names when using debs
This is needed for a later change to avoid installing packages which are
already provided by an installed package.

(From OE-Core rev: ad72dfaaa2caf2c39d033dc1682f0bbbbe45dbbd)

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-02 15:24:58 +01:00
Jan Luebbe
e1aabb2cae lib/oe/package_manager: fix handling of last package
In commit 7d214b34e11dc57316ed5c1c7747c4601286f6d2, only the code in the
for loop was modified to store the pkgarch value. The code used if there
was no empty line at the end was not modified.

Instead of fixing the duplicated code, remove it and just make sure that
a final empty line is processed.

(From OE-Core rev: a7b93c695b23d015607b179d98526b9b14c03d45)

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-02 15:24:58 +01:00
Jan Luebbe
440be55987 lib/oe/package_manager: make sure to not remove packages in apt install
apt install can decide to remove already installed packages if there are
conflicts. Avoid this by explicitly specifying --no-remove. This will
then cause a "E: Packages need to be removed but remove is disabled."
message.

(From OE-Core rev: 9605a488b55042add012e9aeef13ab3f4e70e6e5)

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-02 15:24:58 +01:00
Tim Orling
d5d634f682 lib/oe/utils.py: add get_host_compiler_version()
Add helper function to get the host compiler and version.
Do not assume compiler is gcc.

NOTE: cannot set env to d.getVar("PATH") as that does not contain
the session PATH which was set by environment-setup-... which
breaks the install-buildtools use-case

(From OE-Core rev: 88712929354ff9c876bb1e48b6f15c33af5f2bbc)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-01 11:44:24 +01:00
Peter Kjellerstedt
6a947a90ad prservice.py: Use "with" to control file handle lifetime
(From OE-Core rev: e7f025c87d595d2e37f83b75f11bedf2bce7bcc0)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-31 13:58:01 +01:00
Charles-Antoine Couret
ad366e7383 utils: fix gcc 10 version detection
Utils can not detect GCC 10 correctly due to wrong regex.
It generates this error "ERROR: Can't get compiler version from gcc  --version output"

Sub-version numbers should be 1 or more digits instead of 1 only.

(From OE-Core rev: 186fe4a3d390a52b87282c3e694ce3251e45ee78)

Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 12:05:31 +01:00
Richard Purdie
4bf1c0401a oe/recipeutils: Drop obsolete GITDIR reference
The fetcher GITDIR variable wase dropped a while back, drop the
obsolete reference to it (thanks Robert Day).

(From OE-Core rev: de035e687e26cef96e9b737c47bfc291bdfbea48)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 12:05:31 +01:00
André Draszik
0d7f24ce89 lib/oe/utils: allow to set a lower bound on returned cpu_count()
This will be needed for making xz compression more deterministic,
as xz archives are created differently in single- vs multi-threaded
modes.

This means that due to bitbake's default of using as many threads
as there are cores in the system, files compressed with xz
will be different if built on a multi-core system compared to
single-core systems.

Allowing cpu_count() here to return a lower bound, will allow
forcing xz to always use multi-threaded operation.

(From OE-Core rev: c6dda204e15c5b27559f7617b18a7b3ce604e2f8)

Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06 08:19:17 +00:00
Richard Purdie
fee9be0f65 abi_version/sstatesig: Introduce HASHEQUIV_HASH_VERSION
We've found we need a way to cause a change in signatures and move
to a new hash 'namespace' with hashequiv. This introduces a variable
which allows us to do this.

(From OE-Core rev: 1c2d5aeae2fa2dfc16bc60b6cf0b310f826337fd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-28 23:11:28 +00:00