Similar to rpm, use copies of the ipk/deb directories for rootfs construction.
This means the image creation code can no longer "see" recipes wich aren't in its
dependency chain which is good for a variety of reasons including determinism,
incompatible recipe (e.g. systemd/sysvinit) package conflicts and locking
performance.
(From OE-Core rev: c7c5f4065c102fde4e11d138fb0b6e25bffe0379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function is generic, tweak the variable names and move out the rpm specific
directory name to make it truly generic and reusable for deb/ipk.
(From OE-Core rev: dba876639b1fb8ea3ccb182c91e19966c4052115)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refactor the locale archive function from the SDK to also make it work during
general image creation. This reduces the size of the locales from 900MB to 220MB
in core-image-lsb-sdk.
The exception handling around subprocess was dropped as the standard subprocess
exception printing is better handled than the catchall exception.
(From OE-Core rev: 8ffd93bdb09b0a4a84b27dafcd684c6abba392ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After this replacement, the parent exception handling works so we don't
need subprocess wrapping with bb.error in the underlying functions.
The underlying contexts also have better module handling so the imports
can be cleaned up.
(From OE-Core rev: aa8260adf53139d776a2affe6118d28b295c1fab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As of commit 2c5c6e3ff we create multilib variants of intercept
hooks but we did not account for delay_to_first_boot variants.
This was covered up until commit a335e7867, but will now cause
an error.
(From OE-Core rev: 77f7c75481dceec36b7373f277c3bac811de9ef2)
Signed-off-by: Joe Slater <joe.slater@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously a warning was printed regardless of context and nature
of the failure, and because it was only a warning, it was mostly ignored.
Now, the following is considered when a failure happens:
1) whether we are installing packages into a target image, or populating
a SDK with host or target packages.
2) whether the failure was due to qemu not supporting the target machine.
Accordingly, warnings, notes, and failures are printed, and postponing
to first boot happens if possible.
(From OE-Core rev: a335e78672b1e1ae3ea6427f6a805218e513bb52)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was undocumented, and it's better to just always enable
full debug output, as this allows immediate generation of logs
with full diagnostics when things go not as expected.
Also, change the output of dnf from note to debug level; this
does not affect what is written to log file, but does reduce the
verbosity of bitbake -v.
(From OE-Core rev: 9128fd1396729a71b4832a597cf070c2be922d63)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If package management is disabled we remove the state and configuration for the
package manager after the rootfs has been populated. This list wasn't complete
and the DNF/RPM configuration files were left behind.
As we've added files to the list (and not just directories), expand the
backup/restore package management state code to handle this.
(From OE-Core rev: c8a6422457b4d29e6afd1e193b8b921287472137)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows running the intercepts when creating SDKs, which previously
wasn't possible, as SDK code does not use the rootfs class, and calls
into PackageManager methods directly.
(From OE-Core rev: f830388c5e9125f385a42acd7365d1235967b57c)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will allow handling postinst_intercepts when populating SDKs (which
use PackageManager class directly, and do not utilize RootFS class).
(From OE-Core rev: 9454fd328040fd58c981d028a74fcf181bde8e89)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will be useful when also moving postinst_intercept handling to
package manager class from rootfs class.
(From OE-Core rev: a4cd69bdd5b9dfa1125887f4d9038d41996e39c7)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, RPM_SIGN_PACKAGES is not defined. Add gpgcheck=0 to
oe-remote-repo.repo file, otherwise dnf will complain during
install operation on target
Note, RPM_SIGN_PACKAGES is set only when you inherit sign_rpm explicitly
(From OE-Core rev: 002a71eaa7606828c399972d8fd35e19e7b71929)
Signed-off-by: Manjukumar Matha <manjukumar.harthikote-matha@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously this was done only for rpm packages; now also ipk/deb scriptlet
failures are reported.
In the future this will become a hard error, but it can't yet happen
due to the legacy 'exit 1' way of deferring scriptlet execution to first boot which
needs a deprecation period.
(From OE-Core rev: a36671faf6e0b7623185b0e22814a786d5444592)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running bitbake -c populate_sdk <image_name>, it is expected that
packages matching SDKIMAGE_INSTALL_COMPLEMENTARY name mask (unless
declared in PACKAGE_EXCLUDE_COMPLEMENTARY) are installed to resulting
SDK. Underlying mechanism issues a package manager install call for set
of complementary packages. However the mechanism doesn't seem to inform
the user all too obviously in case the package manager command behind
install_complementary() method fails -- and since it is combined with
attempt_only=True option, user might end up wondering why several *-dev,
*-dbg packages are missing from resulting SDK.
Improve associated install() method behaviour in affected OpkgPM and
DpkgPM classes so that a problematic state of affairs becomes directly
obvious for bitbake user, resulting in shell output like:
WARNING: someimage-1.0-r0 do_populate_sdk: Unable to install packages.
Command '...' returned 1:
Collected errors:
* Solver encountered 1 problem(s):
* Problem 1/1:
* - package somepkg-dev-1.0-r0.x86 requires somepkg = 1.0-r0, but
none of the providers can be installed
*
* Solution 1:
* - allow deinstallation of someotherpkg-1.1-r1.x86
* - do not ask to install a package providing somepkg-dev
* Solution 2:
* - do not ask to install a package providing somepkg-dev
(From OE-Core rev: 2502bd591c37bf532d02dc6b37fc1e8b5224fb0a)
Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a package that does not generate a manifest due to using a
noexec rule, the package name should be printed so the problem can be
tracked down. With out the patch you get an error that makes it look
more like the package_manager is broken as shown below.
oe-core/meta/lib/oe/package_manager.py', lineno: 534, function: create_packages_dir
0530:
0531: for dep in rpmdeps:
0532: c = taskdepdata[dep][0]
0533: manifest, d2 = oe.sstatesig.find_sstate_manifest(c, taskdepdata[dep][2], taskname, d, multilibs)
*** 0534: if not os.path.exists(manifest):
0535: continue
0536: with open(manifest, "r") as f:
0537: for l in f:
0538: l = l.strip()
File: '/usr/lib/python3.5/genericpath.py', lineno: 19, function: exists
0015:# This is false for dangling symbolic links on systems that support them.
0016:def exists(path):
0017: """Test whether a path exists. Returns False for broken symbolic links"""
0018: try:
*** 0019: os.stat(path)
0020: except OSError:
0021: return False
0022: return True
0023:
Exception: TypeError: stat: can't specify None for path argument
(From OE-Core rev: 21924fdba286e5962b1680601664dc0491527e25)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- No need to use bb.utils.which() as subprocess will search $PATH
- Clarity flow by moving the install inside the try/except
(From OE-Core rev: f4d22b7195dd8f08fe26dd353c7e860208e87d6a)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a common function for locating task manifest files rather than
several implementations with missing pieces.
(From OE-Core rev: 68150bac7444f089f19c789e9f6602d59f605d7a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently do_rootfs gets to see all rpms in the deploy directory. This filters
that view to only rpms which the image recipe has actual depends upon which
potentially removes some sources of confusion in the image construction.
This makes builds more reproducibile and also fixes contamination issues
where dnf picks up packages it shouldn't be able to 'see'.
[YOCTO #12039]
(From OE-Core rev: 85e72e129362db896b0d368077033e4a2e373cf9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'exit 1' is not optimal for two reasons:
1) Code is hard to read; it is not obvious that it means 'defer what follows to first boot'.
2) Worse, this hides actual errors in the scriptlets; there is no difference between scriptlet
failing because it's intended to be run on target and scriptlet failing because there's a bug or
a regression somewhere.
The new, supported way is to place the code that has to run on target into pkg_postinst_ontarget(),
or, if a more fine-tuned control is required, call 'postinst-intercepts defer_to_first_boot' from
pkg_postinst() to explicitly request deferral to first boot.
(From OE-Core rev: d12cf56e9ff2a4f13dfbef9290ea5647b52b3f6d)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The repository indexes updated during do_populate_sdk_ext and do_populate_sdk
can conflcit. Add the missing lockfile calls for deb/ipk and in the rpm case,
ensure different directories are used for the index for the two sdk cases.
(From OE-Core rev: 5e5569c962c9ebc898eeb5044214e95117b190e1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we migrated rpm v5 -> v4, we lost the ability to drop "per file"
dependencies from the rpm backend for things like "/bin/bash" and
"/usr/bin/env" which meant the sdks were becomming 'bloated'.
This restores the functionality using a dummy package, similarly to
the way the buildtools perl issue was addressed. It also removes
the non-functional old code so as not to confuse people in future.
I ran into this problem trying to filter dependencies to only rpms
a build directly depends upon and it turns out we have some determinism issues
in this area so this is something key to fix.
(From OE-Core rev: 9d490dc01dcedb216129b22cbe17a6c99efc4f5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It had been removed since 2011:
commit b774bf44ef004276da12a83ebd69715c00b596ac
Author: Lianhao Lu <lianhao.lu@intel.com>
Date: Tue Aug 16 16:26:49 2011 +0800
package(_ipk).bbclass: opkg using ALL_MULTILIB_PACKAGE_ARCHS
(From OE-Core rev: e03cfb5a04e359c0bacb002dc80f3348301445d3)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can make debug easier, for example, makes it easy to run the
command mannually.
(From OE-Core rev: 220b912237da78d257c5310bd07db89489afede1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DNF has a time-based cache policy (and a great sense of humour) so it's possible
that 'dnf makecache' won't actually refresh any caches. Force the cache updates
by passing --refresh.
(From OE-Core rev: a6e34b6aa194a27db9667af1fb0195ac8fb563f5)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we wish to use the package feed for local development, it does
not uses GPG signed feeds by default but dnf uses package signature
check. We need to configure the GPG signature check out so it works
out of box.
With this patch, installing non-signed packages works:
$: dnf install <package>
(From OE-Core rev: bceafe23cbe12a1ab0628b70865cb8867c7d7ee1)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To be able to better debug remove() behaviour, add more logging
to rpm calls via bb.note(). The change also makes remove() logging
more consistent with other package managers' remove() (e.g., opkg).
(From OE-Core rev: 1a90c97616698050529e169a83e6f07a255687df)
Signed-off-by: Mikko Ylinen <mikko.ylinen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Search made with the following regex: getVar ?\((.*), True\).
(From OE-Core rev: dbc0eaf478feb3f752ae22fd184984494fc85d0a)
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If this fails the exception will now be raised. Lets use that and drop
all this 'None' return value ugliness.
(From OE-Core rev: b3c63b3b816179b96f1ed9b5baaf6e1f1c3c7b80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating Packages.gz, do not save the time stamp in the compressed
file metadata.
(From OE-Core rev: 693fc377aa0788825b921d1b7804f5b2eafec76f)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DISTRO_VERSION may contain the current date, and so is prone to mismatches
particularly when keys are created on one date, and dnf is configured to use the keys
on another date.
[YOCTO #11983]
(From OE-Core rev: b8f33f3f0a1e24f89151743d5c0b82efe45a9df1)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If package feed signing is enabled enable repo gpg signature check for
rpm repositories added via PACKAGE_FEED_URIS. This has the implication
that all repositories added via this mechanism must be signed with the
same key.
[YOCTO #11209]
(From OE-Core rev: f7716f1de0791dfe778bb70f1769a7e1e83c7a54)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When support for dnf was introduced the check of the
no NO_RECOMMENDATIONS variable got broken.
This fixes the issue by compairing to the string "1"
rather than the number 1.
(From OE-Core rev: 1849ce3bd7c0af055f3e849a6508e746b6a0dca5)
Signed-off-by: Richard Röjfors <richard@puffinpack.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that:
1) dnf does not complain anymore about releasever not being set and then fail
for the same reason;
2) it's possible to refer to $releasever in dnf package feed configuration
(repo paths in particular) without hardconding the release name (pyro, morty, etc.)
(From OE-Core rev: 789e3fc225adbb61f10aaa3bbc3677856f5f0238)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* since this commit:
commit f5a1013ffa9815f22e13989e2bcb83f966e7ce2c
Author: Ross Burton <ross.burton@intel.com>
Date: Tue Apr 18 16:19:12 2017 +0100
package_manager: don't race on a file when installing complementary
packages
the file isn't closed before oe-pkgdata-util uses it and this
temporary file might look empty to oe-pkgdata-util, because it
wasn't flushed yet. Which resulted in almost empty debugfs tarballs
and no locale packages in regular rootfs.
* without this change:
124K May 30 07:41 core-image-full-cmdline-raspberrypi3-64-20170530054003-dbg.rootfs.tar.gz
* with this change:
173M May 30 07:29 core-image-full-cmdline-raspberrypi3-64-20170530052715-dbg.rootfs.tar.gz
(From OE-Core rev: 877d38db08aa7060d16405443cf70539c559fe82)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During the transition to RPM4, the package names returned by
RpmPM.list_installed() changed from the expected names of the packages
that were installed into the image to some fictitious source RPM
names.
This restores the original functionality so that the
installed-packages.txt files produced by inheriting buildhistory yet
again contains a list of the names of the installed packages.
(From OE-Core rev: 25a36bcea4c02002d4895ba17f49ced09ae821c2)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The architecture list used by dnf/libsolv was in the wrong order.
As a result, the images were built with wrong and unpredictable
packages.
$ MACHINE=intel-corei7-64 bitbake core-image-sato
$ MACHINE=qemux86-64 bitbake core-image-sato
$ MACHINE=intel-corei7-64 bitbake -ccleansstate core-image-sato
$ MACHINE=intel-corei7-64 bitbake core-image-sato
The first image had 0 core2_64 packages in it, but the last one had
583 core2_64 packages (which were built for the qemu image in
between).
Reverse the arch order in etc/dnf/vars/arch.
Fixes [YOCTO #11384].
(From OE-Core rev: 4a82433de42943f8219beca3286f40b67157172f)
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PackageManager.install_complementary() uses WORKDIR/installed_pkgs.txt as a
temporary file but if two tasks are executing for the same recipe which uses
this file (e.g. bitbake my-image my-image:do_populate_sdk) then it's possible
for the file to be overwritten or deleted.
Instead of using a static filename, use tempfile to generate a unique name and
ensure it is cleaned up when finished.
Also move the glob generation/expansion earlier in the function as if there are
no globs to install, we don't need to generate a package list.
(From OE-Core rev: f5a1013ffa9815f22e13989e2bcb83f966e7ce2c)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
dnf requires a serparate repo for each architecture. This patch
writes one config file per PACKAGE_FEED_URIS entry with an entry
for each architecture, if any.
It also uses a space separated version of the repo id as the repo
name instead of just the id again.
(From OE-Core rev: 3eed822b5c5661aa9f43af6582c1481bacf0d39a)
Signed-off-by: Ian.Arkver <ian.arkver.dev@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds repo_name as the name, which is the same as the
repo ID and hence a bit pointless, but it stops dnf from
complaining that the repo doesn't have a configured name.
(From OE-Core rev: c92cff23893e87cfb7ef9f77b16761310104543a)
Signed-off-by: Ian.Arkver <ian.arkver.dev@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default the feed_archs variable is split into single characters
resulting in very many broken short repo_uris. Add a split() to
split the string into words first.
(From OE-Core rev: 7bd6a4d56c08bf7f9ced7c07327e6163216fee08)
Signed-off-by: Ian.Arkver <ian.arkver.dev@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* respect it for incremental rootfs generation
* add lists_dir option to opkg.conf
* also fix setting info_dir and status_file when they use default value, the
problem is that self.opkg_dir is already prefixed with rootfs directory,
comparing it with /var/lib/opkg always returned false and the options were
appended to config file unnecessary
* with opkg 0.3.4 we can use VARDIR prefix added in:
commit d2a8e23dc669adc398f4bb8bcfcabfcf925708f7
Author: Florin Gherendi <floring2502@gmail.com>
Date: Mon Dec 19 12:25:38 2016 +0200
libopkg: make the /var and /etc directories configurable at compile time.
(From OE-Core rev: b14c11d062872c3dcf95e03b61017005dea5b754)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is entirely unnecessary (we can ask the signer backend to export the
key to a file when needed), and was causing confusing selftest failures
due to the variable being set from two different places.
[YOCTO #11191]
(From OE-Core rev: 74ea979044368dc28c24325e7e77471b70aa8fe8)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All documentation refers to dnf binary as 'dnf' yet make install
does not create one - it's done by Fedora's spec file when building
the rpm. Let's replicate this behavior.
(From OE-Core rev: 456c4a8ffc9a292d7a3e036d92baf4a8f14d1f45)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Import the gpg key used in rpm signing into rpmdb. This makes it
possible again to create images when rpm signing is enabled.
Also, instruct dnf to enforce signature check if rpm signing is enabled.
(From OE-Core rev: f30c1653cc5ef9daf594cbd3faad329b9fa08ab7)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
