Commit Graph

531 Commits

Author SHA1 Message Date
Paul Barker
da846838fc wic: Optimise fstab modification for ext2/3/4 and msdos partitions
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.

We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.

(From OE-Core rev: 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
36575a9493 wic: Copy rootfs dir if fstab needs updating
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.

This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.

An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.

Fixes [Yocto #13994]

(From OE-Core rev: ce682a73b7447652f898ce1d1d0416a456df5416)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
f85a4a1462 wic: Update pseudo db when excluding content from rootfs
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.

To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.

(From OE-Core rev: d5db7e268947f0392c2126137571a44acd29ccd6)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
4e6cb084f5 wic: Ensure internal workdir is not reused
If a path is specified for the internal wic working directory using
the -w/--workdir argument then it must not already exist. Re-using a
previous workdir could easily result in rootfs and intermediate files
from a previous build being added to the current image.

(From OE-Core rev: 2e40c8d4109024ff704c5ce40d98050ca7f34dd5)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
cf73a72632 wic: Allow exec_native_cmd to run HOSTTOOLS
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to
be more easily executed by wic. Without this change only programs from
an actual *-native recipe built by bitbake can be executed by wic.

(From OE-Core rev: 8eb186acdecfbb3151c9a0ab148358e3fe5cce39)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-13 23:13:45 +00:00
Paul Barker
1b816d180b wic: Add workdir argument
This allows the path for the temporary workdir used by wic to be set
when running wic from bitbake or directly from the command line.

(From OE-Core rev: 2e7314ac3a3cab89de93c932e8efc75de0feb0a7)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-13 23:13:45 +00:00
Lee Chee Yang
3fa6a47aeb wic/direct/kparser: ensure fsuuid for vfat and msdos align with format
vfat/msdos filesystem should have fsuuid in format 0xYYYYYYYY where "0x"
in front follow with 8 hexadecimal number in uppercase. In wic, when using
custom fsuuid for vfat/msdos partition in wks, it is able to set the value
in any length, with or without leading "0x". This can cause fsuuid
missaligned when fstab updates, fstab expect exactly 10 character
fsuuid for vfat/msdos partition and all in uppercase.

if custom fsuuid for vfat/msdos is set, check the length and format,
error if it exceed the format size. Amend it so it is align with format
0xYYYYYYYY. This is done before image create and fstab update to ensure the
fsuuid are same in all followup process. if custom fsuuid length less than
expected, fill in "0".

[YOCTO #14161]

(From OE-Core rev: d9686ae511ef10a504becfd81bfe296b788e1456)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-01 23:15:13 +00:00
Peter Kjellerstedt
e12072c7e0 wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
(From OE-Core rev: 55618cbd58d6784a82e773f323723be6f722232f)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-20 00:03:04 +00:00
Diego Sueiro
75b74f6b8d wic: Introduce empty plugin to create unformatted empty partitions
The empty wic plugin is used to create unformatted empty partitions for wic
images.

To use it you must pass "empty" as argument for the "--source" parameter in
the wks file. For example:
    part foo --source empty --ondisk sda --size="1024" --align 1024

Also adds a selftest for this plugin where the 'Fstype' column from 'wic
ls' should be empty for the second partition as listed in
test_empty_plugin.wks.

(From OE-Core rev: 77d174fc80663403ef76c5b808aafc1117d3545c)

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-15 22:51:56 +00:00
Chaitanya Vadrevu
ea5880f9f9 isoimage-isohybrid.py: Support adding files/dirs
Add "payload_dir" parameter so user can add other files/dirs
to the ISO.

(From OE-Core rev: bbe84a33c81066fee96dd28a4009d8404b51e642)

Signed-off-by: Chaitanya Vadrevu <chaitanya.vadrevu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-08 14:03:20 +00:00
Richard Purdie
c194e5fac6 wic: Handle new PSEUDO_IGNORE_PATHS variable
Adjust wic to correctly handle the new PSEUDO_IGNORE_PATH variable and avoid
inode corruption issues.

(From OE-Core rev: 13500f5234361385c365c7c35e83f99435500481)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-07 11:18:38 +01:00
Khairul Rohaizzat Jamaluddin
85574ce0cf wic/bootimg-efi: IMAGE_EFI_BOOT_FILES variable added to separate bootimg-efi and bootimg-partition
Due to recent changes in bootimg-efi to include IMAGE_BOOT_FILES,
when both bootimg-partition and bootimg-efi occur in a single .wks
and IMAGE_BOOT_FILES are defined, files listed in IMAGE_BOOT_FILES
will be duplicated in both partition.
Since IMAGE_BOOT_FILES are crucial for bootimg-partition, but
optional for bootimg-efi, hence allowing bootimg-efi to have the option
to ignore it.

The new variable, IMAGE_EFI_BOOT_FILES, was added to help handle this
issue. Its basic usage is the same as IMAGE_BOOT_FILES.
Usage example:
        ${IMGDEPLOYDIR}/${IMAGE_BASENAME}-${MACHINE}.ext4;rootfs.img \

This commit is also squashed with the updated testcase to cover for
this change.

[YOCTO #14011]

(From OE-Core rev: 945339e06b273df1935cfd784f548ef57e0b7f4c)

Signed-off-by: Khairul Rohaizzat Jamaluddin <khairul.rohaizzat.jamaluddin@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-15 11:53:28 +01:00
Vijai Kumar K
435ec90a67 wic: misc: Add /bin to the list of searchpaths
/bin is also a valid path where one can find executables. Add
that to the search path.

(From OE-Core rev: ca0a6025351cb2135e87cecf828633cf12aa34c6)

Signed-off-by: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-02 16:00:46 +01:00
Joshua Watt
5fa3c3d2e3 wic: Add 512 Byte alignment to --offset
Allows the --offset argument to use the "s" or "S" suffix to specify
that it is reporting the number of 512 byte sectors.

This is required for some SoCs where the mask ROM looks for an item at a
sector that isn't aligned to a 1KB boundary.

(From OE-Core rev: 938595d1dc4abaf5f7f3a7900add3f0492b805d0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-22 15:46:55 +01:00
Daniel Ammann
f39694933a wic: fix typo
(From OE-Core rev: 9ab4c4571d899d2eefcd2fc53af0851863f29008)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-06 15:12:39 +01:00
Khasim Mohammed
801f07fc00 wic/bootimg-efi: Add support for IMAGE_BOOT_FILES
List of files defined using IMAGE_BOOT_FILES are installed into
the boot partition when preparing an image using the wic tool with
the bootimg-efi source plugin.

The code snippet introduced is taken as is from bootimg-partition.py

Change-Id: I8dbb6b4e7c24870f587a6f31e6ae4a87d7033782
Issue-Id: PLATFORMS-3134
(From OE-Core rev: a44ab3a4ee5b3c57812909c6194456f299d6ba7f)

Signed-off-by: Khasim Syed Mohammed <khasim.mohammed@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-25 15:11:05 +01:00
Kevin Hao
b617cd5b47 wic/filemap: Fall back to standard copy when no way to get the block map
For some filesystems, such as aufs which may be used by docker container,
don't support either the SEEK_DATA/HOLE or FIEMAP to get the block
map. So add a FileNobmap class to fall back to standard copy when there
is no way to get the block map.

[Yocto #12988]

(From OE-Core rev: 7934ed49179242f15b413c0275040a3bb6b68876)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-18 11:06:31 +01:00
Kevin Hao
2212d938c5 wic/filemap: Drop the unused get_unmapped_ranges()
This method is not used by any code, so drop it.

(From OE-Core rev: 6513fd9302b9989f97fc9d95e76e06ad5d266774)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-18 11:06:31 +01:00
Kevin Hao
9818a04678 wic/filemap: Drop the unused block_is_unmapped()
This method is not used by any code, so drop it.

(From OE-Core rev: fb62a15349597ee026c67a0bb0a6ca2cc9bfe420)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-18 11:06:31 +01:00
Joshua Watt
5c82d30dc1 wic: Fix error message when reporting invalid offset
The error message was reporting the calculated offset instead of the
current offset, which made it confusing.

(From OE-Core rev: 2be775cfe1b49ce3889b5dc326e2b67a9667f18a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-16 23:39:28 +01:00
Joshua Watt
4f3aca1a56 wic: Fix --extra-space argument handling
467f84e12b ("wic: Add --offset argument for partitions") broke the
--extra-space argument handling in wic. Fix the option and add a unit
test for the argument.

(From OE-Core rev: 87722a92c18f94917c8f70afc8cd0763462a5c25)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-12 08:25:04 +01:00
Joshua Watt
ba9be4bfa5 wic: Add --offset argument for partitions
Add support for an --offset argument when defining a partition. Many
SoCs require that boot partitions be located at specific offsets. Prior
to this argument, most WKS files were using the --align attribute to
specify the location of these fixed partitions but this is not ideal
because in the event that the partition couldn't be placed in the
specified location, wic would move it to the next sector with that
alignment, often preventing the device from booting. Unlike the --align
argument, wic will fail if a partition cannot be placed at the exact
offset specified with --offset.

Changes in V2:
* Fixed a small typo that prevented test_fixed_size_error from passing

(From OE-Core rev: 467f84e12b96bc977d57575023517dd6f8ef7f29)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:32 +01:00
Ricardo Ribalda Delgado
31b14186e9 wic: rootfs: Combine path_validation in one function
Combine all the common path validation in a function to avoid code
duplication.

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: ebd12b10d17db0b4176b0188407d7e9b8420eab1)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Ricardo Ribalda Delgado
a293c76c53 wic: root: Add an opt. destination on include-path
Allow specifying an optional destination to include-path and make the
option aware of permissions and owners.

It is very useful for making a partition that contains the rootfs for a
host and a target Eg:

/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)

Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.

With this patch we can use something like:

part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ --include-path hello

on the .wks file.

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: e8c21c6ebaebde88151697381bdb2452f1171090)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Ricardo Ribalda Delgado
fc2589384b wic: misc: Do not find for executables in ASSUME_PROVIDED
Executables like tar won't be available on the native sysroot, as they
are part of the ASSUME_PROVIDED variable.

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: 2f574d535f8665b26dab65c14668cf8fc7b751c0)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Ricardo Ribalda Delgado
c81d5c6243 wic: Avoid creating invalid pseudo directory
If the source of the rootfs is not a bitbake cooked image, or it is not
pointing to the root of one, we call pseudo again, which will produce
a new pseudo folder at rootfs/../pseudo

Eg:

part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: 799a24ae78655f7a3eda7456b1a0ffaf3e43ec16)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Ricardo Ribalda Delgado
19e778fe1b wic: Continue if excluded_path does not exist
If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.

Eg:

part / --source rootfs --fstype=ext4 --exclude-path=opt/private_keys

Where /opt/private_keys in only populated by some of the image.bb files.

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: 2dbd692b8e563cf991fb4ae1ef6129fda0d7e3c4)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Ricardo Ribalda Delgado
d19004da2f wic: Add --change-directory argument
This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.

Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:

part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=etc

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: 2265d089a58e1f78f26d623ee667c420cb1c3bd4)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Ricardo Ribalda Delgado
c58711f0ea wic: Fix multi images .wks with bitbake
In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.

Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4

The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).

Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: dde90a5dd2b22a539095d1bac82acc15c6380ac8)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Ricardo Ribalda Delgado
90164f0f3d wic: Fix permissions when using exclude or include path
When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

This bug can be easily triggerd with the following .wks

part / --source rootfs --fstype=ext4 --exclude-path=home

And this sequence:

$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.

WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions will be invalid

Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: fd739c15cdba221d9d497d3402b7d929c0be2ca4)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Lee Chee Yang
df77ee4723 wic: use Filesystem UUID when expand swap partition
part.get("uuid") return the Partition UUID from the partition
table instead of Filesystem UUID. This lead to swap partition
UUID not match/change when wic write expand swap partition.

change it to read the filesystem UUID using blkid. The output
from blkid should looks like this:
wic-partvzhiwq3s: LABEL="swap1" UUID="04e55c19-3f3f-4491-9e32-44eea8daa827" VERSION="1" TYPE="swap" USAGE="other"

[YOCTO #13313]

(From OE-Core rev: 683297eefebe83f848daad9927871242ab28ef91)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Richard Purdie
f4e0b26568 Revert "wic: Fix permissions when using exclude or include path"
This reverts commit 36993eea89d1c011397b7692b9b8d61b499d0171.

After discussion on the mailing list it was felt these changes
were not ready yet.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-07 21:57:16 +01:00
Richard Purdie
157a12d499 Revert "wic: Add --embed-rootfs argument"
This reverts commit efdcf94801f6abe8e4099e324d9a3deccd8d4384.

After discussion on the mailing list it was felt these
changes were not ready yet.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-07 21:57:16 +01:00
Ricardo Ribalda Delgado
242412656b wic: Add --embed-rootfs argument
This option adds the content of a rootfs on a specific location on the
rootfs.

It is very useful for making a partition that contains the rootfs for a
host and a target Eg:

/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)

Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.

With this patch we can use something like:

part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2

on the .wks file.

(From OE-Core rev: efdcf94801f6abe8e4099e324d9a3deccd8d4384)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:45:11 +01:00
Ricardo Ribalda Delgado
6bac089383 wic: Fix permissions when using exclude or include path
When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

(From OE-Core rev: 36993eea89d1c011397b7692b9b8d61b499d0171)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:45:11 +01:00
Lee Chee Yang
b5a3625e59 wic: align layer plugin path
wic look for external layer source plugin under scripts/lib/
while other tools look for lib/.

allow wic to check for source plugin at both scripts/lib/ and
lib/ to align with other tools while avoid breaking any existing
source plugin in external layer.

[YOCTO #13056]

(From OE-Core rev: 986baff26bd96a6265f5fe2d631818fff9f66374)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-02 15:24:59 +01:00
Chee Yang Lee
590555e3b8 wic/direct: reserve 2 sector for extended partition
extended partition should reserve at least 2 unallocated sectors
between the start of the extended partition and the start of a
logical partition.

[YOCTO #13658]

(From OE-Core rev: 884c8a0e50a49bdca5e048197a9dc0ff37cc8d1a)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 16:39:40 +00:00
Armin Kuster
26525f2f4f wic/engine: lets display an error not a traceback
If the requested partition does not exist in this request "wic ls {path}:pnum"
display a nice message not a trackback

Also fix displaying the pnum and not "%s"

(From OE-Core rev: 15d1722950a22649905cf8a5789d3cfe48a2a892)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-02 21:00:54 +00:00
Pierre-Jean Texier
0c8892fa1d wic/engine: dump the partition table in JSON format only
since commit 03154d2cf25c1d5ce908da9e72d324004b8d5722 ("sfdisk: add -J between mutually exclusive options")
in util-linux [1], the '-J' can't be used with the '-d' option anymore.

So, since we're using json format output, drop the '-d' option.

Fixes:

ERROR: _exec_cmd: <snip>/usr/sbin/sfdisk -dJ <snip>/core-image-minimal-qemux86-64.wic returned '1' instead of 0
output: sfdisk: mutually exclusive arguments: --list-free --json --dump

[1] - 03154d2cf2 (diff-31d5cc325380dbdafe64d0e019846f12)

(From OE-Core rev: 49e7f3e8e3a775510d8b0e2826a06d000934706b)

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-01 10:40:11 +00:00
Alexander Kanavin
e88fe83014 qemux86: do not add vga=0 to kernel parameters
This was added ages ago to enable GL passthrough with
vmware driver, and is no longer relevant, as std or virgl is used
instead nowadays.

Original commit:

commit 072545b111
Author: Richard Purdie <rpurdie@linux.intel.com>
Date:   Wed Jan 21 17:40:51 2009 +0000

    scripts/poky-qemu-internal: Add support for GL passthrough in qemux86 images

(From OE-Core rev: 857078ba8eda153f4a097683db551a7d310ecc01)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-18 23:53:54 +00:00
Kevin Hao
dd372f09fc wic: Honor APPEND
APPEND is used to add additional parameters to kernel by features, such
as read-only-rootfs. So we should honor this variable when we compose
the kernel parameter in wic. I know we also can resolve this kind of
issue by using the .wks.in template introduced by commit 42e870c5ed
("image_types.bbclass: support template .wks.in files for wic"), but the
APPEND is needed by all the wks, it would seems pretty ridiculous to me
to change all the .wks to .wks.in and then foist the APPEND into them.
So the APPEND is definitely deserved to export to the wic directly.

[Yocto #12809]

(From OE-Core rev: 18981b8a457104391dfd94938c247eac04e4ed50)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-15 22:42:14 +00:00
Maxim Uvarov
8290a50fe2 wic: fix images build in parallel
OE wic plugins create temporary file with the index of the line
tmp file name. This causes race in case several builds run in time.
If source_params['file'] is an absolute path, the cr_workdir prefix
is not applied by os.path.join(). So instead it writes to a ".1"
file next to the original image - this is outside the WORKDIR
and at risk of collision.

(From OE-Core rev: c68d8a37ba4348fe1c0e75c63b5668187d326ec2)

Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
Suggested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Suggested-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:24:39 +00:00
Paul Barker
2c81446248 wic: Add --include-path argument
This option adds the contents of the given path to a partition built
with the rootfs source plugin. The path is relative to the directory in
which wic is running not the rootfs itself so use of an absolute path
is recommended. This option is most useful when multiple copies of the
rootfs are added to an image and it is required to add extra content to
only one of these copies. This option only has an effect with the
rootfs source plugin.

(From OE-Core rev: d4cd27a9837426e809190548a83c6c7c76505114)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-10 21:18:22 +00:00
Paul Barker
ab6e8dae1e wic: Preserve intermediate files
When debugging issues or trying to add new features in wic it's useful
to examine the intermediate files created by the tool such as images of
the individual partitions. Previously these files were created in a
subdirectory of IMGDEPLOYDIR and, deleted by wic itself and then the
working subdirectory completely removed at the end of IMAGE_CMD_wic.
Instead we should work in a staging directory named build-wic and leave
behind intermediate files for analysis if necessary.

This has the added benefit of allowing individual partition images to
be preserved by adding commands to IMAGE_CMD_wic_append() in an image
recipe if needed. This is useful where an update system requires exact
sized and pre-formatted partition images for deployment.

(From OE-Core rev: 07b6b284204e912b5d08bdd0ca51b766ab7dd9c1)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-10 21:18:22 +00:00
Kalle Lampila
b14588a778 wic/filemap: If FIGETBSZ iotctl fail, failback to os.stat
Some file systems don't support fetching the block size (notably the
file system Docker uses for containers), so if iotctl() fail, try to use
failback via os.stat() to get block size.

(From OE-Core rev: e219f5175177a640dd62833082ea19adc1c13d42)

Signed-off-by: Kalle lampila <kalle.lampila@lempea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-02 16:39:13 +00:00
Chee Yang Lee
89288705c6 wic: 'wic cp' to copy from image
currently 'wic cp' only works for copy file from local storage to
wic image.

enhance 'wic cp' to copy file/directory from wic image to local storage.

include selftest and 'wic help' updates.

[YOCTO#12169]

(From OE-Core rev: bd669c1809a378f93580eb9e0679a26ec6746cb8)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-27 13:25:18 +00:00
Chee Yang Lee
3555766ea8 wic: rm with -r flag support
wic currently unable to remove non-empty directory in ext* partition.
enable wic rm to remove non-empty directory and all the sub-content
with -r flag.
update help documents for 'wic rm'.
[YOCTO #12404]

(From OE-Core rev: 5cb7a329d0aaac8fe5328eb2001692c540aa5ade)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-21 23:08:19 +00:00
Chee Yang Lee
621b10f007 wic: beautify 'wic help'
The Wic help returned to the user is unreadable.

Use a custom ArgumentParser to override argparse help message.

change help message as suggest in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12205
[YOCTO #12205]

changes applies to 'wic help', 'wic -h', 'wic --h' and 'wic --help'

(From OE-Core rev: d29d553cc096f4e1208c44dc595e1cf365c3dff3)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Haris Okanovic
a98c7c7aa3 isoimage-isohybrid.py: Parameterize ESP partition size
Add "esp_extra_blocks" plugin parameter so that caller may change
ESP's free space from the default 100 blocks.

(From OE-Core rev: db4be094b0d6741ca4f9e2cbb499fc9c224fa690)

Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-10 14:06:10 +00:00
Haris Okanovic
36bb63c3d2 isoimage-isohybrid.py: Parameterize ESP label
Add "esp_label" plugin parameter so that caller may override default
ESP partition label 'EFIimg'.

(From OE-Core rev: 6b3b9042a55d47205b16ef792f93a9bb240fd7cb)

Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-10 14:06:10 +00:00