0b6ad0bc1a
While working on splitting-out wic from oe-core, on my openSUSE Leap
16.0 machine, the moment I split wic out, 2 oe-selftests always failed
with 100% reproducibility:
- wic.ModifyTests.test_wic_cp_ext
- wic.Wic2.test_expand_mbr_image
In both cases the symptom is the same: the filesystem has inode tables
that are completely zeroed out. Both issues are linked together to the
same underlying fault.
FilemapSeek._get_ranges() is a generator. Due to the nature of finding
each hole/data extent one at a time using the lseek() system call,
it calls os.lseek() on a raw file descriptor, then yields, then the
caller, sparse_copy(), calls file.seek() + file.read() on a Python
BufferedReader wrapping that same fd — then the generator resumes and
calls os.lseek() again. This interleaving of raw os.lseek() and buffered
I/O on the same fd is undefined behaviour from Python's perspective.
The BufferedReader tracks its own idea of the fd's position and buffer
contents; os.lseek() changes the position behind its back. This can
corrupt its internal state and cause read() to return stale/zero data.
This code, however, has existed in wic since it was written, so why
was it not noticed before? It turns out this bug was being masked by a
number of implementation details that changed, especially when wic was
split out for oe-core. These changes conspired together to cause the bug
to be triggered.
One of the root causes of this bug is that Python 3.14 increased the
default buffer size from 8KB to 128KB[1]. With 8 KB buffers, read()s
either go through the direct-read path leaving the buffer empty, or
if it fills in 8KB chunks the buffer is fully drained. Either way,
with a small buffer, read()s do a real raw seek. No fast path. No
corruption. With a 128KB buffer, however, a much larger window exists
where BufferedReader.seek() can take the fast-path after the raw file
descriptor has already been repositioned by os.lseek() in the generator.
With the smaller buffer, this window was too narrow to hit in practice.
This is fixed by opening a second file object in FilemapSeek.__init__()
dedicated to SEEK_DATA/SEEK_HOLE probes, leaving the data-reading handle
(self._f_image) untouched.
This explains why the corruption is deterministic and tied to specific
block boundaries, why it only manifests with the split-out version using
Python 3.14 (on systems that are using Python versions less than 3.14 on
the host), and why using a separate file descriptor for reading bypasses
the issue entirely.
This is not an intermittent bug. For a more detailed explanation
including log files, in-depth analysis, and a standalone Python
reproducer, please see the linked bugzilla entry.
Fixes: [YOCTO #16197]
[1] b1b4f9625c
b1b4f9625c5f ("gh-117151: IO performance improvement, increase io.DEFAULT_BUFFER_SIZE to 128k (GH-118144)")
AI-Generated: codex/claude-opus-4.6 (xhigh)
(From OE-Core rev: 37a45219dd204b07bad40576fefccb2cf85b255c)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 481969844385f2fa40a1230ca50253ec4ff516cd)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Paul Barker <paul@pbarker.dev>
Poky
Poky is an integration of various components to form a pre-packaged build system and development environment which is used as a development and validation tool by the Yocto Project. It features support for building customised embedded style device images and custom containers. There are reference demo images ranging from X11/GTK+ to Weston, commandline and more. The system supports cross-architecture application development using QEMU emulation and a standalone toolchain and SDK suitable for IDE integration.
Additional information on the specifics of hardware that Poky supports is available in README.hardware. Further hardware support can easily be added in the form of BSP layers which extend the systems capabilities in a modular way. Many layers are available and can be found through the layer index.
As an integration layer Poky consists of several upstream projects such as BitBake, OpenEmbedded-Core, Yocto documentation, the 'meta-yocto' layer which has configuration and hardware support components. These components are all part of the Yocto Project and OpenEmbedded ecosystems.
The Yocto Project has extensive documentation about the system including a reference manual which can be found at https://docs.yoctoproject.org/
OpenEmbedded is the build architecture used by Poky and the Yocto project. For information about OpenEmbedded, see the OpenEmbedded website.
Contribution Guidelines
Please refer to our contributor guide here: https://docs.yoctoproject.org/dev/contributor-guide/ for full details on how to submit changes.
Where to Send Patches
As Poky is an integration repository (built using a tool called combo-layer), patches against the various components should be sent to their respective upstreams:
OpenEmbedded-Core (files in meta/, meta-selftest/, meta-skeleton/, scripts/):
- Git repository: https://git.openembedded.org/openembedded-core/
- Mailing list: openembedded-core@lists.openembedded.org
BitBake (files in bitbake/):
- Git repository: https://git.openembedded.org/bitbake/
- Mailing list: bitbake-devel@lists.openembedded.org
Documentation (files in documentation/):
- Git repository: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/
- Mailing list: docs@lists.yoctoproject.org
meta-yocto (files in meta-poky/, meta-yocto-bsp/):
- Git repository: https://git.yoctoproject.org/cgit/cgit.cgi/meta-yocto
- Mailing list: poky@lists.yoctoproject.org
If in doubt, check the openembedded-core git repository for the content you intend to modify as most files are from there unless clearly one of the above categories. Before sending, be sure the patches apply cleanly to the current git repository branch in question.