f85c68cb9c
There is a race condition when iterating directories which are being altered whilst iterating, which is something that can and does happen when do_package_qa runs at the same time as eg do_package_write_ipkg (the opkg metadata is written inside the build tree). The race is that naive code will list a directory contents and then stat() each name to determine if its a directory or file. The classic failure that we see is that CONTROL/ is found on a listdir but deleted by the time the stat happens, so is incorrectly listed as a file (because it is not a directory). Since Python 3.5, os.walk() uses scandir() instead of listdir() which mitigates this race by returning the file type alongside the name, so a stat is no longer needed to identify the type. However, cachedpath.walk() was copied from Python before this, so it uses listdir() and has this race condition. Since I changed insane to use cachedpath.walk()[1] I inadvertently reintroduced this race. I believe there's actually no need to use cachedpath.walk() and a logical fix is to simply use os.walk(): With os.walk() each directory is listed and categorised in a single os.scandir() as the underlying syscall, getdents64, returns the type. However, cachedpath.walk() uses os.listdir() which ignores the type field returned and has to do a stat() on every file to determine the type. Thus, we should switch users of cachedpath.walk() to os.walk(): there's no real gain in what is effectively just a prefetch for the stat cache, but depending on what the calling code does may result in more stat() calls than needed. In the future we may want to redesign cachedpath to reimplement walk so that it can also cache the DirEntry instances as returned by scandir() as that will avoid needing to call stat() at all in many cases. However I believe we should instead use a caching pathlib.Path instance instead. [1] cad3c8 insane: use oe.cachedpath.CachedPath instead of os.path (From OE-Core rev: 22e4486d65e4874bf48d89160d69118f318278e8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Poky
Poky is an integration of various components to form a pre-packaged build system and development environment which is used as a development and validation tool by the Yocto Project. It features support for building customised embedded style device images and custom containers. There are reference demo images ranging from X11/GTK+ to Weston, commandline and more. The system supports cross-architecture application development using QEMU emulation and a standalone toolchain and SDK suitable for IDE integration.
Additional information on the specifics of hardware that Poky supports is available in README.hardware. Further hardware support can easily be added in the form of BSP layers which extend the systems capabilities in a modular way. Many layers are available and can be found through the layer index.
As an integration layer Poky consists of several upstream projects such as BitBake, OpenEmbedded-Core, Yocto documentation, the 'meta-yocto' layer which has configuration and hardware support components. These components are all part of the Yocto Project and OpenEmbedded ecosystems.
The Yocto Project has extensive documentation about the system including a reference manual which can be found at https://docs.yoctoproject.org/
OpenEmbedded is the build architecture used by Poky and the Yocto project. For information about OpenEmbedded, see the OpenEmbedded website.
Contribution Guidelines
Please refer to our contributor guide here: https://docs.yoctoproject.org/dev/contributor-guide/ for full details on how to submit changes.
Where to Send Patches
As Poky is an integration repository (built using a tool called combo-layer), patches against the various components should be sent to their respective upstreams:
OpenEmbedded-Core (files in meta/, meta-selftest/, meta-skeleton/, scripts/):
- Git repository: https://git.openembedded.org/openembedded-core/
- Mailing list: openembedded-core@lists.openembedded.org
BitBake (files in bitbake/):
- Git repository: https://git.openembedded.org/bitbake/
- Mailing list: bitbake-devel@lists.openembedded.org
Documentation (files in documentation/):
- Git repository: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/
- Mailing list: docs@lists.yoctoproject.org
meta-yocto (files in meta-poky/, meta-yocto-bsp/):
- Git repository: https://git.yoctoproject.org/cgit/cgit.cgi/meta-yocto
- Mailing list: poky@lists.yoctoproject.org
If in doubt, check the openembedded-core git repository for the content you intend to modify as most files are from there unless clearly one of the above categories. Before sending, be sure the patches apply cleanly to the current git repository branch in question.