c984b03f02
For gitsm recipes, it's possible that some URL is used more than once.
e.g.,
A -> B:rev1 (B is a submodule of A)
A -> C (C is a submodule of A)
C -> B:rev2 (B is a submodule of C)
A anc C are both using B as submodules, but on different revs.
Now if we have:
B:rev1 -> D
B:rev2 -> E
Then, the mirror will not be fully used.
Say we have all repo mirrors for A, B, C, D, E, then in theory it's not
necessary to reach out to any network for downloading. But it's not the
case. After downloading B(rev1) and its submodule D from mirrors, the fetch
process continues to download C, thus B(rev2) and E. Now it finds that B
needs an update because its submodule E needs an update. Of course this is
true because E is not downloaded yet. Now the problem comes to whether to
use mirror or not. The git.py defines try_premirror to return 'False' when
the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and
try_mirror returns False, resulting in not using mirror and going to upstream
directly.
We can see that the mirrors are not fully used. This is usually not problem,
as the cost is only some network download. But in case the following two
settings are there, we get errors.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.allowed.domain"
In such case, the gitsm recipe A will fail to fetch. Note that all contents
that A needs are in mirrors and now it's failing to fetch. This is unexpected.
Note that the different revs of the same repo in gitsm recipe is not the only
way to reveal this problem. For example, there might be a recipe call B that
uses B:rev3. Check the protobuf and grpc recipes as an example.
For now, we can use the following steps to reproduce this issue. To be clear,
the grpc recipe in meta-oe is now 1.60.0.
1. Add in local.conf:
DL_DIR = "${TOPDIR}/downloads-premirror"
bitbake grpc -c fetch
2. Comment out the DL_DIR setting in local.conf and add the following lines:
PREMIRRORS:append = " \
git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
"
3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'.
This command succeeds and this shows that the premirror holds everything we need.
4. Add the following lines and then 'bitbake grpc -c fetch'.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.domain"
After step 4, the error message is as below:
ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used
This patch fixes this problem by handling this corner case, that is, if the URL is
not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should
try premirrors because trying to reach upstream is destined to fail.
(Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Poky
Poky is an integration of various components to form a pre-packaged build system and development environment which is used as a development and validation tool by the Yocto Project. It features support for building customised embedded style device images and custom containers. There are reference demo images ranging from X11/GTK+ to Weston, commandline and more. The system supports cross-architecture application development using QEMU emulation and a standalone toolchain and SDK suitable for IDE integration.
Additional information on the specifics of hardware that Poky supports is available in README.hardware. Further hardware support can easily be added in the form of BSP layers which extend the systems capabilities in a modular way. Many layers are available and can be found through the layer index.
As an integration layer Poky consists of several upstream projects such as BitBake, OpenEmbedded-Core, Yocto documentation, the 'meta-yocto' layer which has configuration and hardware support components. These components are all part of the Yocto Project and OpenEmbedded ecosystems.
The Yocto Project has extensive documentation about the system including a reference manual which can be found at https://docs.yoctoproject.org/
OpenEmbedded is the build architecture used by Poky and the Yocto project. For information about OpenEmbedded, see the OpenEmbedded website.
Contribution Guidelines
Please refer to our contributor guide here: https://docs.yoctoproject.org/dev/contributor-guide/ for full details on how to submit changes.
Where to Send Patches
As Poky is an integration repository (built using a tool called combo-layer), patches against the various components should be sent to their respective upstreams:
OpenEmbedded-Core (files in meta/, meta-selftest/, meta-skeleton/, scripts/):
- Git repository: https://git.openembedded.org/openembedded-core/
- Mailing list: openembedded-core@lists.openembedded.org
BitBake (files in bitbake/):
- Git repository: https://git.openembedded.org/bitbake/
- Mailing list: bitbake-devel@lists.openembedded.org
Documentation (files in documentation/):
- Git repository: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/
- Mailing list: docs@lists.yoctoproject.org
meta-yocto (files in meta-poky/, meta-yocto-bsp/):
- Git repository: https://git.yoctoproject.org/cgit/cgit.cgi/meta-yocto
- Mailing list: poky@lists.yoctoproject.org
If in doubt, check the openembedded-core git repository for the content you intend to modify as most files are from there unless clearly one of the above categories. Before sending, be sure the patches apply cleanly to the current git repository branch in question.