c3b3d7c14f
The cached revisions which are used to decide if a repository doesn't need to be updated are misleading when used in conjunction with mirror tarballs and can cause partial fetches to happen, resulting in unpack errors as repositories were not fetched. A concrete example: edk2-firmware in meta-arm is at version 202102 (ef91b0). This is built on the autobuilder so the source mirror contains the repository as a mirror tarball. If I build edk2-firmware 202102 the gitsm fetcher will initially download the top-level repository and then iterate into the submodules to also fetch those repositories, including cmocka from cryptomilk.org. edk2-firmware will then unpack and build successfully. I then update edk2-firmware to 202105 (e1999b) and build it. Gitsm.needs_update() starts by calling Git.needs_update() which returns False, as the mirror tarball contains this revision. It then looks at the "nuggets" which are SRCREVs it has fetched before. The mirror tarball itself contains the nugget for e1999b as this has been built on the autobuilder, so needs_update return False, no more fetching is done, and the build proceeds to unpack. However, as part of the 202105 upgrade the URL of the cmocka submodule changed, and this new repository was never fetched. This means that unpack fails as one of the required git repositories isn't available. The nugget codepaths appear to be an attempt at optimising the fetch process, but have demonstratable failure cases. Just removing them entirely solves the edk2-firmware example, and all of the fetcher test cases still pass. (Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake
BitBake is a generic task execution engine that allows shell and Python tasks to be run efficiently and in parallel while working within complex inter-task dependency constraints. One of BitBake's main users, OpenEmbedded, takes this core and builds embedded Linux software stacks using a task-oriented approach.
For information about Bitbake, see the OpenEmbedded website: http://www.openembedded.org/
Bitbake plain documentation can be found under the doc directory or its integrated html version at the Yocto Project website: https://docs.yoctoproject.org
Contributing
Please refer to http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded for guidelines on how to submit patches, just note that the latter documentation is intended for OpenEmbedded (and its core) not bitbake patches (bitbake-devel@lists.openembedded.org) but in general main guidelines apply. Once the commit(s) have been created, the way to send the patch is through git-send-email. For example, to send the last commit (HEAD) on current branch, type:
git send-email -M -1 --to bitbake-devel@lists.openembedded.org
Mailing list:
http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Source code:
http://git.openembedded.org/bitbake/
Testing:
Bitbake has a testsuite located in lib/bb/tests/ whichs aim to try and prevent regressions. You can run this with "bitbake-selftest". In particular the fetcher is well covered since it has so many corner cases. The datastore has many tests too. Testing with the testsuite is recommended before submitting patches, particularly to the fetcher and datastore. We also appreciate new test cases and may require them for more obscure issues.