mirrors/poky
1
0
Fork 0
mirror of https://git.yoctoproject.org/poky synced 2026-03-17 04:39:40 +01:00
Code Issues Packages Projects Releases Wiki Activity

bitbake: fetch2/gitsm: remove the 'nugget' SRCREV caching

Browse Source 
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.

A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball.  If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org.  edk2-firmware will then unpack and build
successfully.

I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before.  The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.

However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.

The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases.  Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.

(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Ross Burton
2021-10-05 15:50:01 +01:00
committed by Richard Purdie
parent 07996f492b
commit c3b3d7c14f
1 changed files with 0 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
bitbake/lib/bb/fetch2/gitsm.py
View File

@@ -140,16 +140,6 @@ class GitSM(Git):
if Git.need_update(self, ud, d):
return True
try:
# Check for the nugget dropped by the download operation
known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \
(ud.basecmd), d, workdir=ud.clonedir)
if ud.revisions[ud.names[0]] in known_srcrevs.split():
return False
except bb.fetch2.FetchError:
pass
need_update_list = []
def need_update_submodule(ud, url, module, modpath, workdir, d):
url += ";bareclone=1;nobranch=1"
@@ -172,11 +162,6 @@ class GitSM(Git):
shutil.rmtree(tmpdir)
else:
self.process_submodules(ud, ud.clonedir, need_update_submodule, d)
if len(need_update_list) == 0:
# We already have the required commits of all submodules. Drop
# a nugget so we don't need to check again.
runfetchcmd("%s config --add bitbake.srcrev %s" % \
(ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir)
if len(need_update_list) > 0:
logger.debug('gitsm: Submodules requiring update: %s' % (' '.join(need_update_list)))
@@ -209,9 +194,6 @@ class GitSM(Git):
shutil.rmtree(tmpdir)
else:
self.process_submodules(ud, ud.clonedir, download_submodule, d)
# Drop a nugget for the srcrev we've fetched (used by need_update)
runfetchcmd("%s config --add bitbake.srcrev %s" % \
(ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir)
def unpack(self, ud, destdir, d):
def unpack_submodules(ud, url, module, modpath, workdir, d):