We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:
socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null
and pointing SSTATE_MIRRORS in OE at that address.
Adding a timeout to the checkstatus calls of 30s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.
[YOCTO #13716]
(Bitbake rev: c0348de8121c3a842bf44906f7e2f79e93f7275b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.
(Bitbake rev: c222eddcebe892ae209aea7776cfc1147ac1df6e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.
(Bitbake rev: c5b8a2fce98c362ea77d74a8bc472d01b739a98a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b3cccaa6a896c41d8c9be5eebc327f726542d16b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a potential race over the mirror tarballs where a partial git repo
could be extracted causing fetcher failures if the tarball is being rewritten
whilst another build accesses it.
Create the mirror tarball atomically to avoid this.
[YOCTO #14441]
(Bitbake rev: e3da0ecbd282da060b52a4bcf3ed36497295fde0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3250bc950c56bd7dd2114df26e5a8e13b04ceac8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.
This issue was introduced in this
commit: 977b7268bf
[YOCTO #14283]
(Bitbake rev: b4acaa314cd8bc86c50e14464ff01c2d110122c4)
Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e05d79a6ed92c9ce17b90fd5fb6186898a7b3bf8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With introduction of python3targetconfig class in core this is no longer
needed.
(Bitbake rev: fa94374baea75a94e3a488126ca7d8e241a77acd)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 47b64cfacd7c498ef9ed5486d117f2d69a39f225)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.
[YOCTO #14191]
(Bitbake rev: 2351b496bb63b96920d4ae67bec816f00d510df2)
Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0efac66043662e7a2027192f50e92e982db2ba1c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Function "contains_lfs" was only looking at the master branch when searching for LFS
content. LFS may be configured in specific branches only, so we need to use the
correct branch.
(Bitbake rev: 274f790ac9161187810df3242d3570261e407b11)
Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4fa67c2af830035a1ddedc14592ee25a15ebff22)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The message "Repository %s has LFS content but it is not being fetched" was
being printed, even when Git-LFS was available and "lfs=1" was set. In those
situations, we want to fetch LFS content, so that message would not make sense.
(Bitbake rev: 2795c6150c0b3218716195c0d10ff2843b1ef695)
Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 45028dfda5a29a34ab408cb3f11d72ae17963340)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Git-LFS objects were being fetched even when lfs=0 was not set.
This patch disables LFS smudging when lfs=0. That way, only the LFS pointers
are downloaded during checkout.
(Bitbake rev: 3d09608cabd404733b95ea92d5b00830ded77834)
Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 646d86df7de774255246a3d7051c308e43eb257d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Microsoft's TFS VCS system allows for spaces in a git repository url.
An example of a valid url is:
ssh://tfs-my-company.org:22/tfs/My Projects/FooBar
This commit adds support for such urls by implementing two changes.
Firstly, when bitbake makes a git command line call the url is
surrounded by quotes so that the url, regardless of spaces, is
treated as one argument. Secondly, additional parsing of various
filepath variables, which are based off of the url, are now
completed with any spaces in the url replaced with underscores.
(Bitbake rev: c4a51b7f4fd8a3c7d63c184675c014ec955d2606)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eb38b6f0935763f7ba19e5618f376fcae1dac41a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.
We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.
I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.
(Bitbake rev: 6e527ea72b0816bf5f454473faf6523c365daa18)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 820ab886e79eea516560c0c008e4cf059c6e11a3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a website is behind a CDN like Cloudflare there may be a "Browser
Integrity Check" or other test applied to requests before they are
allowed through to the server. Downloading via wget passes these tests
as headers are set appropriately, however the Python urllib module may
fail these tests unless additional headers are set. This causes
Wget.checkstatus() to fail where Wget.download() would actually succeed.
For Cloudflare in particular a valid User-Agent is needed, it's easy to
add this to the headers in Wget.checkstatus(). The user agent string is
copied from Wget._fetch_index().
(Bitbake rev: 4679d3cdb9cdf23f3962aa61c599ad7474591f9f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Upon inspection its clear the way the keys for this cache were being handled
would break it and cause the cache to never be used. Fix this.
(Bitbake rev: 9a5dd1be63395c76d3fac2c3c7ba6557fe47b442)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As per mailing list discussion, the cache policy was behaving correctly before
and wouldn't expect to update after the initial fetch even for AUTOREV.
This reverts commit ba093a38539960e645e994a66ed7872a604c00a9.
(Bitbake rev: 51f827911b7202de3e855e683fdbd732d7a84e09)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked temporarily so that the .gitmodules file can be examined.
(Bitbake rev: 3987db953e414255ce278bc25a5f6cec0f2a30c7)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake will list many checksums for a recipe when none are
present, encouraging users to add them all to a recipe. We don't
need/want them all.
We used to show md5 and sha256 but given the concerns about md5,
switch to showing just sha256 going forward which seems like the
sensible one to standardise upon.
There will be no change to existing recipe functionality.
(Bitbake rev: 47f0c849ed13ba554d9523b926d92405e8251702)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The update codepath would trigger:
Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment
Fix this so the code functions as intended in both fetch and update cases.
[YOCTO #13798]
(Bitbake rev: 16c4e930ff37ea6eac2ac0cb2197908ce3a1cc53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Documentation says:
"path_spec": A specific directory in which to checkout the specified
svn module.
but existing svn fetcher uses "module" always as path of checked out
svn-module, regardless whether path_spec was given or not.
(Bitbake rev: 75223644ab9bc94fc268f1bab775e66c4188f279)
Signed-off-by: Jens Rehsack <sno@NetBSD.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mark any keys used to cache the srcrevs for a recipe as "dontcache" if
BB_DONT_CACHE is set for the recipe. Remove any such keys upon the
next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will
make sure the srcrev is updated as expected if ${AUTOREV} is used.
(Bitbake rev: ba093a38539960e645e994a66ed7872a604c00a9)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This seems to have been broken for a very long time. Now it also works
regardless of BB_SRCREV_POLICY.
(Bitbake rev: ffd663a8e07e2e39e8ca2d2493f4f98037c5f9e4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit adds a new npmsw fetcher that fetches every npm dependencies
described in a npm shrinkwrap file:
https://docs.npmjs.com/files/shrinkwrap.json.html
The main package must be fetched separately:
SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \
npmsw://${THISDIR}/npm-shrinkwrap.json"
Since a separation has been created between the package and its
dependencies, the package can also be fetched with a non npm fetcher
without impacting the general behavior:
SRC_URI = "git://github.com/foo/bar.git;protocol=https \
npmsw://${THISDIR}/npm-shrinkwrap.json"
(Bitbake rev: f5223be54450bf20e0bfbd53b372a7748a44b475)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit refactors the npm fetcher to improve some points and fix
others:
- The big change is that the fetcher is only fetching the package
source and no more the dependencies. Thus the npm fetcher act as the
other fetchers e.g git, wget. The dependencies will be handled later.
- The fetcher only resolves the url of the package using 'npm view' and
then forwards it to a proxy fetcher.
- This commit also fixes a lot of issues with the package names (exotic
characters, scoped packages) which were badly handled.
- The validation files - lockdown.json and npm-shrinkwrap.json - are no
longer used by the fetcher. Instead, the downloaded tarball is
verified with the 'integrity' and 'shasum' provided in the 'npm view'
of the package [1][2].
1: https://docs.npmjs.com/files/package-lock.json#integrity
2: https://www.w3.org/TR/SRI
(Bitbake rev: 0f451cdc43130d503ada53ed1b4fc5a24943f6ef)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using a download filename with characters which can be interpreted
by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the
filename fixes the issue.
(Bitbake rev: ed652dce5200161068eccdbfaaaefde33136eb09)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the done condition to a
proxy fetcher.
(Bitbake rev: ee3a2545e99e6e99559a72bcda64797ae674ec71)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "try_mirrors" functions to
a proxy fetcher.
(Bitbake rev: 462c9a2b368a1720da276310b1d5d0423b7cefea)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "verify_donestamp" and
"update_stamp" functions to a proxy fetcher.
(Bitbake rev: f7612c0704b4252bba5157ce9a94d8888c6d0760)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit enables the "sha1", "sha384" and "sha512" hash functions in
the supported checksum list. This allows to use more SRC_URI checksums
functions for a url:
SRC_URI[sha1sum] = "..."
SRC_URI[sha384sum] = "..."
SRC_URI[sha512sum] = "..."
The npm fetcher needs this to support subresource integrity:
https://www.w3.org/TR/SRI/
(Bitbake rev: cd80a646aa841b71e68282bb8d11194abb5df0e4)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit refactors the way checksums are verified to be more generic.
The support of new hash functions is now limited to the update of the
CHECKSUM_LIST variable.
(Bitbake rev: debd9eeaf5638755d8956b2d65b904fe02826966)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From a SRCREV caching point of view, there is no reason to treat the
following upstreams as different:
SRC_URI = "git://github.com/file/file.git"
SRC_URI = "git://github.com//file/file.git"
(Bitbake rev: 425e21c14955dd38868c6e97637df3bbe0f89fac)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior to this change, two different recipes pulling from the same
exact repo could get a different SRCREV during a single parse session.
This was originally observed using git. For git at least, it still
allows recipes to pull from the same repo, but with different branches
or tags, since the form of the srcrev cache key for git is:
"git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name]
Where the 'unresolvedrev' part is the branch or tag name.
(Bitbake rev: 6c938e6fd29beebe09b32be839dae008fe6491d2)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.
Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"
(Bitbake rev: 923aff060d8aba8456979c35b16d300ba7c13ff9)
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to ensure the correct working directory is used for this
tar command to succeed.
(Bitbake rev: e1a2a0e29a75dead3426ae083387181ad52da91f)
Signed-off-by: Christian Herzig <cherzig@gauselmann.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/lib/bb/fetch2/clearcase.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="?
if command is 'mkview':
bitbake/lib/bb/fetch2/clearcase.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif command is 'rmview':
bitbake/lib/bb/fetch2/clearcase.py:159: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif command is 'setcs':
Python 3.8 is quite correct and we so mean "==" here, fix it to
avoid the warnings.
(Bitbake rev: 1fab03f6e10eaa13b8a89ce0b2f9fe8ce5157189)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix mercurial fetching after breakage from changes to the core fetcher.
Fix username and password usage and setting moddir needed by setup_revisions.
(Bitbake rev: 8962b27abc68427eae085624f26300108ae88aa1)
Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a weird error in OE-Core where "devtool modify virtual/kernel"
was showing basehash mismatch errors. This was due to SRCPV sometimes being:
AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28.
The latter hash comes from KBRANCH and meant sometimes the correct branch
was seen, sometimes it was not. The issue was complicated by the execution
using a remote datastore over tinfoil.
The problem turns out to be a fetcher caching error. If the datastore
changes, the cached url data may not be valid.
We therefore ensure we match cached url data against the datastore that
generated it, which appears to fix this issue.
(Bitbake rev: 0540abd338ed2ffd822edbd2947cab2f18873422)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When bitbake determines if a git clone needs updating, it only checks for the
needed srcrevs, not the revs listed in BB_GIT_SHALLOW_REVS, which will fail if
using shallow and the needed rev was added to the upstream git repo after a
previous fetch. Ensure that we also check for shallow revs.
[YOCTO #13586]
(Bitbake rev: 93b65e3c6ca64f644946953980595c44fbbcc748)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refactor the git-lfs checking: this means both clearer code in the download()
function and allows unit testing to monkeypatch the functionality.
(Bitbake rev: 33cf9172ded50a869f7201ba463ab9ecc69b8252)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new 'lfs' option to the git fetcher so that we can optionally not fetch
git-lfs content, for repositories that contain LFS data that we don't
actually need for building.
By default lfs is set to 1, so if the repository has LFS content then git-lfs is
required. Setting lfs to 0 will mean that git-lfs won't be required to fetch,
and some files will be missing.
(Bitbake rev: be0b78ccfc5ede98041bc0545a15092494b12b26)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not all servers and repositories have this property set
which results in failures like this when actual svn checkout
command succeeded:
svn: warning: W200017: Property 'svn:externals' not found on ''
svn: E200000: A problem occurred; see other errors for details
(Bitbake rev: 238636f033cbf18e5741f0ea0e64db40e84f5838)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* noticed on read-only sshfs premirror
* it was showing the warning about renaming the file:
WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441
and then failed because of movefile() issue with python3 (fixed in previous commit):
ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:
with movefile() fixed, it let do_fetch continue and re-fetch locally with the right
checksum, but still the renamed file didn't exist, because of movefile failure - add
another warning when the movefile fails - for whatever reason - unfortunately movefile
prints error messages with just print() so the real error is hidden only in log.do_fetch
in this case:
movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441'
(Bitbake rev: 9a1bf4ba9ec00c2a222d820f8f83d1f056b021d6)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code says that some servers respond with 403 codes when they really
mean 405 codes. But we still need to account for legitimate 403 codes.
Before this change, I noticed that sstate mirror checking was taking a
very long time when I purposely entered incorrect credentials into my
.netrc file for our sstate mirror. Instrumenting the code, I discovered
tracebacks like the following for every mirror access attempt:
File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus
fetcher.checkstatus()
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus
ret = try_mirrors(self, self.d, ud, mirrors, True)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors
ret = try_mirror_url(fetch, origud, uds[index], ld, check)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url
found = ud.method.checkstatus(fetch, ud, ld)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus
opener.open(r)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 504, in error
result = self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
unverifiable=True))
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 504, in error
result = self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
unverifiable=True))
... (repeats until recursion depth is reached)
Solution is to make sure we only attempt the GET request once when handling 403/405 error codes.
(Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using npm pack to download the main node module and its dependencies
allow for the use of private npm modules and access to them via .npmrc
(Bitbake rev: e5eda3871893e4eadeb311aeb997e183675598f4)
Signed-off-by: Mads Andreasen <mads@andreasen.cc>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME;
unfortunately, this also affects Python scripts run as fetch commands like
git-make-shallow, breaking it with a message like
Failed to import the site module
Traceback (most recent call last):
File "/usr/lib/python3.6/site.py", line 570, in <module>
main()
File "/usr/lib/python3.6/site.py", line 556, in main
known_paths = addusersitepackages(known_paths)
File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages
user_site = getusersitepackages()
File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/usr/lib/python3.6/site.py", line 254, in getuserbase
USER_BASE = get_config_var('userbase')
File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var
return get_config_vars().get(name)
File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars
_init_posix(_CONFIG_VARS)
File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix
_temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata'
on an Ubuntu 18.04 system (and likely others) when building with
BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS.
Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this.
(Bitbake rev: d94ccd506d04aff182ab48f501f6f366d5dd14f5)
Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #13344]
When parsing manually the 'npm view --json' ouput, an extra closing
brackets in a JSON string can leads the fetcher to fail with a
JSONDecodeError exception.
This commit use the JSON parser to extract:
- The last object in the returned array if there are multiple results.
- The returned object if there is only one result.
(Bitbake rev: 3d319c79981811d3cfd4732885057db4fd5afcc2)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new option to the svn fetcher url "externals=allowed". This will allow
a user to enable svn co w/ externals. However, this does avoid the fetcher,
network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a
deterministic download. The system does attempt to identify SVN repos that
have externals enabled, and will warn the user. It is up to the user to
determine if these are necessary for the recipe. They may disable the warning
by adding "externals=nowarn" to the url.
In the future we would like to parse this list and see if the items are already
in the SRC_URI for that recipe, but with SVN being in limited use these days
that extra work is likely not worth the trouble.
Add test cases that generated a local SVN tree, with an external source
set to github bitbake in svn format. One test case checks that externals are
ignored, and one checks that they in downloaded.
(Bitbake rev: bf53f07c3647e57d8452a7743a2b04bcb72c80d6)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Because we are trying to avoid network activity and use our own fetcher,
the system emulates the behavior of 'git submodule init'.
git submodule init uses the .gitmodules file, where typically the module
name and path are the same. However, in this case the module name and
path (in the tree) were different. i.e.:
[submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"]
path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared
url = https://github.com/Azure/azure-c-shared-utility.git
Previously the code assumed the 'path' was both the checkout location
under .git/modules, as well as the path to extract the components. This
proved to be incorrect as the .git/modules path needs to match the submodule
'name'. This causes the components that were fetched to be initialized in
the wrong location, which later caused the 'git submodule update' process to
skip not properly initialized modules.
A test case was added for this specific case to ensure a regression does
not appear in the future.
(Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
