Paul Barker ab26fdae9e bitbake: fetch2/wget: Set User-Agent when checking status of a URL ...

When a website is behind a CDN like Cloudflare there may be a "Browser
Integrity Check" or other test applied to requests before they are
allowed through to the server. Downloading via wget passes these tests
as headers are set appropriately, however the Python urllib module may
fail these tests unless additional headers are set. This causes
Wget.checkstatus() to fail where Wget.download() would actually succeed.

For Cloudflare in particular a valid User-Agent is needed, it's easy to
add this to the headers in Wget.checkstatus(). The user agent string is
copied from Wget._fetch_index().

(Bitbake rev: 4679d3cdb9cdf23f3962aa61c599ad7474591f9f)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

2020-04-24 14:31:42 +01:00

..

__init__.py

bitbake: fetch2: Fix urldata_cache key issues

2020-03-30 12:33:50 +01:00

bzr.py

bitbake: lib: remove unused imports

2020-01-19 13:31:05 +00:00

clearcase.py

bitbake: lib: remove unused imports

2020-01-19 13:31:05 +00:00

cvs.py

bitbake: lib: remove unused imports

2020-01-19 13:31:05 +00:00

git.py

bitbake: fetch2/git: _revision_key: collapse adjacent slashes

2020-01-19 13:31:05 +00:00

gitannex.py

bitbake: lib: remove unused imports

2020-01-19 13:31:05 +00:00

gitsm.py

bitbake: fetch2/gitsm: Unpack shallow mirror tarballs

2020-03-11 14:43:25 +00:00

hg.py

bitbake: lib: remove unused imports

2020-01-19 13:31:05 +00:00

local.py

bitbake: bitbake: Strip old editor directives from file headers

2019-05-04 10:44:10 +01:00

npm.py

bitbake: fetch2/npm: refactor the npm fetcher

2020-01-27 16:48:10 +00:00

npmsw.py

bitbake: fetch2: add the npmsw fetcher

2020-01-27 16:48:10 +00:00

osc.py

bitbake: lib: amend code to use proper singleton comparisons where possible

2020-01-19 13:31:05 +00:00

perforce.py

bitbake: lib: amend code to use proper singleton comparisons where possible

2020-01-19 13:31:05 +00:00

repo.py

bitbake: bitbake: Strip old editor directives from file headers

2019-05-04 10:44:10 +01:00

s3.py

bitbake: bitbake: Strip old editor directives from file headers

2019-05-04 10:44:10 +01:00

sftp.py

bitbake: bitbake: Strip old editor directives from file headers

2019-05-04 10:44:10 +01:00

ssh.py

bitbake: lib: amend code to use proper singleton comparisons where possible

2020-01-19 13:31:05 +00:00

svn.py

bitbake: fetch2/svn: Avoid UnboundLocalError exception

2020-02-20 11:56:37 +00:00

wget.py

bitbake: fetch2/wget: Set User-Agent when checking status of a URL

2020-04-24 14:31:42 +01:00