Commit Graph

899 Commits

Author SHA1 Message Date
Chen Qi
07f08063c1 bitbake: fetch2/git: fix shallow clone for tag containing slash
If a tag contains slash, e.g., debian/5.22, then shallow clone
fails because it's using a wrong ref.

To reproduce the issue, add the following lines in local.conf:

  BB_GIT_SHALLOW = "1"
  BB_GENERATE_SHALLOW_TARBALLS = "1"

And then run 'bitbake debianutils -c fetch'.

What the original os.path.basename(ref) wanted to do is to remove
the strings such as refs/heads/. So we do it explitly to fix this
issue.

Fixes: [YOCTO #15862]

(Bitbake rev: c6d6999f1ed01e7445b8f177a888038edacf555c)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-15 10:05:49 +01:00
Philip Lorenz
3eeac69385 bitbake: fetch2: Fix LFS object checkout in submodules
Skipping smudging prevents the LFS objects from replacing their
placeholder files when `git submodule update` actually checks out the
target revision in the submodule. Smudging cannot happen earlier as the
clone stored in `.git/modules` is bare.

This should be fine as long as all LFS objects are available in the
download cache (which they are after the other fixes are applied).

(Bitbake rev: d270e33a07c50bb9c08861cf9a6dc51e1fd2d874)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 11:37:32 +01:00
Philip Lorenz
eb6d89e9e6 bitbake: fetch2: Fix incorrect lfs parametrization for submodules
The existing code would pass `True` or `False` to the git fetcher. As
the fetcher expects `lfs` to be set to `1` this always lead to LFS
fetching being disabled.

(Bitbake rev: 5e487a5a096400271ed1e29b0df72903f2304e49)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 11:37:32 +01:00
Philip Lorenz
9962d198c8 bitbake: fetch2: Use git-lfs fetch to download objects
Its not clear which exact git-lfs versions failed to work with bare
repositories, however git-lfs 2.13.2 which is shipped by Debian 10 (i.e.
the oldest supported distribution by scarthgap) shows no issue when
fetching into a bare repository. Switch to git-lfs fetch which in turn
eliminates issues seen when using the gitsm fetcher with submodules
utilizing lfs.

In these scenarios, fetching of LFS objects did not actually happen as
the gitsm fetcher parametrizes the to be fetched repositories with
`bareclone=1` which in turn means that the target revision was never
checked out (and therefore no LFS objects were fetched).

(Bitbake rev: 501d3317df5780a5b0e3c2562b26ed276c1dbe90)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 11:37:32 +01:00
Philip Lorenz
74945e4678 bitbake: fetch2: Simplify git LFS detection
Its unclear why this function does not operate on the desired source
revision to begin with (which really should be the decider on whether a
particular source revision uses LFS or not). Simplify the decision logic
by always checking the `.gitattributes` file of the target revision.

(Bitbake rev: b3faa0ce5c0a6945f26b5b303a7f38c00d132397)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 11:37:32 +01:00
Philip Lorenz
7642477dc8 bitbake: fetch2: Check for git-lfs existence before using it
So far, existence of `git-lfs` was only checked during unpacking. As the
binary is also used in earlier steps also check for its existence there.

Additionally, factor out the LFS existence check into a dedicated
function and call it wherever git-lfs is used for the first time.

(Bitbake rev: 5818367db9b261b7e07c347d38044e6cba8f9727)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 11:37:32 +01:00
Philip Lorenz
67990e1550 bitbake: fetch2: Clean up no longer used name parameter
There's no need to pass `name` when it is no longer used.

(Bitbake rev: b132d35dee643e270e3e6dd536dcc90334a0111c)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-29 13:11:24 +01:00
Peter Marko
af91ed1691 bitbake: fetch/git: correct typo in comment and style
Correct typo Fash -> Fast.
With it also delete some trailing spaces which are unnecessary and
editors are trying to remove when changing anything in the file.

(Bitbake rev: b96040a0a066344527f0bc6ca4d09cda6b9d2f11)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-10 11:07:38 +01:00
Peter Marko
d1dae30b79 bitbake: fetch/git: always fetch lfs when creating shallow tarball
Before fast shallow tarballs the shallow tarball with lfs worked only
if the original .../downloads/git2/... directory existed. Once it was
not there, do_unpack on shallow tarball without lfs files failed due to
disabled network.
It was fixed for fast shallow tarballs and this commit fixes the other
scenarios, too.

This can happen when such shallow tarball is put to mirror or if
someone does cleanup to reclaim disk space.

(Bitbake rev: 4e6fb31a88448cdacf4e9f84cf0a8a8035f84d63)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-10 11:07:38 +01:00
Julian Haller
f3948d6ee3 bitbake: bitbake: fetch: Fix BB_FETCH_PREMIRRORONLY for git mirror tarballs
When invoking the original git fetcher after downloading a mirror
tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch
commands targeting the upstream source being executed silently. Ensure
setting BB_NO_NETWORK before invoking the original fetcher. While this
was only observed for git, setting this in general for all fetcher
types makes sense at this location.

(Bitbake rev: 1b1321f2b60c0a66159e3f20c6befcb0b3ccc4c7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-10 11:07:38 +01:00
Richard Purdie
49ec431d95 bitbake: fetch2/git: Handle srcrevs for annotated tags in tag check
If SRCREV points at an annotated tag, the comparision code can fail
as the resolved tag might not be the same sha.

Handle this by also resolving the SRCREV. We only need to do this if
they don't match in the first place for a minor performance win.

Also add a test for this.

(Bitbake rev: 136c06e251de68ed64355ec6b47a522ff3a372e3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-03 11:03:43 +01:00
Ross Burton
3233f3b231 bitbake: fetch2/wget: use long arguments for clarity
It's best practise to use long-form arguments in scripts where the
conciseness of short arguments is less useful than in an interactive
terminal.

(Bitbake rev: 54039bc9d169871af6d36578df4c21bff296f6de)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
372a9c2d56 bitbake: fetch2/wget: don't conditionalise --continue
If the target file exists, we pass --continue so that wget will try to
resume the download instead of starting from the beginning.  However if
the file doesn't exist --continue has no effect, so there's no need to
conditionalise the use of the argument.

(Bitbake rev: a5ee50d4d2b2e6c00abe1551382afd9799345dbf)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
f976a7d4fb bitbake: fetch2/wget: consider downloadfilename when checking for upstream
latest_versionstring() currently looks at just the end of the URI when
guessing what the filename to look for is, but this doesn't work if the
URL filename is not simple.

For example, miniupnpd has a SRC_URI of:

  http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz;downloadfilename=${BP}.tar.gz

The filename component of this is "download.php", which causes the
heuristics in latest_versionstring() to exit early.

Instead, if the downloadfilename is set then use that, as it's often the
actual filename that we're after.

(Bitbake rev: 2d5f135e997d13fabda0ad266fd5c928ee33f487)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Christian Lindeberg
c00ad319d4 bitbake: fetch2/gomod: Fix mirroring problem
Build the 'downloadfilename' parameter by replacing path separators in
the module path like the git fetcher builds the mirror tar ball name.
Copy the downloaded file in the fetcher's unpack method like the crate
fetcher instead of calling the base fetcher's unpack method.

(Bitbake rev: 7762cea087597019460d66b04268757bd46befdf)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 11:19:33 +00:00
Richard Purdie
bdd2b0fee1 bitbake: fetch: Drop multiple branch/revision support for single git urls
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).

We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.

This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.

Some test cases are removed since they are no longer needed.

(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-20 13:50:40 +00:00
Richard Purdie
b56a335481 bitbake: fetch/git: Ensure tags work with shallow clones
If we want to validate a tag when using shallow clones, we need to ensure
the tag is imported to the clone. Adding to extra_refs allows this to happen
ensuring tag validation in shallow clones works.

(Bitbake rev: 03945475886b8d31c7eccc80fe594ab5306bcf5d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Richard Purdie
18a533bfe6 bitbake: fetch/git: Rework tag parameter handling
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.

Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.

This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.

The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.

(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Patrik Nordvall
8180865c22 bitbake: fetch2/git: Restore escape quoting for the git url when used
This fixes a bug where escapes in the url path would not be properly
restored for the git commands in the git fetcher. For example, a
space which is encoded as '%20' was not properly encoded before the
clone command.

e.g.

SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https"

resulted in
git clone 'https://git.openembedded.org/bitbake example/bitbake'

instead of
git clone 'https://git.openembedded.org/bitbake%20example/bitbake'

(Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680)

Signed-off-by: Patrik Nordvall <patrik.nordvall95@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-18 11:03:17 +00:00
Eric Meyers
57adda32c5 bitbake: fetch2/npm: Adding npmrc support for private registry authentication
(Bitbake rev: 5fa6137b6d98544766f3152b874e67d04fafb88f)

Signed-off-by: Eric Meyers <eric.meyers@arthrex.com>
Cc: Geoff Parker <geoffrey.parker@arthrex.com>
Cc: Chuck Wolber <chuckwolber@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-09 20:10:06 +00:00
Stefan Herbrechtsmeier
bae86fe618 bitbake: fetch2: Partial revert decodeurl() to not use the URI class
This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f
which changes decodeurl() to use the URI class to parse the URL instead
of doing it itself. While reusing code is generally a good idea, using
urllib.parse.urlparse() (which the URI class does) to parse the regular
expression "URLs" that are used in PREMIRRORS and MIRRORS does not work.
A regular expression URL containing https?://... would be silently
ignored, while a URL using a negative lookahead such as
git://(?!internal\.git\.server).*/.* would result in a cryptic error:

  Exception: re.error: missing ), unterminated subpattern at position 0

The problem is that urllib.parse.urlparse() treats the ? as the start of
URL parameters and thus stops parsing whatever part of the URL it was
parsing. Restore the old function and use it in the PREMIRRORS and
MIRRORS code.

(Bitbake rev: f8a7712754e6d0199a0d227fca288307b935368d)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-07 15:38:15 +00:00
Stefan Koch
22728acd78 bitbake: fetch2/gitsm: Unpack even when ud.clonedir is not available
(Bitbake rev: 13d76361ec37faecd84e7b81da22ada7d4e0ba90)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Stefan Koch
65ae50cd16 bitbake: fetch2/git: Add support for fast initial shallow fetch
When `ud.shallow == 1`:
- Prefer an initial shallow clone over an initial full bare clone,
  while still utilizing any already existing full bare clones.
- If the Git error "Server does not allow request for unadvertised object"
  occurs, the initial full bare clone is fetched automatically.
  This may happen if the Git server does not allow the request
  or if the Git client has issues with this functionality,
  especially with the Git client from Ubuntu 20.04.

This improves:
- Resolve timeout issues during initial clones on slow internet connections
  by reducing the amount of data transferred.
- Eliminate the need to use an HTTPS tarball `SRC_URI`
  to reduce data transfer.
- Allow SSH-based authentication (e.g. cert and agent-based) when
  using non-public repos, so additional HTTPS tokens may not be required.

(Bitbake rev: 457288b2fda86fd00cdcaefac616129b0029e1f9)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Michael Estner
e1836b205a bitbake: lib: Remove double imports
* Remove double imports mentioned by pylint

(Bitbake rev: 741db6719efca5aa9ef2c15e60cdd624e4aa1a8d)

Signed-off-by: Michael Estner <michaelestner@web.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-27 15:12:55 +00:00
Richard Purdie
ba91179519 bitbake: fetch2: Fix BB_FETCH_PREMIRRORONLY network disabling
When using BB_FETCH_PREMIRRORONLY we write to the datastore to disable the network.
This change needs to be undo when handling later urls, so operate on a copy of the
datastore to allow this.

Reported by Julian Haller <julian.haller@philips.com>

(Bitbake rev: 67a5ede8ae92ed7dcad29fd0dcfd62c6640b10b2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-25 12:49:52 +00:00
Stefan Herbrechtsmeier
c49684b48b bitbake: fetch2: clearcase: remove double DL_DIR from localfile
(Bitbake rev: bcf090ed631bbd523a5341baebba0765f1a847f8)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
f62042523a bitbake: fetch2: ssh: use common localpath handling
(Bitbake rev: 22ac6e84c70034a74f1729c7e0f31ca928fa1dea)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
170dd77e4a bitbake: fetch2: remove unnecessary unquote
The URI path is already unquoted. Remove the unnecessary unquote
function calls for URI path values.

(Bitbake rev: 3de12bbc28b5a4189b849720735cf3e268d3941d)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
f17c51c695 bitbake: fetch2: local: use path variable
Use the path variable from the fetch data instead of decoding the path
manually from the plain unexpanded url.

(Bitbake rev: ad3a29fa6ea53741d4e1786de35f8e7fc4292e7a)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
3e543e8eaa bitbake: fetch2: remove unnecessary expand function calls
The fetch data class already expands the type, host, path, user, pswd
and parm variables. The fetcher classes already expand the localfile
variable. The getVar function expands the returned string per default.
Remove unnecessary expand function calls to simplify the code.

(Bitbake rev: 1b1eb037b861fbf20491ac17e519e9eaf232b858)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
2935d76bb4 bitbake: fetch2: remove duplicated code in url decode and encode
Use the URI class to decode and encode an URL. Remove duplicate code and
unify the behavior.

(Bitbake rev: a5d569c94700f04b8193c6bccae5af619931b00f)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
2b71696f35 bitbake: fetch2: do not decode user from file URI
A file URI can't contain a user. Do not treat the @ as reserved
character for a file URI.

(Bitbake rev: 11cf4062f48536547a352e24d6e963d91fdd1190)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Bin Lan
33271912bf bitbake: bb/fetch2/__init__.py: remove a DeprecationWarning in uri_replace()
There is the following warning when executing to bitbake linux-yocto:
  bitbake/lib/bb/fetch2/__init__.py:464: DeprecationWarning: 'count' is passed as positional argument

This is because the 4th parameter of re.sub(pattern, repl, string, count=0, flags=0)
is a keyword parameter. We use keyword arguments for parameters that are not positional.

(Bitbake rev: 4bd62639f7023eec9f55ad7909fa9250538de936)

Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-14 09:51:45 +00:00
Stefan Herbrechtsmeier
f00e8e1233 bitbake: utils: add Go mod h1 checksum support
Add support for the Go mod h1 hash. The hash is based on the Go dirhash
package. The package defines hashes over directory trees and is uses for
Go mod files and zip archives.

(Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-11 18:35:01 +00:00
Stefan Herbrechtsmeier
b242f0a007 bitbake: fetch2: read checksum from SRC_URI flag for npm
(Bitbake rev: 8d3232152e545be958f8f3b069e7477c6f310583)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Stefan Herbrechtsmeier
d7c6afc9a9 bitbake: fetch2: do not prefix embedded checksums
The fetcher support entries with an embedded checksum like 'sha256sum'
in the SRC_URI. It adds the parameter 'name' as prefix to the checksums
if the parameter is set. This behavior is unexpected and leads to hacks
in fetchers. Fallback to the checksum without the useless prefix and
set the parameter 'name' in the gomod fetcher unconditional.

(Bitbake rev: 7a86c5a20ea2586f1ae240613644e065e7b21683)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Stefan Herbrechtsmeier
eb0c87fa4c bitbake: fetch2: npmsw: remove old lockfile format support
Remove support for the old lockfile format. The old lockfile format is
required by npm 6 / Node.js 14 which is out of maintenance [2].

[1] https://docs.npmjs.com/cli/v6/configuring-npm/package-lock-json
[2] https://nodejs.org/en/about/previous-releases

(Bitbake rev: 7824e19483d9b60a259d6e3a4c7068fade94f2bf)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Ross Burton
c03be6b81c bitbake: fetch2/wget: correctly construct the target URL in checkstatus()
ud.path has been unescaped (eg %20 is space) but as we're reconstructing
a URL we should re-escape it. For example, unzip has a SRC_URI
containing "UnZip%206.x%20%28latest%29/UnZip%206.0/unzip60.tar.gz" which
then throws exceptions if the unescaped string " (latest)" is used.

Also, this code uses the extracted ud.host and ud.path variables. These
are unescaped but potentially stale as eg the cargo fetcher subclasses
Wget() and reassigns ud.url on construction.

Simplify the code by reconstructing a URL from ud.url directly instead
of bouncing through intermediate variables that may be wrong or
unescaped.

(Bitbake rev: c9906bb289dcfd9ae41f10bd5399ccc17a4cc437)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:25:28 +00:00
Ross Burton
e8ab6bc411 bitbake: fetch2/wget: handle HTTP 308 Permanent Redirect
urllib2.HTTPRedirectHandler.redirect_request doesn't handle HTTP reponse
code 308 (Permanent Redirect). This was fixed in c379bc5 but can't be
worked around without copying the entire redirect_request() method.

When we can depend on Python 3.13, FixedHTTPRedirectHandler can be
removed.

(Bitbake rev: 365829a2803b954ee6cb0364749551a91d806075)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:25:28 +00:00
Stefan Herbrechtsmeier
02f8ea6865 bitbake: fetch2: npmsw: ignore bundled dependencies in packages
The npm package lock and shrinkwrap file list bundled dependencies which
are supplied together with the parent dependency. The flag depends on
the lock file version. Add the missing `inBundle` flag to packages list
processing.

(Bitbake rev: 042d7121ddc93c18306bf03d26ea774f66379a90)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:18:59 +00:00
Pascal Eberhard
b35dc1a99d bitbake: fetch2: fix unpack of a single zstd file
do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when
uncompressing a single 'file.zst', zstd command line returns the error
'unexpected end of file', and the input file 'file.zst' becomes empty.

The command line is 'zstd --decompress --stdout file.zst > file.zst'.
So the issue is that the input and output filenames are the same.

Fix the output filename when using zstd for a single file.

(Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8)

Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-29 18:26:14 +00:00
Richard Purdie
b4bd930fb2 bitbake: fetch/wget: Increase timeout to 100s from 30s
Testing shows the worst case CDN response time can be up to 100s. The wget fetcher
is used for accessing sstate from the CDN so increase our timeouts there to match
our worst case repsonse times.

(Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Robert Yang
0a45ba64ad bitbake: git: Clean broken symlink
The file may be a symlink when mirror from local disk, and the symlink will be
broken when the linked file is removed, use os.path.islink() to check and
remove the broken symlink.

(Bitbake rev: d0deb5fe6e7a84b07b49ef685b209729d2fdc780)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
dd9d183621 bitbake: git: Clean shallow mirror tarball
Fixed:
BB_GIT_SHALLOW = "1"
BB_GENERATE_SHALLOW_TARBALLS = "1"
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall

The downloads/gitsmshallow_github.com.nemtrif.*.tar.gz won't be cleaned without
this fix.

(Bitbake rev: bab7a8970a0237a9d24217685a595e76a1336c07)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
5bce38fbae bitbake: gitsm: Add clean function
Fixed:
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall

The downloads/git2/github.com.nemtrif.ftest won't be cleaned without this fix.

(Bitbake rev: 79f25fc5c1b8d0e08540f4aa07875309f5325f47)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
8d9c02ccfc bitbake: gitsm: Remove downloads/tmpdir when failed
The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix.

(Bitbake rev: 2ba8d3214759142afc11f0a88d80eb30a8bcde3a)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
04c1ee924b bitbake: gitsm: Add call_process_submodules() to remove duplicated code
There are 14 lines can be removed, and can make it easy to maintain.

(Bitbake rev: 0ea2c1ac079d63349407a69172ff80cd9acc7252)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Alexander Kanavin
74c93ec961 bitbake: fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()
This eliminates the last usage of 'fake mozilla' in bitbake, and
it's then truthful everywhere about presenting itself, or wget
(when that is used).

I understand this will make people nervous so I want to provide
an extended decription.

1. How was this tested?

- bitbake-selftest -k FetchCheckStatusTest
(tests a few hardcoded URIs, all passed)

- bitbake -k -c checkuri world
(runs checkstatus() over all recipes in oe-core, and all passed again -
this hopefully goes a long way to reassure everyone that hosts around
the world and various CDNs typically do not have a problem with user-agent
strings they haven't seen before or bitbake user-agent specifically)

2. What about that removed cloudflare comment?

I digged into git history, and I think it is not fully accurate. First, 'fake
mozilla' agent is used only for checkstatus() - in actual fetching with wget
it is not. And that has not been a problem for anyone.

Second, here's how the comment occured. Usage of 'fake mozilla' was introduced here:
https://git.yoctoproject.org/poky/commit/?h=master&id=ab26fdae9e5ae56bb84196698d3fa4fd568fe903

At that point it did not have to be specifically 'mozilla', the commit message
indicates that any User-Agent would have been ok. Mozilla was simply copied
from upstream version check for convenience.

Later on, the string was updated to a more recent Mozilla:
https://git.yoctoproject.org/poky/commit/?h=master&id=9f123238261a68e37cec634782e9320633cac5d4

The claim in the added comment become something else: that User-Agent *must* a browser,
without evidence or tests. Even though it demonstrably doesn't have to be - wget is ok.

3. What if someone has a server that is ok with wget agent, but not ok with bitbake agent?

Please see point one. It's not impossible but I think it's highly unlikely. I do think
we should rather tell servers the truth, and learn where the actual issues are. Then
we can consider options - whether that would be pretending to be wget, or allowing user-agent
to be configured. We should also add such servers to bitbake-selftest so we know what they
are.

(Bitbake rev: 234f9e810494394527f59fdf22eb86435d046d53)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-22 11:16:32 +01:00
Alexander Kanavin
9eaa1e5024 bitbake: fetch2/wget.py: use default wget user-agent in upstream version checks
This has been set to various mozilla versions from this point onwards without
explanation or specific examples:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6

However, sourceforge has recently banned the current 'fake mozilla' version
we've been supplying, resulting in broken version checks:

alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk
...
--2024-10-12 11:53:23--  https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ...
Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-10-12 11:53:24 ERROR 403: Forbidden.

gptfdisk                  1.0.10          UNKNOWN_BROKEN  Alexander Kanavin <alex.kanavin@gmail.com>

On the other hand, using wget's default user-agent works just fine:

--2024-10-12 12:16:06--  https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ...
Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

(Bitbake rev: 138117dfc05248dcb48707c838c8c1cc09f12d07)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-22 11:16:32 +01:00
Oliver Kästner
aa546ea0c3 bitbake: fetch2: add GitHub Codespaces support
GitHub Codespaces allow users to use GitHub-hosted pre-configured Docker
containers to work on GitHub repositories in VS Code within the browser
or using a locally running instance of VS Code [1].

Users can define access permissions for Codespaces containers, allowing
them to clone or work with other private repositories [2].
To do this, a git credential helper is injected into the container at
/.codespaces/bin/gitcredential_github.sh, which contains the following
lines:

    #!/bin/sh
    echo protocol=https
    echo host=github.com
    echo path=
    echo username=PersonalAccessToken
    echo password=$GITHUB_TOKEN

The `GITHUB_TOKEN` environment variable is automatically set when the
Codespaces container is created. If authorized by the user, it has the
specified permissions. It is thus required to be able to fetch private
git repos from a Codespace container, so add it to `FETCH_EXPORT_VARS`
to make it available for invocations of git.

[1] https://docs.github.com/en/codespaces/overview
[2] https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces

(Bitbake rev: 71ca83bdd541f70737a3a85f05e40c222611af2f)

Signed-off-by: Oliver Kästner <git@oliver-kaestner.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-14 21:44:38 +01:00