Commit Graph

889 Commits

Author SHA1 Message Date
Richard Purdie
49ec431d95 bitbake: fetch2/git: Handle srcrevs for annotated tags in tag check
If SRCREV points at an annotated tag, the comparision code can fail
as the resolved tag might not be the same sha.

Handle this by also resolving the SRCREV. We only need to do this if
they don't match in the first place for a minor performance win.

Also add a test for this.

(Bitbake rev: 136c06e251de68ed64355ec6b47a522ff3a372e3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-03 11:03:43 +01:00
Ross Burton
3233f3b231 bitbake: fetch2/wget: use long arguments for clarity
It's best practise to use long-form arguments in scripts where the
conciseness of short arguments is less useful than in an interactive
terminal.

(Bitbake rev: 54039bc9d169871af6d36578df4c21bff296f6de)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
372a9c2d56 bitbake: fetch2/wget: don't conditionalise --continue
If the target file exists, we pass --continue so that wget will try to
resume the download instead of starting from the beginning.  However if
the file doesn't exist --continue has no effect, so there's no need to
conditionalise the use of the argument.

(Bitbake rev: a5ee50d4d2b2e6c00abe1551382afd9799345dbf)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
f976a7d4fb bitbake: fetch2/wget: consider downloadfilename when checking for upstream
latest_versionstring() currently looks at just the end of the URI when
guessing what the filename to look for is, but this doesn't work if the
URL filename is not simple.

For example, miniupnpd has a SRC_URI of:

  http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz;downloadfilename=${BP}.tar.gz

The filename component of this is "download.php", which causes the
heuristics in latest_versionstring() to exit early.

Instead, if the downloadfilename is set then use that, as it's often the
actual filename that we're after.

(Bitbake rev: 2d5f135e997d13fabda0ad266fd5c928ee33f487)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Christian Lindeberg
c00ad319d4 bitbake: fetch2/gomod: Fix mirroring problem
Build the 'downloadfilename' parameter by replacing path separators in
the module path like the git fetcher builds the mirror tar ball name.
Copy the downloaded file in the fetcher's unpack method like the crate
fetcher instead of calling the base fetcher's unpack method.

(Bitbake rev: 7762cea087597019460d66b04268757bd46befdf)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 11:19:33 +00:00
Richard Purdie
bdd2b0fee1 bitbake: fetch: Drop multiple branch/revision support for single git urls
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).

We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.

This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.

Some test cases are removed since they are no longer needed.

(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-20 13:50:40 +00:00
Richard Purdie
b56a335481 bitbake: fetch/git: Ensure tags work with shallow clones
If we want to validate a tag when using shallow clones, we need to ensure
the tag is imported to the clone. Adding to extra_refs allows this to happen
ensuring tag validation in shallow clones works.

(Bitbake rev: 03945475886b8d31c7eccc80fe594ab5306bcf5d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Richard Purdie
18a533bfe6 bitbake: fetch/git: Rework tag parameter handling
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.

Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.

This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.

The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.

(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Patrik Nordvall
8180865c22 bitbake: fetch2/git: Restore escape quoting for the git url when used
This fixes a bug where escapes in the url path would not be properly
restored for the git commands in the git fetcher. For example, a
space which is encoded as '%20' was not properly encoded before the
clone command.

e.g.

SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https"

resulted in
git clone 'https://git.openembedded.org/bitbake example/bitbake'

instead of
git clone 'https://git.openembedded.org/bitbake%20example/bitbake'

(Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680)

Signed-off-by: Patrik Nordvall <patrik.nordvall95@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-18 11:03:17 +00:00
Eric Meyers
57adda32c5 bitbake: fetch2/npm: Adding npmrc support for private registry authentication
(Bitbake rev: 5fa6137b6d98544766f3152b874e67d04fafb88f)

Signed-off-by: Eric Meyers <eric.meyers@arthrex.com>
Cc: Geoff Parker <geoffrey.parker@arthrex.com>
Cc: Chuck Wolber <chuckwolber@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-09 20:10:06 +00:00
Stefan Herbrechtsmeier
bae86fe618 bitbake: fetch2: Partial revert decodeurl() to not use the URI class
This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f
which changes decodeurl() to use the URI class to parse the URL instead
of doing it itself. While reusing code is generally a good idea, using
urllib.parse.urlparse() (which the URI class does) to parse the regular
expression "URLs" that are used in PREMIRRORS and MIRRORS does not work.
A regular expression URL containing https?://... would be silently
ignored, while a URL using a negative lookahead such as
git://(?!internal\.git\.server).*/.* would result in a cryptic error:

  Exception: re.error: missing ), unterminated subpattern at position 0

The problem is that urllib.parse.urlparse() treats the ? as the start of
URL parameters and thus stops parsing whatever part of the URL it was
parsing. Restore the old function and use it in the PREMIRRORS and
MIRRORS code.

(Bitbake rev: f8a7712754e6d0199a0d227fca288307b935368d)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-07 15:38:15 +00:00
Stefan Koch
22728acd78 bitbake: fetch2/gitsm: Unpack even when ud.clonedir is not available
(Bitbake rev: 13d76361ec37faecd84e7b81da22ada7d4e0ba90)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Stefan Koch
65ae50cd16 bitbake: fetch2/git: Add support for fast initial shallow fetch
When `ud.shallow == 1`:
- Prefer an initial shallow clone over an initial full bare clone,
  while still utilizing any already existing full bare clones.
- If the Git error "Server does not allow request for unadvertised object"
  occurs, the initial full bare clone is fetched automatically.
  This may happen if the Git server does not allow the request
  or if the Git client has issues with this functionality,
  especially with the Git client from Ubuntu 20.04.

This improves:
- Resolve timeout issues during initial clones on slow internet connections
  by reducing the amount of data transferred.
- Eliminate the need to use an HTTPS tarball `SRC_URI`
  to reduce data transfer.
- Allow SSH-based authentication (e.g. cert and agent-based) when
  using non-public repos, so additional HTTPS tokens may not be required.

(Bitbake rev: 457288b2fda86fd00cdcaefac616129b0029e1f9)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Michael Estner
e1836b205a bitbake: lib: Remove double imports
* Remove double imports mentioned by pylint

(Bitbake rev: 741db6719efca5aa9ef2c15e60cdd624e4aa1a8d)

Signed-off-by: Michael Estner <michaelestner@web.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-27 15:12:55 +00:00
Richard Purdie
ba91179519 bitbake: fetch2: Fix BB_FETCH_PREMIRRORONLY network disabling
When using BB_FETCH_PREMIRRORONLY we write to the datastore to disable the network.
This change needs to be undo when handling later urls, so operate on a copy of the
datastore to allow this.

Reported by Julian Haller <julian.haller@philips.com>

(Bitbake rev: 67a5ede8ae92ed7dcad29fd0dcfd62c6640b10b2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-25 12:49:52 +00:00
Stefan Herbrechtsmeier
c49684b48b bitbake: fetch2: clearcase: remove double DL_DIR from localfile
(Bitbake rev: bcf090ed631bbd523a5341baebba0765f1a847f8)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
f62042523a bitbake: fetch2: ssh: use common localpath handling
(Bitbake rev: 22ac6e84c70034a74f1729c7e0f31ca928fa1dea)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
170dd77e4a bitbake: fetch2: remove unnecessary unquote
The URI path is already unquoted. Remove the unnecessary unquote
function calls for URI path values.

(Bitbake rev: 3de12bbc28b5a4189b849720735cf3e268d3941d)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
f17c51c695 bitbake: fetch2: local: use path variable
Use the path variable from the fetch data instead of decoding the path
manually from the plain unexpanded url.

(Bitbake rev: ad3a29fa6ea53741d4e1786de35f8e7fc4292e7a)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
3e543e8eaa bitbake: fetch2: remove unnecessary expand function calls
The fetch data class already expands the type, host, path, user, pswd
and parm variables. The fetcher classes already expand the localfile
variable. The getVar function expands the returned string per default.
Remove unnecessary expand function calls to simplify the code.

(Bitbake rev: 1b1eb037b861fbf20491ac17e519e9eaf232b858)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
2935d76bb4 bitbake: fetch2: remove duplicated code in url decode and encode
Use the URI class to decode and encode an URL. Remove duplicate code and
unify the behavior.

(Bitbake rev: a5d569c94700f04b8193c6bccae5af619931b00f)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
2b71696f35 bitbake: fetch2: do not decode user from file URI
A file URI can't contain a user. Do not treat the @ as reserved
character for a file URI.

(Bitbake rev: 11cf4062f48536547a352e24d6e963d91fdd1190)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Bin Lan
33271912bf bitbake: bb/fetch2/__init__.py: remove a DeprecationWarning in uri_replace()
There is the following warning when executing to bitbake linux-yocto:
  bitbake/lib/bb/fetch2/__init__.py:464: DeprecationWarning: 'count' is passed as positional argument

This is because the 4th parameter of re.sub(pattern, repl, string, count=0, flags=0)
is a keyword parameter. We use keyword arguments for parameters that are not positional.

(Bitbake rev: 4bd62639f7023eec9f55ad7909fa9250538de936)

Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-14 09:51:45 +00:00
Stefan Herbrechtsmeier
f00e8e1233 bitbake: utils: add Go mod h1 checksum support
Add support for the Go mod h1 hash. The hash is based on the Go dirhash
package. The package defines hashes over directory trees and is uses for
Go mod files and zip archives.

(Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-11 18:35:01 +00:00
Stefan Herbrechtsmeier
b242f0a007 bitbake: fetch2: read checksum from SRC_URI flag for npm
(Bitbake rev: 8d3232152e545be958f8f3b069e7477c6f310583)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Stefan Herbrechtsmeier
d7c6afc9a9 bitbake: fetch2: do not prefix embedded checksums
The fetcher support entries with an embedded checksum like 'sha256sum'
in the SRC_URI. It adds the parameter 'name' as prefix to the checksums
if the parameter is set. This behavior is unexpected and leads to hacks
in fetchers. Fallback to the checksum without the useless prefix and
set the parameter 'name' in the gomod fetcher unconditional.

(Bitbake rev: 7a86c5a20ea2586f1ae240613644e065e7b21683)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Stefan Herbrechtsmeier
eb0c87fa4c bitbake: fetch2: npmsw: remove old lockfile format support
Remove support for the old lockfile format. The old lockfile format is
required by npm 6 / Node.js 14 which is out of maintenance [2].

[1] https://docs.npmjs.com/cli/v6/configuring-npm/package-lock-json
[2] https://nodejs.org/en/about/previous-releases

(Bitbake rev: 7824e19483d9b60a259d6e3a4c7068fade94f2bf)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Ross Burton
c03be6b81c bitbake: fetch2/wget: correctly construct the target URL in checkstatus()
ud.path has been unescaped (eg %20 is space) but as we're reconstructing
a URL we should re-escape it. For example, unzip has a SRC_URI
containing "UnZip%206.x%20%28latest%29/UnZip%206.0/unzip60.tar.gz" which
then throws exceptions if the unescaped string " (latest)" is used.

Also, this code uses the extracted ud.host and ud.path variables. These
are unescaped but potentially stale as eg the cargo fetcher subclasses
Wget() and reassigns ud.url on construction.

Simplify the code by reconstructing a URL from ud.url directly instead
of bouncing through intermediate variables that may be wrong or
unescaped.

(Bitbake rev: c9906bb289dcfd9ae41f10bd5399ccc17a4cc437)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:25:28 +00:00
Ross Burton
e8ab6bc411 bitbake: fetch2/wget: handle HTTP 308 Permanent Redirect
urllib2.HTTPRedirectHandler.redirect_request doesn't handle HTTP reponse
code 308 (Permanent Redirect). This was fixed in c379bc5 but can't be
worked around without copying the entire redirect_request() method.

When we can depend on Python 3.13, FixedHTTPRedirectHandler can be
removed.

(Bitbake rev: 365829a2803b954ee6cb0364749551a91d806075)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:25:28 +00:00
Stefan Herbrechtsmeier
02f8ea6865 bitbake: fetch2: npmsw: ignore bundled dependencies in packages
The npm package lock and shrinkwrap file list bundled dependencies which
are supplied together with the parent dependency. The flag depends on
the lock file version. Add the missing `inBundle` flag to packages list
processing.

(Bitbake rev: 042d7121ddc93c18306bf03d26ea774f66379a90)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:18:59 +00:00
Pascal Eberhard
b35dc1a99d bitbake: fetch2: fix unpack of a single zstd file
do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when
uncompressing a single 'file.zst', zstd command line returns the error
'unexpected end of file', and the input file 'file.zst' becomes empty.

The command line is 'zstd --decompress --stdout file.zst > file.zst'.
So the issue is that the input and output filenames are the same.

Fix the output filename when using zstd for a single file.

(Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8)

Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-29 18:26:14 +00:00
Richard Purdie
b4bd930fb2 bitbake: fetch/wget: Increase timeout to 100s from 30s
Testing shows the worst case CDN response time can be up to 100s. The wget fetcher
is used for accessing sstate from the CDN so increase our timeouts there to match
our worst case repsonse times.

(Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Robert Yang
0a45ba64ad bitbake: git: Clean broken symlink
The file may be a symlink when mirror from local disk, and the symlink will be
broken when the linked file is removed, use os.path.islink() to check and
remove the broken symlink.

(Bitbake rev: d0deb5fe6e7a84b07b49ef685b209729d2fdc780)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
dd9d183621 bitbake: git: Clean shallow mirror tarball
Fixed:
BB_GIT_SHALLOW = "1"
BB_GENERATE_SHALLOW_TARBALLS = "1"
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall

The downloads/gitsmshallow_github.com.nemtrif.*.tar.gz won't be cleaned without
this fix.

(Bitbake rev: bab7a8970a0237a9d24217685a595e76a1336c07)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
5bce38fbae bitbake: gitsm: Add clean function
Fixed:
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall

The downloads/git2/github.com.nemtrif.ftest won't be cleaned without this fix.

(Bitbake rev: 79f25fc5c1b8d0e08540f4aa07875309f5325f47)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
8d9c02ccfc bitbake: gitsm: Remove downloads/tmpdir when failed
The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix.

(Bitbake rev: 2ba8d3214759142afc11f0a88d80eb30a8bcde3a)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Robert Yang
04c1ee924b bitbake: gitsm: Add call_process_submodules() to remove duplicated code
There are 14 lines can be removed, and can make it easy to maintain.

(Bitbake rev: 0ea2c1ac079d63349407a69172ff80cd9acc7252)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Alexander Kanavin
74c93ec961 bitbake: fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()
This eliminates the last usage of 'fake mozilla' in bitbake, and
it's then truthful everywhere about presenting itself, or wget
(when that is used).

I understand this will make people nervous so I want to provide
an extended decription.

1. How was this tested?

- bitbake-selftest -k FetchCheckStatusTest
(tests a few hardcoded URIs, all passed)

- bitbake -k -c checkuri world
(runs checkstatus() over all recipes in oe-core, and all passed again -
this hopefully goes a long way to reassure everyone that hosts around
the world and various CDNs typically do not have a problem with user-agent
strings they haven't seen before or bitbake user-agent specifically)

2. What about that removed cloudflare comment?

I digged into git history, and I think it is not fully accurate. First, 'fake
mozilla' agent is used only for checkstatus() - in actual fetching with wget
it is not. And that has not been a problem for anyone.

Second, here's how the comment occured. Usage of 'fake mozilla' was introduced here:
https://git.yoctoproject.org/poky/commit/?h=master&id=ab26fdae9e5ae56bb84196698d3fa4fd568fe903

At that point it did not have to be specifically 'mozilla', the commit message
indicates that any User-Agent would have been ok. Mozilla was simply copied
from upstream version check for convenience.

Later on, the string was updated to a more recent Mozilla:
https://git.yoctoproject.org/poky/commit/?h=master&id=9f123238261a68e37cec634782e9320633cac5d4

The claim in the added comment become something else: that User-Agent *must* a browser,
without evidence or tests. Even though it demonstrably doesn't have to be - wget is ok.

3. What if someone has a server that is ok with wget agent, but not ok with bitbake agent?

Please see point one. It's not impossible but I think it's highly unlikely. I do think
we should rather tell servers the truth, and learn where the actual issues are. Then
we can consider options - whether that would be pretending to be wget, or allowing user-agent
to be configured. We should also add such servers to bitbake-selftest so we know what they
are.

(Bitbake rev: 234f9e810494394527f59fdf22eb86435d046d53)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-22 11:16:32 +01:00
Alexander Kanavin
9eaa1e5024 bitbake: fetch2/wget.py: use default wget user-agent in upstream version checks
This has been set to various mozilla versions from this point onwards without
explanation or specific examples:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6

However, sourceforge has recently banned the current 'fake mozilla' version
we've been supplying, resulting in broken version checks:

alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk
...
--2024-10-12 11:53:23--  https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ...
Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-10-12 11:53:24 ERROR 403: Forbidden.

gptfdisk                  1.0.10          UNKNOWN_BROKEN  Alexander Kanavin <alex.kanavin@gmail.com>

On the other hand, using wget's default user-agent works just fine:

--2024-10-12 12:16:06--  https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ...
Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

(Bitbake rev: 138117dfc05248dcb48707c838c8c1cc09f12d07)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-22 11:16:32 +01:00
Oliver Kästner
aa546ea0c3 bitbake: fetch2: add GitHub Codespaces support
GitHub Codespaces allow users to use GitHub-hosted pre-configured Docker
containers to work on GitHub repositories in VS Code within the browser
or using a locally running instance of VS Code [1].

Users can define access permissions for Codespaces containers, allowing
them to clone or work with other private repositories [2].
To do this, a git credential helper is injected into the container at
/.codespaces/bin/gitcredential_github.sh, which contains the following
lines:

    #!/bin/sh
    echo protocol=https
    echo host=github.com
    echo path=
    echo username=PersonalAccessToken
    echo password=$GITHUB_TOKEN

The `GITHUB_TOKEN` environment variable is automatically set when the
Codespaces container is created. If authorized by the user, it has the
specified permissions. It is thus required to be able to fetch private
git repos from a Codespace container, so add it to `FETCH_EXPORT_VARS`
to make it available for invocations of git.

[1] https://docs.github.com/en/codespaces/overview
[2] https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces

(Bitbake rev: 71ca83bdd541f70737a3a85f05e40c222611af2f)

Signed-off-by: Oliver Kästner <git@oliver-kaestner.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-14 21:44:38 +01:00
Richard Purdie
76d24b00ff bitbake: checksum/fetch2: Switch from persist_data to a standard cache file
The sqlite connection handling is causing problems with python 3.13. The
connection can be closed at gc time which causing warnings and those
can appear at 'random' points and break output, causing weird failures
in different tinfoil tools and other tests.

Using sqlite as an IPC was never a great idea so drop that usage entirely
and just use the standard cache mechanism we already have for other
situations.

(Bitbake rev: fdc55bb649cb77456d0ac48a9600ef289a52af18)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 13:04:30 +01:00
Richard Purdie
8cd4ab4771 bitbake: fetch2: Avoid deprecation warning
>From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated.
In future Python versions they will be keyword-only parameters."

Avoid the warning.

(Bitbake rev: 016d872dc3c36792af5c84da1d35a5c86996d795)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 13:04:30 +01:00
Richard Purdie
312488228a bitbake: fetch2/git: Use quote from shlex, not pipes
The pipes module is removed in python 3.13. It was already using the
quote function from shlex so use that directly instead.

The module already imports shlex too so it is an easy substitution.

(Bitbake rev: 9ef4f2a30127455562e38e869793a723eed6c07c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-01 12:42:24 +01:00
Ross Burton
d3a98ba771 bitbake: fetch2: don't try to preserve all attributes when unpacking files
When copying files as part of the unpack we currently use cp -p, which
is a shortcut for --preserve=mode,ownership,timestamps.

We do want to preserve timestamps, because some fetchers set these
explicitly.

We don't care about ownership. If the files are owned by us then they
ill remain owned by us, and if they're not then the attempt to change
ownership will be silently ignored.  In a shared DL_DIR where files have
group ownership this group access isn't relevant in the single-user build
tree.

We do want to preserve executable bits in the mode, but cp always does
this.  The difference between --preserve=mode and no --preserve is that
the mode isn't preserved exactly (no sticky bits, no suid, umask is
applied) but this also isn't a relevant difference in a build tree.

Also expand the arguments to be clearer about what options are being
passed.

The impetus for this is that coreutils 9.4 includes a change in gnulib[1]
and will now try to preserve permission-based xattrs if asked to preserve
the mode.  This can result in cp failing when copying a file from a
NFSv4 server with ACLs stored in xattrs to a non-NFS directory where
those xattrs cannot be written:

cp: preserving permissions for ‘./jquery-3.7.1.js’: Operation not supported

The error comes from the kernel refusing to write a system.nfs4_acl
xattr to a file on ext4.

This situation doesn't appear on all systems with coreutils 9.4, at the
time of writing it fails on Ubuntu 24.04 onwards but not Fedora 40. This
is because /etc/xattr.conf is used to determine which xattrs describe
permissions, and Fedora 40 has removed the NFSv4 attributes[2].

Also, use long-form options to make the cp command clearer.

[1] https://github.com/coreutils/gnulib/commit/eb6a8a4dfb
[2] https://src.fedoraproject.org/rpms/attr/blob/rawhide/f/0003-attr-2.4.48-xattr-conf-nfs4-acls.patch

[ YOCTO #15596 ]

(Bitbake rev: 2f35dac0c821ab231459922ed98e1b2cc599ca9a)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-30 17:01:25 +01:00
Peter Kjellerstedt
f290651279 bitbake: fetch2/gomod: Support URIs with only a hostname
When calculating the module name for a gomod URI with only a hostname,
e.g.:

  gomod://go.opencensus.io;version=v0.24.0;sha256sum=203a767d7f8e7c1ebe5588220ad168d1e15b14ae70a636de7ca9a4a88a7e0d0c

the non-existing path would actually be treated as "/", which resulted
in a trailing slash being added to the module name preventing the unpack
method from correctly locating the go.mod file.

(Bitbake rev: f0e02e1de4d649e647e4ab61341042dd38d0eeb0)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-17 12:17:28 +01:00
Martin Jansa
1e6e8cb896 bitbake: Revert "fetch2/gitsm: use configparser to parse .gitmodules"
This reverts commit a4f42e396e2942fde94b8b4944487c1c45f7a295.

Unfortunately configparser is a bit more strict and fails to parse e.g.:
e104dd7f27/.gitmodules

[submodule "components/bt/controller/lib_esp32"]
	path = components/bt/controller/lib_esp32
        url = ../../espressif/esp32-bt-lib.git

The url is using 8 spaces while path 1 tab causing:
Exception: configparser.NoOptionError: No option 'url' in section: 'submodule "components/bt/controller/lib_esp32"'

It was fixed in:
62ca8e2fb4 (diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584)

but mcuboot is using a bit older esp-idf revision in:
https://github.com/ATmobica/mcuboot/blame/main/.gitmodules

and mcuboot is then used as submodule in:
https://github.com/project-chip/connectedhomeip/blob/master/.gitmodules

so it might take a while for the fix to be propagated everywhere.

Not sure how common these issues are, but configparser parses
"url = ../../espressif/esp32-bt-lib.git" as 2nd line of "path"
value (because it's indented differently) while git submodule
and old gitsm implementation parses it as separate path and
url keys.

(Bitbake rev: d9eb2650256292a524a0c0bb1c0562d87bedcbc4)

Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-16 13:52:27 +01:00
Ross Burton
d016d18a9f bitbake: fetch2/gitsm: use configparser to parse .gitmodules
.gitmodules is basically ini-style, so use configparser instead of manually
parsing by hand.

(Bitbake rev: a4f42e396e2942fde94b8b4944487c1c45f7a295)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-12 16:17:20 +01:00
Christian Lindeberg
87c29b5a94 bitbake: fetch2: Add gomodgit fetcher
Add a go module fetcher for downloading module dependencies to the
module cache directly from a git repository. The fetcher can be used
with the go-mod class in OE-Core.

A module dependency can be specified with:

  SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;srcrev=..."

(Bitbake rev: 29ff38ccf0d5389a5bee81e252a78548361a9d7c)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-10 13:05:00 +01:00
Christian Lindeberg
dd7631426c bitbake: fetch2: Add gomod fetcher
Add a go module fetcher for downloading module dependencies to the
module cache from a module proxy. The fetcher can be used with the
go-mod class in OE-Core.

A module dependency can be specified with:

  SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."

(Bitbake rev: 5ff4694bf305e266ebf0abab5d9745c6b6d07d67)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-10 13:05:00 +01:00
Leonard Göhrs
ee6bf285d7 bitbake: fetch2/npm: allow the '@' character in package names
The '@types/ramda' [1] npm package has recently gained a dependency on
the 'types-ramda' [2] npm package. Both have the same version number.

The name mangling results in the tarballs of both packages sharing the same
name, but different contents.

Fix that by accepting '@' as valid character in the package name,
resulting in one package named @types-ramda and one called types-ramda.

[1]: https://www.npmjs.com/package/@types/ramda
[2]: https://www.npmjs.com/package/types-ramda

(Bitbake rev: 7c9573cb6ea2081bc585eb65267f3124fd4d7e43)

Signed-off-by: Leonard Göhrs <l.goehrs@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-08-25 15:37:02 +01:00