Fixes [YOCTO 15917]
When fetching a git repository the .gitattributes file is scanned, checking if LFS
support is required for the repository. This scan checks if the word "lfs" is present
in the file, however the used regex doesn't account for comments, which makes some
repositories[1] be to misidentified as requiring LFS support (which fails fetching, in case
lfs support isn't installed on the build host).
To avoid it, change the used regex to ignore lines starting with "#".
[1]: e.g. https://github.com/MicrochipTech/cryptoauthlib
(Bitbake rev: 7917a758fc328747116c7899e689171bd0efc883)
Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When recreating the uri in wget's checkstatus method, we only use the
scheme, netloc and path. This completely strips the query parameters
from the final URI and potentially breaks the checking functionality
from certain fetchers.
This is the case for the Azure storage fetcher, as it requires a SAS
token that is formatted as a series of query parameters. The error
manifests itself when using a private storage account as a PREMIRROR or
SSTATE_MIRROR (since regular SRC_URI won't run the checkstatus).
This problem is present in scarthgap, but wasn't in kirkstone.
CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
(Bitbake rev: 096301250455e2a83bdd818a56317c62436c9981)
Signed-off-by: Philippe-Alexandre Mathieu <pamathieu@poum.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It makes sense to allow the shallow clone mirror tarball name to include the tag
name so that tags can be added to existing urls and the tarballs will be maintained
correctly. The code already allows this to be done easily just by moving the tag
handling code.
(Bitbake rev: 68fce3be14e4dd801661f4ef302d229fb16a04b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, tags are fetched as just a name, which works but means they're not
seen as tags by git commands like git describe. Instead, fetch them as refs/tags/XXX
which means such commands then work correctly.
(Bitbake rev: c1f30ad61f5e55beb377451887bbbc5cb569f2e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will allow setting the variable in oe-core such that it matches
S, which in turn will allow dropping S from almost all git:// fetching recipes,
and not having to set it in newly written ones.
(Bitbake rev: 378db0fdd95f8704fccd852452555bfddcad384d)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AZ_SAS token should be prefixed with a question mark. Add a sanity check for
this and fix the documentation.
[YOCTO #15882]
(Bitbake rev: 22011765202514600314732b97f1bb938e21f585)
Signed-off-by: Robbin Van Damme <robbinvandamme@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Since `ud.path` contains in that case the `PREMIRRORS` prefix path,
this change ensures that a correct symlink is set up.
(Bitbake rev: 37ed18e45aa17406162efc5ee3ddb2d6b33d07b9)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some .deb files contain uncompressed data tarballs which do not have an
additional file extension after `.tar`. Add support for such cases while
preserving the existing behaviour.
(Bitbake rev: e3834deb427ceb1d1ae9a96c1e0dec990d7f3805)
Signed-off-by: Isaac True <isaac.true@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a tag contains slash, e.g., debian/5.22, then shallow clone
fails because it's using a wrong ref.
To reproduce the issue, add the following lines in local.conf:
BB_GIT_SHALLOW = "1"
BB_GENERATE_SHALLOW_TARBALLS = "1"
And then run 'bitbake debianutils -c fetch'.
What the original os.path.basename(ref) wanted to do is to remove
the strings such as refs/heads/. So we do it explitly to fix this
issue.
Fixes: [YOCTO #15862]
(Bitbake rev: c6d6999f1ed01e7445b8f177a888038edacf555c)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Skipping smudging prevents the LFS objects from replacing their
placeholder files when `git submodule update` actually checks out the
target revision in the submodule. Smudging cannot happen earlier as the
clone stored in `.git/modules` is bare.
This should be fine as long as all LFS objects are available in the
download cache (which they are after the other fixes are applied).
(Bitbake rev: d270e33a07c50bb9c08861cf9a6dc51e1fd2d874)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code would pass `True` or `False` to the git fetcher. As
the fetcher expects `lfs` to be set to `1` this always lead to LFS
fetching being disabled.
(Bitbake rev: 5e487a5a096400271ed1e29b0df72903f2304e49)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its not clear which exact git-lfs versions failed to work with bare
repositories, however git-lfs 2.13.2 which is shipped by Debian 10 (i.e.
the oldest supported distribution by scarthgap) shows no issue when
fetching into a bare repository. Switch to git-lfs fetch which in turn
eliminates issues seen when using the gitsm fetcher with submodules
utilizing lfs.
In these scenarios, fetching of LFS objects did not actually happen as
the gitsm fetcher parametrizes the to be fetched repositories with
`bareclone=1` which in turn means that the target revision was never
checked out (and therefore no LFS objects were fetched).
(Bitbake rev: 501d3317df5780a5b0e3c2562b26ed276c1dbe90)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its unclear why this function does not operate on the desired source
revision to begin with (which really should be the decider on whether a
particular source revision uses LFS or not). Simplify the decision logic
by always checking the `.gitattributes` file of the target revision.
(Bitbake rev: b3faa0ce5c0a6945f26b5b303a7f38c00d132397)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So far, existence of `git-lfs` was only checked during unpacking. As the
binary is also used in earlier steps also check for its existence there.
Additionally, factor out the LFS existence check into a dedicated
function and call it wherever git-lfs is used for the first time.
(Bitbake rev: 5818367db9b261b7e07c347d38044e6cba8f9727)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no need to pass `name` when it is no longer used.
(Bitbake rev: b132d35dee643e270e3e6dd536dcc90334a0111c)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Correct typo Fash -> Fast.
With it also delete some trailing spaces which are unnecessary and
editors are trying to remove when changing anything in the file.
(Bitbake rev: b96040a0a066344527f0bc6ca4d09cda6b9d2f11)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before fast shallow tarballs the shallow tarball with lfs worked only
if the original .../downloads/git2/... directory existed. Once it was
not there, do_unpack on shallow tarball without lfs files failed due to
disabled network.
It was fixed for fast shallow tarballs and this commit fixes the other
scenarios, too.
This can happen when such shallow tarball is put to mirror or if
someone does cleanup to reclaim disk space.
(Bitbake rev: 4e6fb31a88448cdacf4e9f84cf0a8a8035f84d63)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When invoking the original git fetcher after downloading a mirror
tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch
commands targeting the upstream source being executed silently. Ensure
setting BB_NO_NETWORK before invoking the original fetcher. While this
was only observed for git, setting this in general for all fetcher
types makes sense at this location.
(Bitbake rev: 1b1321f2b60c0a66159e3f20c6befcb0b3ccc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SRCREV points at an annotated tag, the comparision code can fail
as the resolved tag might not be the same sha.
Handle this by also resolving the SRCREV. We only need to do this if
they don't match in the first place for a minor performance win.
Also add a test for this.
(Bitbake rev: 136c06e251de68ed64355ec6b47a522ff3a372e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's best practise to use long-form arguments in scripts where the
conciseness of short arguments is less useful than in an interactive
terminal.
(Bitbake rev: 54039bc9d169871af6d36578df4c21bff296f6de)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the target file exists, we pass --continue so that wget will try to
resume the download instead of starting from the beginning. However if
the file doesn't exist --continue has no effect, so there's no need to
conditionalise the use of the argument.
(Bitbake rev: a5ee50d4d2b2e6c00abe1551382afd9799345dbf)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
latest_versionstring() currently looks at just the end of the URI when
guessing what the filename to look for is, but this doesn't work if the
URL filename is not simple.
For example, miniupnpd has a SRC_URI of:
http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz;downloadfilename=${BP}.tar.gz
The filename component of this is "download.php", which causes the
heuristics in latest_versionstring() to exit early.
Instead, if the downloadfilename is set then use that, as it's often the
actual filename that we're after.
(Bitbake rev: 2d5f135e997d13fabda0ad266fd5c928ee33f487)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Build the 'downloadfilename' parameter by replacing path separators in
the module path like the git fetcher builds the mirror tar ball name.
Copy the downloaded file in the fetcher's unpack method like the crate
fetcher instead of calling the base fetcher's unpack method.
(Bitbake rev: 7762cea087597019460d66b04268757bd46befdf)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).
We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.
This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.
Some test cases are removed since they are no longer needed.
(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we want to validate a tag when using shallow clones, we need to ensure
the tag is imported to the clone. Adding to extra_refs allows this to happen
ensuring tag validation in shallow clones works.
(Bitbake rev: 03945475886b8d31c7eccc80fe594ab5306bcf5d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.
Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.
This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.
The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.
(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes a bug where escapes in the url path would not be properly
restored for the git commands in the git fetcher. For example, a
space which is encoded as '%20' was not properly encoded before the
clone command.
e.g.
SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https"
resulted in
git clone 'https://git.openembedded.org/bitbake example/bitbake'
instead of
git clone 'https://git.openembedded.org/bitbake%20example/bitbake'
(Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680)
Signed-off-by: Patrik Nordvall <patrik.nordvall95@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f
which changes decodeurl() to use the URI class to parse the URL instead
of doing it itself. While reusing code is generally a good idea, using
urllib.parse.urlparse() (which the URI class does) to parse the regular
expression "URLs" that are used in PREMIRRORS and MIRRORS does not work.
A regular expression URL containing https?://... would be silently
ignored, while a URL using a negative lookahead such as
git://(?!internal\.git\.server).*/.* would result in a cryptic error:
Exception: re.error: missing ), unterminated subpattern at position 0
The problem is that urllib.parse.urlparse() treats the ? as the start of
URL parameters and thus stops parsing whatever part of the URL it was
parsing. Restore the old function and use it in the PREMIRRORS and
MIRRORS code.
(Bitbake rev: f8a7712754e6d0199a0d227fca288307b935368d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When `ud.shallow == 1`:
- Prefer an initial shallow clone over an initial full bare clone,
while still utilizing any already existing full bare clones.
- If the Git error "Server does not allow request for unadvertised object"
occurs, the initial full bare clone is fetched automatically.
This may happen if the Git server does not allow the request
or if the Git client has issues with this functionality,
especially with the Git client from Ubuntu 20.04.
This improves:
- Resolve timeout issues during initial clones on slow internet connections
by reducing the amount of data transferred.
- Eliminate the need to use an HTTPS tarball `SRC_URI`
to reduce data transfer.
- Allow SSH-based authentication (e.g. cert and agent-based) when
using non-public repos, so additional HTTPS tokens may not be required.
(Bitbake rev: 457288b2fda86fd00cdcaefac616129b0029e1f9)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using BB_FETCH_PREMIRRORONLY we write to the datastore to disable the network.
This change needs to be undo when handling later urls, so operate on a copy of the
datastore to allow this.
Reported by Julian Haller <julian.haller@philips.com>
(Bitbake rev: 67a5ede8ae92ed7dcad29fd0dcfd62c6640b10b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URI path is already unquoted. Remove the unnecessary unquote
function calls for URI path values.
(Bitbake rev: 3de12bbc28b5a4189b849720735cf3e268d3941d)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the path variable from the fetch data instead of decoding the path
manually from the plain unexpanded url.
(Bitbake rev: ad3a29fa6ea53741d4e1786de35f8e7fc4292e7a)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetch data class already expands the type, host, path, user, pswd
and parm variables. The fetcher classes already expand the localfile
variable. The getVar function expands the returned string per default.
Remove unnecessary expand function calls to simplify the code.
(Bitbake rev: 1b1eb037b861fbf20491ac17e519e9eaf232b858)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the URI class to decode and encode an URL. Remove duplicate code and
unify the behavior.
(Bitbake rev: a5d569c94700f04b8193c6bccae5af619931b00f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A file URI can't contain a user. Do not treat the @ as reserved
character for a file URI.
(Bitbake rev: 11cf4062f48536547a352e24d6e963d91fdd1190)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the following warning when executing to bitbake linux-yocto:
bitbake/lib/bb/fetch2/__init__.py:464: DeprecationWarning: 'count' is passed as positional argument
This is because the 4th parameter of re.sub(pattern, repl, string, count=0, flags=0)
is a keyword parameter. We use keyword arguments for parameters that are not positional.
(Bitbake rev: 4bd62639f7023eec9f55ad7909fa9250538de936)
Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Go mod h1 hash. The hash is based on the Go dirhash
package. The package defines hashes over directory trees and is uses for
Go mod files and zip archives.
(Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher support entries with an embedded checksum like 'sha256sum'
in the SRC_URI. It adds the parameter 'name' as prefix to the checksums
if the parameter is set. This behavior is unexpected and leads to hacks
in fetchers. Fallback to the checksum without the useless prefix and
set the parameter 'name' in the gomod fetcher unconditional.
(Bitbake rev: 7a86c5a20ea2586f1ae240613644e065e7b21683)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ud.path has been unescaped (eg %20 is space) but as we're reconstructing
a URL we should re-escape it. For example, unzip has a SRC_URI
containing "UnZip%206.x%20%28latest%29/UnZip%206.0/unzip60.tar.gz" which
then throws exceptions if the unescaped string " (latest)" is used.
Also, this code uses the extracted ud.host and ud.path variables. These
are unescaped but potentially stale as eg the cargo fetcher subclasses
Wget() and reassigns ud.url on construction.
Simplify the code by reconstructing a URL from ud.url directly instead
of bouncing through intermediate variables that may be wrong or
unescaped.
(Bitbake rev: c9906bb289dcfd9ae41f10bd5399ccc17a4cc437)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
urllib2.HTTPRedirectHandler.redirect_request doesn't handle HTTP reponse
code 308 (Permanent Redirect). This was fixed in c379bc5 but can't be
worked around without copying the entire redirect_request() method.
When we can depend on Python 3.13, FixedHTTPRedirectHandler can be
removed.
(Bitbake rev: 365829a2803b954ee6cb0364749551a91d806075)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The npm package lock and shrinkwrap file list bundled dependencies which
are supplied together with the parent dependency. The flag depends on
the lock file version. Add the missing `inBundle` flag to packages list
processing.
(Bitbake rev: 042d7121ddc93c18306bf03d26ea774f66379a90)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when
uncompressing a single 'file.zst', zstd command line returns the error
'unexpected end of file', and the input file 'file.zst' becomes empty.
The command line is 'zstd --decompress --stdout file.zst > file.zst'.
So the issue is that the input and output filenames are the same.
Fix the output filename when using zstd for a single file.
(Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8)
Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
