Commit Graph

742 Commits

Author SHA1 Message Date
Robert Yang
c6939e8b7f bitbake: fetch/git: Fix local clone url to make it work with repo
The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:

6f054f9fb3

But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:

* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink

(Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-11 11:31:52 +01:00
Martin Jansa
6505459809 bitbake: fetch2/git: show SRCREV and git repo in error message about fixed SRCREV
* in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to
  be one character shorter and because fetcher uses:
    if not ud.revisions[name] or len(ud.revisions[name]) != 40  or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
  to decide which SRCREV values are fixed SRCREVs this one was
  considered as tag or branch name, because it was only 39 chars long

  The original error message wasn't very helpful as it doesn't show
  which repo or which SRCREV was considered missing:

  do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)

  with SRCPV included in PV as error recomments it's a bit better:

  bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16
  The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T

  with this change the first error will read:

  do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)

(Bitbake rev: 9bbdedc0ba7ca819b898e2a29a151d6a2014ca11)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23 22:23:07 +00:00
Marek Vasut
c0b0b9a862 bitbake: fetch2/git: Clarify the meaning of namespace
Namespace in this context means a branch, a tag, etc., clarify
it in the description. Also, fix a typo "a any", replace with
plain "any".

This patch is based of feedback on new applied patch
d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata")

(Bitbake rev: 86f2fa5261da959cda706c794a0047e5e89d4d6b)

Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 10:34:05 +00:00
Marek Vasut
f97a46b07f bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository metadata
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every
single object in the remote repository. This works poorly with gitlab
and github, which use the remote git repository to track its metadata
like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/*
and refs/keep-around/* and they all contain massive amount of data that
are useless for the bitbake build purposes. The amount of useless data
can in fact be so massive (e.g. with FDO mesa.git repository) that some
proxies may outright terminate the 'git fetch' connection, and make it
appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher such
that it only fetches refs/heads/* and refs/tags/* . Avoid using negative
refspecs as those are only available in new git versions.

Per feedback on the ML, Gerrit may push commits outsides of branches or
tags during CI runs, which currently works with the 'nobranch=1' fetcher
parameter. To retain this functionality, keep fetching everything in case
the 'nobranch=1' is present. This still avoids fetching massive amount of
data in the common case, since 'nobranch=1' is rare. Update 'nobranch'
documentation.

Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
(Bitbake rev: c17fc1468ab84663b919e2809606b1b8ea2bebd9)

Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 10:34:05 +00:00
Pavel Zhukov
ca83262bdd bitbake: gitsm: Fix regression in gitsm submodule path parsing
Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression
in submodules path parsing. As the result gitsm fetcher fails on each
submodule which name begins from the name of the parent repo which is
totally valid usecase [Yocto #14045] [1]
Fix the code to error out only if submodule's name is equal to parent
name but not if it's part of it.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4

(Bitbake rev: 7e268c107bb0240d583d2c34e24a71e373382509)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-09 13:02:50 +00:00
Pavel Zhukov
189a6d452e bitbake: gitsm: Error out if submodule refers to parent repo
If submodule refers to specific revision of the parent repository it
causes deadlock in bitbake locking mechanism (lock is acquired to fetch
the parent and cannot be released before all submodules are fetched).
raise FetchError in such situation to prevent deadlocking.

[Yocto 14045]

(Bitbake rev: 987712c4c8fefd86a1f5116c11ee86e296e852ee)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-11 21:57:28 +01:00
Neil Horman
4b9ac6d5aa bitbake: Fix npm to use https rather than http
Hit this error while building nlf-native recently:
{
  "error": {
    "summary": "URI malformed",
    "detail": ""
  }
}

Some poking about led me to discover that:
1) The npm.py tool replaces npm:// with http://, not https://
2) Some versions of the npm tool don't handle 301 redirects properly,
   choosing to display the above error instead when using the default
   nodejs registry

It would be good to go fix npm to handle the redirect properly, but it
seems like it would also be good to assume secure http when contacting a
registry, hence, this patch

(Bitbake rev: bb5c43220f5f1c3d82334c65aff1ce13008db8d9)

Signed-off-by: Neil Horman <nhorman@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-11 21:57:28 +01:00
Richard Purdie
0ac71c5480 bitbake: bitbake: Add copyright headers where missing
Where copyright headers were not present, add them to make things
clear.

(Bitbake rev: e591325b2bd901c381003deb96a7b32a7148e93e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-11 21:57:28 +01:00
Richard Purdie
9c26cd5d70 bitbake: fetch2: Ensure directory exists before creating symlink
If the mirrors code is trying to create a symlink and the
parent directory doesn't exist, as might be the case for sstate
mirrors where the fetch is into a subdir, it can silently fail.

Ensure the directory exists in this case to avoid issues.

(Bitbake rev: ff3afb1c1bb236c4a52c62a74f2917071e0af55b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eff16e474ee7dc49ae433420a4c8d15d3314a618)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-16 17:53:22 +01:00
Gennaro Iorio
43e98fb455 bitbake: fetch2: gitsm: fix incorrect handling of git submodule relative urls
As specified by git submodule manual relative urls can start either
with '..' or './', second case was incorrectly managed leading to an
interpretation of urls starting with './' as absoulte urls.

(Bitbake rev: d828cd2a16ddf4f084e61ffe44471483e132653a)

Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-16 17:53:22 +01:00
Gunjan Gupta
7c0dd56d28 bitbake: fetch2/osc: Small fixes for osc fetcher
The current fetcher seemed to have some issues that made it difficult when
trying to use the same. This patch fixes the following

* Make consistent use of the path that needs to be used as oscdir
* The path mentioned in os.access in download function was not same as
  ud.moddir which would result into invoking of fetch command instead of
  update command even if directory already existed
* Before creating oscrc, make sure oscdir exists and create it if it does
  not exist
* Updated the configuration to use apiurl and added a new parameter to
  control whether http or https needs to be used to connect to apiurl

(Bitbake rev: 8ac6e09447d884e658c556388d6014279c50f202)

Signed-off-by: Gunjan Gupta <viraniac@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3ec78686f3c0ea2304097b86a965f9be4b0cb879)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-25 18:00:44 +01:00
Richard Purdie
0d81d82645 bitbake: fetch/git : Use cat as pager
We don't have less in HOSTTOOLS in OE and this can confuse git. Force the
pager to cat to be consistent and minimal everywhere.

(Bitbake rev: 59c16ae6c55c607c56efd2287537a1b97ba2bf52)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d3d406e8552fdd865dc58b419a84411736475ad2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-12 13:41:42 +01:00
Robert Yang
b34c299467 bitbake: fetch2/ssh.py: decode path back for ssh
The path has been encoded by urllib.parse.quote(), so decode it back for ssh.

Fixed when fetch from PREMIRRORS via ssh:
$ bitbake bonnie++ libsigc++-2.0 -cfetch

scp: /path/to/downloads/libsigc%2B%2B-2.10.7.tar.xz: No such file or directory

(Bitbake rev: 3969786a787eea34e096b932d52cd02978aacb8e)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c1c8fc678eb4783cea3974328a5fa8d1b79f1266)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-12 13:41:42 +01:00
Richard Purdie
b6df0485bc bitbake: fetch2/osc: Add missing parameter
This probably means the osc fetcher isn't being used but fix the missing
parameter.

(Bitbake rev: 73fbb743a2def2037d4053605e77e09d2d8a9fd0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a23c201cb6efc5c0abf763c26f905442f0eebb68)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-12 13:41:42 +01:00
Richard Purdie
08ee5cf2ea bitbake: fetch2/crate: Drop unused import
(Bitbake rev: 43bb92a2728832eeb1a207809e21404c86ff1710)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 879f17ecd5ba09e217cef74f6a51339b145e8ef5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-12 13:41:42 +01:00
Ricardo Salveti
af5cb9d6fe bitbake: fetch2/crate: fix logger.debug line
logger.debug was giving an integer value (2) as event message, causing
knotty to crash when running with debug enabled.

bitbake/lib/bb/ui/knotty.py", line 685, in main
  event.msg = taskinfo['title'] + ': ' + event.msg
TypeError: can only concatenate str (not "int") to str

Same issue also happens in the original code that was taken from
oe-core (openembedded-core/meta/lib/crate.py honister) / meta-rust.

(Bitbake rev: c212b0f3b542efa19f15782421196b7f4b64b0b9)

Signed-off-by: Ricardo Salveti <ricardo@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-15 17:30:34 +01:00
Pavel Zhukov
6ec07ea632 bitbake: fetch2: Add GIT_SSH_COMMAND to the list of exports
GIT_SSH_COMMAND is more convinient to use if arguments have to be passed
and the user doesn't want to create a wrapper script around ssh.

(Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-14 09:48:33 +01:00
Olaf Mandel
d1f254d56f bitbake: fetch2/git: canonicalize ids in generated tarballs
Change the owner information in the mirror tarballs generated using
BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit
0178ab83, which used the original pokybuild:user information, but failed
to clean up the numerical user and group ids. Now set the more canonical
values of oe:oe and 0:0.

(Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
CC: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-14 09:48:33 +01:00
Olaf Mandel
5cbdd2b483 bitbake: fetch2/git: stop generated tarballs from leaking info
When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs
of git repositories, they leaked local information: username, group and
time of the last fetch. Remove all these by setting fixed information:

 * uname = pokybuild
 * gname = users
 * mtime = committer time of newest commit in repo

The username and group value were taken from the archives available on
the downloads.yoctoproject.org mirror. The modification time is chosen
so it still retains some relationship to the contents of the archive.

(Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-26 09:27:43 +00:00
Daniel Wagenknecht
fe0d6dbef5 bitbake: fetch2: ssh: support checkstatus
This implements support for sstate mirrors using ssh as transport
protocol.

(Bitbake rev: 0a3b5b3de7bcb1c5c3748cba42d394cc484e966b)

Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-04 17:24:53 +00:00
Daniel Wagenknecht
5937b2e073 bitbake: fetch2: ssh: fix path handling
Fix absolute paths and paths containing the ":" character. Both is
necessary for supporting sstate mirrors via ssh (not implemented yet).

(Bitbake rev: df5505a1ba15524c3a185360d687854300aef342)

Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-04 17:24:53 +00:00
Daniel Wagenknecht
30c21aaf3e bitbake: fetch2: ssh: username and password are optional
Support URLs like ssh://HOST/PATH. They were previously not recognized
due to a missing @ sign.

(Bitbake rev: a2aa18bd27dac8902e52b466cb7118f71367d3dc)

Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-04 17:24:53 +00:00
Scott Murray
09a62d96c7 bitbake: lib: Replace remaining "blacklist"/"whitelist" usage
In line with the inclusive language migration defined at:

https://wiki.yoctoproject.org/wiki/Inclusive_language

replace the remaining non-backwards-compatibility related usage
of "blacklist"/"whitelist" with "allowed"/"disallowed" in a few
places as appropriate.

(Bitbake rev: f579fb8c23d2919d25641fa4234f8a1e9c06a922)

Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-21 23:37:26 +00:00
Richard Purdie
8e71d0e21b bitbake: fetch2: Abstract fetcher environment to a function
The changing of the environment inside the wget fetcher can race if
threading is used, such as with sstate in OE-Core. Abstract the function
so the environment can be correct before the function is called, removing
the race since the enviroment is then no longer changed.

(Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:25 +00:00
Zygmunt Krynicki
2b427b9224 bitbake: wget: Fix grammar "can happen"
(Bitbake rev: 52630eefb5174e4ca357ac57085093a7f5767bd8)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:25 +00:00
Zygmunt Krynicki
e03ac523e4 bitbake: fetch2/git: Remove duplicate "the"
(Bitbake rev: 02e29e90eac56e46635e9ae44e5c5da684b70ed2)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:24 +00:00
Zygmunt Krynicki
07fa614e8f bitbake: fetch2: Fix typo "conform" and "processed"
(Bitbake rev: e1c13a820f9e3154699ec32d84cce502464e4f6b)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:24 +00:00
Zygmunt Krynicki
59fdcddf05 bitbake: fetch2/wget: move loop-invariant load of BB_ORIGENV
BB_ORIGENV is used as a fallback environment block. It is repeatedly
accessed inside the loop. Since it is a loop invariant, move it out of
the loop.

(Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:24 +00:00
Richard Purdie
ebfa1700f4 bitbake: fetch2/cooker: Fix source revision handling with floating upstreams
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:

* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results

To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).

Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.

Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.

(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-17 17:51:03 +00:00
Peter Kjellerstedt
bef3fe4969 bitbake: fetch2: npm: Put all downloaded files in the npm2 directory
Previously npm files that specify downloadfilename= in the SRC_URI
would be downloaded to the root of ${DL_DIR} rather than in the
${DL_DIR}/npm2 directory where all other npm files are downloaded.
This should make it simpler when setting up and configuring a
premirror with the downloaded npm packages.

(Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
88993ae030 bitbake: fetch2: Correct handling of replacing the basename in URIs
The solution implementated in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror) missed two corner cases. The
first is if the basename of the original URI also appears somewhere
else in the replacement URI, in which case it would also be replaced.
The second is if the basename of the original URI partially matches
the basename of the replacement URI, in which case the inital part of
the basename from the replacement URI would be left behind.

The second case caused test_npm_premirrors_with_specified_filename to
fail.

The solution is to prefix the basename with a slash when matching to
avoid partial matches, and only replace the basename at the end of the
URI.

This also adds two test cases that test for these problems. Before
this they would give the following errors:

  - ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  + ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  ?                  +++++       ++++++++++

  - ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  + ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  ?                  +++++

(Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Richard Purdie
12c76723fb bitbake: fetch2: Add crate fetcher
This imports the crate fetcher from OE-Core to resolve various module issues
and adds some very very basic tests of that new fetcher.

(Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-19 23:03:43 +00:00
Stefan Herbrechtsmeier
31bc9cab25 bitbake: fetch: npm: Use temporary file for empty user config
Always use a temporary file for the user config 'NPM_CONFIG_USERCONFIG'
because npm otherwise failed if configs and npmrc aren't set:

double-loading config "/dev/null" as "global", previously loaded as "user"

(Bitbake rev: 9f272ad7f76c1559e745e9af686d0a529f917659)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-14 22:47:04 +00:00
Stefan Herbrechtsmeier
bb42e4b356 bitbake: fetch: npm: Quote destdir in run chmod command
Quote destdir in run chmod command to support special characters in
package name and to avoid syntax error for packages like
'@(._.)/execute'.

(Bitbake rev: a701dfce3f0e74b4d7c687eeda83fe9c8e7240b1)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-14 22:47:04 +00:00
Alexander Kanavin
9bf81a3d09 bitbake: fetch2/wget: do not hardcode tarball compressors in version check
(Bitbake rev: ba3aa8591327d43935f000c6884637997438ecb2)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-08 20:22:53 +00:00
Justin Bronder
acd77c3ac9 bitbake: fetch2/wget: add redirectauth parameter
Add a parameter that limits sending Basic authentication in the
Authorization header to only the first host and not any that we're
redirected to.  Ignoring potential security concerns, temporary AWS URLs
will reject any request that includes authentication details in both the
query parameters (from the redirect) and in the Authorization header.

Temporary AWS URLs are now being used for release assets from private
Github repositories.  According to the previous discussion linked below,
they're also in use by bitbucket.

See also:
https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/

(Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0)

Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-08 20:22:53 +00:00
Stefan Herbrechtsmeier
e241d0e78c bitbake: fetch2: Add striplevel support to unpack
Add a parameter `striplevel` to the SRC_URI to strip NUMBER leading
components (levels) from file names on extraction.

For example, if the archive `archive.tar.gz` contains `some/file`,
the SRC_URI `https://.../archive.tar.gz;subdir=other;striplevel=1`
will extract `some/file` to `other/file`.

This is useful to extract archives to a specified directory instead of
the original root component of the archive. The feature is required for
the npm support. The npm package contents should reside in a subfolder
inside a npm archive (usually it is called package/). npm strips one
directory layer when installing the package.

(Bitbake rev: aa4926e5d9c92f33b4434e2da709ff0bf3049f5b)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-26 17:01:37 +00:00
Stefan Herbrechtsmeier
8091ad4347 bitbake: fetch2: Unify tar command in unpack
The tar command and its arguments are repeated for many archive types in
the unpack function. Unify the common parts in a variable to prepare
further extension.

(Bitbake rev: a08e57c9eaec1d9740a96149bf4843e576da4e5c)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-26 17:01:37 +00:00
Richard Purdie
ffe4af4feb bitbake: fetch2: Fix url remap issue and add testcase
Using "" as a target for .replace() is a really bad idea as it duplicates the replacement
for every character in the string. Add a testcase which triggered this and correct the
code to return the correct result.

(Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-21 11:05:51 +00:00
Richard Purdie
711d173d6e bitbake: fetch: Add README on fetcher design constraints
There have been requests to better document the contraints of fetcher
design and operation. This README attempts to start that.

(Bitbake rev: d9cda7835816ecd5a60f0575f6ce832ec9c6aced)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-11 17:27:20 +00:00
Richard Purdie
2c46245f44 bitbake: fetch/wget: Add timeout for checkstatus calls (30s)
We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:

socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null

and pointing SSTATE_MIRRORS in OE at that address.

Adding a timeout to the checkstatus calls of 15s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.

[YOCTO #13716]

(Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-10 19:25:38 +00:00
Richard Purdie
a4c1325251 bitbake: fetch: Handle mirror user/password replacements correctly
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.

[YOCTO #13823]

(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
34e4eebc32 bitbake: lib/bb: Fix string concatination potential performance issues
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.

The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.

This patch was based on issues highligthted by a report from AWS Codeguru.

(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
9abab49f2b bitbake: lib/bb: Clean up use of len()
(Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
5fb6b453f6 bitbake: fetch/git: Show warning for invalid github urls
On master, tell the users they need to update their urls for github.

(Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
489fe8ff39 bitbake: fetch/git: Handle github dropping git:// support
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.

(Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
d576d9ca3a bitbake: fetch2/git: Add a warning asking users to set a branch in git urls
There is much uncertainty around what tools and hosting providers will
do about default git branch naming in the future. To help ensure we
can handle the various scenarios, we will make branch names required in
SRC_URI. To start that process, show users a warning if it isn't set.
This may also allow us to change the default at some point in the future.

(Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
5acc697506 bitbake: fetch2/git: Allow git fetcher to support subdir param
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.

Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.

We could consider dropping destsuffix as a parameter as some future point.

Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.

(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Stefan Herbrechtsmeier
e17d4895ca bitbake: fetch2: npmsw: Add support for github prefix in npm shrinkwrap version
(Bitbake rev: 1d8af6aed0a929f493d2c3e31b8d3ee3a70beb43)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 11:12:16 +00:00
Stefan Herbrechtsmeier
a3a26e500a bitbake: fetch2: npmsw: Add support for duplicate dependencies without url
(Bitbake rev: 9dcd9a3c671b24982aff9a994e31cb9f491471f5)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 11:12:16 +00:00