Commit Graph

701 Commits

Author SHA1 Message Date
Richard Purdie
a4c1325251 bitbake: fetch: Handle mirror user/password replacements correctly
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.

[YOCTO #13823]

(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
34e4eebc32 bitbake: lib/bb: Fix string concatination potential performance issues
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.

The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.

This patch was based on issues highligthted by a report from AWS Codeguru.

(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
9abab49f2b bitbake: lib/bb: Clean up use of len()
(Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
5fb6b453f6 bitbake: fetch/git: Show warning for invalid github urls
On master, tell the users they need to update their urls for github.

(Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
489fe8ff39 bitbake: fetch/git: Handle github dropping git:// support
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.

(Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
d576d9ca3a bitbake: fetch2/git: Add a warning asking users to set a branch in git urls
There is much uncertainty around what tools and hosting providers will
do about default git branch naming in the future. To help ensure we
can handle the various scenarios, we will make branch names required in
SRC_URI. To start that process, show users a warning if it isn't set.
This may also allow us to change the default at some point in the future.

(Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
5acc697506 bitbake: fetch2/git: Allow git fetcher to support subdir param
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.

Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.

We could consider dropping destsuffix as a parameter as some future point.

Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.

(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Stefan Herbrechtsmeier
e17d4895ca bitbake: fetch2: npmsw: Add support for github prefix in npm shrinkwrap version
(Bitbake rev: 1d8af6aed0a929f493d2c3e31b8d3ee3a70beb43)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 11:12:16 +00:00
Stefan Herbrechtsmeier
a3a26e500a bitbake: fetch2: npmsw: Add support for duplicate dependencies without url
(Bitbake rev: 9dcd9a3c671b24982aff9a994e31cb9f491471f5)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 11:12:16 +00:00
Chen Qi
c94075d623 bitbake: fetch2: fix downloadfilename issue with premirror
The following commit to fix [Yocto #13039] causes regression of
the behavior of PREMIRRORS.

  "bitbake: fetch2: fix premirror URI when downloadfilename defined"

Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
as an example.
SRC_URI = "\
    http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
    ...
"
With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.

The commit above tries to avoid fetching from invalid url such as:
https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
regression.

This patch fixes the above regression. For Yocto #13039, it now tries
to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.

(Bitbake rev: 96c30007dc0b32eee2b15771daec7948bc9bfd97)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 11:12:16 +00:00
Richard Purdie
491b23a932 bitbake: fetch2/perforce: Fix typo
(Bitbake rev: cf421235bd6f5eb12e9193634c0e870ab035b191)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-26 13:47:24 +01:00
Caner Altinbasak
bd5a07fbcb bitbake: fetch2: Do not attempt to verify donestamp if download is not complete
npmsw(and maybe other fetchers) may try to download same files simulatenously.

if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.

In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.

(Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8)

Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 22:37:43 +01:00
Caner Altinbasak
8e6aaed8ff bitbake: npmsw: Avoid race condition with multiple npm fetchers
If multiple npmsw fetchers are trying to download the same npm file, one of them
can try to download the file while other is calling verify. npmsw methods gets
called without holding the lock, which causes race conditions in fetching and
verification etc. Lock the lockfile before calling proxy fetcher methods.

(Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a)

Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 22:37:43 +01:00
Ross Burton
4a791f342f bitbake: fetch2: document checkstatus API
The checkstatus semantics are unconventional, so document them.

(Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 21:31:09 +01:00
Stefan Herbrechtsmeier
fb437d6db1 bitbake: fetch2: npmsw: Add support for local tarball and link sources
(Bitbake rev: 4f983dc419a1a6f635a5d333f253d49244cec374)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:06 +01:00
Stefan Herbrechtsmeier
00f2b6d0b7 bitbake: fetch2: npm: Create config npmrc in environment instantiation
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.

Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.

(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Stefan Herbrechtsmeier
d2d40e7332 bitbake: fetch2: npm: Support npm archives with missing search directory mode
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.

(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Ross Burton
c3b3d7c14f bitbake: fetch2/gitsm: remove the 'nugget' SRCREV caching
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.

A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball.  If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org.  edk2-firmware will then unpack and build
successfully.

I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before.  The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.

However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.

The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases.  Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.

(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Ross Burton
07996f492b bitbake: fetch2: clarify the command-no-found error message
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.

Add "not found" to clarify what happened.

(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Richard Purdie
c4fa4f9735 bitbake: fetch2/svn: Allow peg-revision functionality to be disabled
Sometimes the peg revision functionality we use in the svn fetcher is
not the correct option. Add a parameter nopegrevision which can be added
to disable this behaviour.

[YOCTO #6258]

(Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-21 23:44:08 +01:00
Richard Purdie
435bce9bf9 bitbake: fetch2/git: Use os.rename instead of mv
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.

(Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-21 23:44:08 +01:00
Valentin Danaila
a4c8d8e21a bitbake: fetch2/s3: allow to switch profile from environment variable
Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles
  and switch between profiles as export of the AWS_PROFILE environment variable

(Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58)

Signed-off-by: Valentin Danaila <drlv85@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-17 07:26:24 +01:00
Richard Purdie
626e949292 bitbake: fetch2/git: Avoid races over mirror tarball creation
There is a potential race over the mirror tarballs where a partial git repo
could be extracted causing fetcher failures if the tarball is being rewritten
whilst another build accesses it.

Create the mirror tarball atomically to avoid this.

[YOCTO #14441]

(Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-17 07:26:23 +01:00
Daniel Ammann
4fb4a6e229 bitbake: fetch2/wget: Enable ftps
The fetcher would fail with:
Could not find a fetcher which supports the URL: ftps://...

(Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-16 09:51:14 +01:00
Richard Purdie
433cff7297 bitbake: fetch2: Add recursion guard
Users sometimes put ${S} references in ${SRC_URI} without realising this can be
problematic. Improve the error messages if they accidentally do.

[YOCTO #11593]

(Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-11 22:39:19 +01:00
Scott Weaver
4939a41982 bitbake: bitbake: fetch2: fix premirror URI when downloadfilename defined
When downloadfilename is defined in a recipe's SRC_URI and PREMIRRORS is also
defined using the same URI, the downloadfilename is appended to the mirror
URI and it should not be.

[YOCTO #13039]

(Bitbake rev: 8a3ff9f3eaf19d4258eb070c5dc230dface269b2)

Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-07 21:54:39 +01:00
Enrico Scholz
e8182a794d bitbake: fetch2/wget: fix 'no_proxy' handling
The urllib.request.ProxyHandler constructor only reads the $http_proxy
+ $https_proxy environment variables.

$no_proxy is evaluated later when the url is opened.

It is therefore not sufficient to just construct the proxy handler in
the

|        with bb.utils.environment(**newenv):

context, but the 'opener.open(r)' call must also be made there.

(Bitbake rev: 076baf4fbd328d247508fd399866a397eb34f67e)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-23 08:30:54 +01:00
Ross Burton
1e2e9a84d6 bitbake: fetch2/wget: fetch securely by default
The days of broken certificates are behind us now, so instead of always
passing --no-check-certificate to wget, don't pass it by default and
instead only pass it BB_CHECK_SSL_CERTS = "0".

[ YOCTO #14108 ]

(Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-12 06:28:01 +01:00
Ross Burton
ad507bd5c4 bitbake: fetch2/wget: ensure all variables are set when calling urllib
Instead of just exporting the proxy variables when calling into urllib,
use bb.utils.environment() to export all of the known variables that are
needed for proper connectivity.

Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can
find the certificates in buildtools environments

(Bitbake rev: 116637b0e9aabae7f680b102dbf3577b8a58f049)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-12 06:28:01 +01:00
Ross Burton
3483841352 bitbake: fetch2: expose environment variable names that need to be exported
There is a list of environment variable names that need to be exported
when executing external commands, such as 'http_proxy'.  To avoid
duplication, make this a top-level variable.

Also add SSL_CERT_FILE, which is used by OpenSSL to locate the
certificate bundle. This is needed in buildtools environments where the
default path isn't valid.

(Bitbake rev: 5c2cf57fd11d91f749a3b562f6f0a863f15013ed)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-12 06:28:01 +01:00
Richard Purdie
7f0f1179eb bitbake: fetch/tests/toaster: Override conversion fixups
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.

(Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-06 06:34:58 +01:00
Richard Purdie
2d7cf6c056 bitbake: doc/lib: Update to use new override syntax containing colons
This runs the overrides conversion script in OE-Core over the bitbake code
base including the docs. A handful of things were excluded in toaster
and for the Changelog file.

(Bitbake rev: 47f8d3b24fd52381bf3b41e2f55a53e57841344c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:10 +01:00
Adam Romanek
5dce2f3da2 bitbake: fetch2/s3: allow to use credentials from environment variables
Previously access to AWS S3 was expected to be preconfigured and
credentials to be stored in ~/.aws/credentials. With this change
one can use Bitbake s3 fetcher without AWS credentials stored
permanently as above, just with them exported as the following
environment variables:
 - AWS_ACCESS_KEY_ID,
 - AWS_SECRET_ACCESS_KEY.
 - AWS_DEFAULT_REGION.

(Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48)

Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-11 23:01:25 +01:00
Adam Romanek
6cd8032e35 bitbake: fetch/git: run gc in foreground to avoid race with tar
It looks like git gc can interrupts the package creation when
BB_GENERATE_MIRRORS_TARBALL is in use.

Log excerpts:

tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output:
tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it
tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.

(Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877)

Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-10 22:43:41 +01:00
Scott Weaver
cbb8744735 bitbake: fetch2: add check for empty SRC_URI hash string
No error was being reported when the hash string was set to empty.
For example: SRC_URI[md5sum] = ""

On a related note (not a bug):
Because whitespace in the string will result in a checksum mismatch, the error
message was updated to make it a little clearer why the error was thrown.
For example: SRC_URI[md5sum] = " " or
SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1"

Now creates a message like this:
File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5
checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected

[YOCTO #14232]

(Bitbake rev: a13510d0028e234ea2f4744b0d0c38558395c70f)

Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-06 23:18:26 +01:00
Alexander Kanavin
37792f1410 bitbake: fetch2/wget: when checking latest versions, consider all numerical directories
Previously the regex was maching x.y, but wasn't matching x, which is a problem
e.g. here:
https://download.gnome.org/sources/epiphany/
(the new gnome version scheme adds 40-series at the end).

(Bitbake rev: c03101576f447263ea38e8464210d3a3a2c27226)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-14 07:57:57 +01:00
Harald Brinkmann
e155a81ab6 bitbake: fetch/svn: Fix parsing revision of SVN repos with redirects
svn was printing a message when encountering HTTP redirects.
This confused the revision parser.

(Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146)

Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:08 +01:00
Przemyslaw Gorszkowski
6da327c788 bitbake: fetch/s3: Add progress handler for S3 cp command
Adds progress support for fetchers from S3.

(Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Niels Avonds
d12a06cd49 bitbake: fetch/gitsm: Fix crash when using git LFS and submodules
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.

This issue was introduced in this
commit: 977b7268bf

[YOCTO #14283]

(Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)

Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Henning Schild
2cb6ce7788 bitbake: fetch/git: add support for disabling shared clones on unpack
By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour.

Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.

With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.

The switch is a global bitbake switch and will affect all git urls.
Build systems that need "noshared" most likely need it for everything
they do with git.

(Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Alejandro Hernandez Samaniego
0f84d24df8 bitbake: bitbake: Add Azure Storage fetcher implementation
Allows bitbake to fetch from an Azure Storage account.

        The fetcher submodule is compatible with the az:// URI protocol, its
        functionality is based on bitbakes wget fetcher, superior in performance
        to using a propietary tool like azcopy which can handle cloud storage
        account operations with more functionality (that we dont need in a fetcher)
	but less compatibility.

        A sample URI uses can be defined in the following way:
	SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz"

        This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.:

        SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n"

        PREMIRRORS_prepend = "\
            git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
        "

        Can also be used with non-public access Azure Storage accounts/containers via a
        Shared Access Signature by declaring the AZ_SAS variable which will be
        automatically used by the fetcher:

        AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature"

(Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Alejandro Hernandez Samaniego
255fdb9f74 bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatus
If an exception is raised when running host python code, the fetcher
    immediately crashes, this might be temporary depending on the servers
    reliability.

    Catch the exception when the connection was reset and try once again
    to fetch the data.

    File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto
         0665:        if self._timeout_occurred:
         0666:            raise OSError("cannot read from timed out object")
         0667:        while True:
         0668:            try:
     *** 0669:                return self._sock.recv_into(b)
         0670:            except timeout:
         0671:                self._timeout_occurred = True
         0672:                raise
         0673:            except error as e:
    Exception: ConnectionResetError: [Errno 104] Connection reset by peer

(Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 17:47:56 +00:00
Joshua Watt
75f87db413 bitbake: logging: Make bitbake logger compatible with python logger
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.

Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.

[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Matt Hoosier
45c02843d1 bitbake: fetch/git: download LFS content too during do_fetch
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.

[YOCTO #14191]

(Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c)

Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:10:46 +00:00
Ross Burton
5d81a91861 bitbake: fetch2: handle empty elements in _param_str_split
_param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws
an exception if a lone semicolon is used as the value doesn't split into two
items.

Fix by checking that the result of the first split has content.

(Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Richard Purdie
5920b3f129 bitbake: fetch2/perforce: Fix localfile to include ud.module
As reported by Katu Txakur, the output depends on the module path
set so this needs to be accounted for in localfile.

(Bitbake rev: 9861ed37bb1c5d09c3b4852d2a252e3f3c86ab14)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Christophe Priouzeau
9f12323826 bitbake: fetch2/wget: Update user-agent
With the usage of enterprise proxy, the user-agent defined are
too old and refused by proxy configuration. Updating to something
more modern is desirable.

(Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400)

Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com>
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Joey Degges
10c69538c0 bitbake: fetch/git: Fix usehead for non-default names
The usehead url parameter for git repositories causes bitbake to use
whatever commit the repository HEAD is pointing to if the repository
happens to have the name 'default'. This is the default name so in many
cases it works just fine, but if a different name is specified with the
url parameter 'name=newName' then it will fail to parse the recipe with
an error along the lines of:

ERROR: ExpansionError during parsing /path/to/my/recipe.bb
Traceback (most recent call last):
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init:
    >        ud.setup_revisions(d)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions:
             for name in self.names:
    >            self.revisions[name] = srcrev_internal_helper(self, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'):
         if srcrev == "AUTOINC":
    >        srcrev = ud.method.latest_revision(ud, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d, name)
                 return rev
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'):
             raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
    >            (ud.unresolvedrev[name], ud.host+ud.path))
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo

Let's fix this by setting the unresolved rev of _all_ repository names
to 'HEAD' when the usehead url parameter is specified. Update the
currently failing test, test_local_gitfetch_usehead_withname, to now
expect success.

This change preserves existing behavior that allows usehead to be
overridden by a valid looking revision if one happens to be specified
instead of AUTOREV.

(Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c)

Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Richard Purdie
4db8652250 bitbake: Revert "fetch2: use relative symlinks for anything pulled from PREMIRRORS"
This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb.

Unfortunately this caused problems where incorrect paths were found in paths
and files were not being found. This was reported on the mailing list but no
invetigation or resolution was found. Revert until we can figure out
what was wrong.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-09 12:36:22 +00:00
Hombourger, Cedric
53217b1121 bitbake: fetch2: use relative symlinks for anything pulled from PREMIRRORS
try_mirror_url() creates a symlink named as the original file to make
everything look like files specified in SRC_URI were downloaded from
their original location. The link is however created as an absolute
reference, this makes DL_DIR non-relocatable. This also causes issues
with the Isar project since it bind mounts DL_DIR to /downloads to
perform some of its build tasks in a chrooted environment (rendering
all symbolic links from DL_DIR invalid). Modify ensure_symlink() to
take an optional "relative" argument and have that function use
os.path.relpath() to produce a relative symlink.

(Bitbake rev: 481e66ea8fc2fc91903127d66b0f1b0fe86baedb)

Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
Cc: Chris Larson <Chris_Larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:26:12 +00:00