Commit Graph

690 Commits

Author SHA1 Message Date
Caner Altinbasak
bd5a07fbcb bitbake: fetch2: Do not attempt to verify donestamp if download is not complete
npmsw(and maybe other fetchers) may try to download same files simulatenously.

if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.

In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.

(Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8)

Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 22:37:43 +01:00
Caner Altinbasak
8e6aaed8ff bitbake: npmsw: Avoid race condition with multiple npm fetchers
If multiple npmsw fetchers are trying to download the same npm file, one of them
can try to download the file while other is calling verify. npmsw methods gets
called without holding the lock, which causes race conditions in fetching and
verification etc. Lock the lockfile before calling proxy fetcher methods.

(Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a)

Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 22:37:43 +01:00
Ross Burton
4a791f342f bitbake: fetch2: document checkstatus API
The checkstatus semantics are unconventional, so document them.

(Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 21:31:09 +01:00
Stefan Herbrechtsmeier
fb437d6db1 bitbake: fetch2: npmsw: Add support for local tarball and link sources
(Bitbake rev: 4f983dc419a1a6f635a5d333f253d49244cec374)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:06 +01:00
Stefan Herbrechtsmeier
00f2b6d0b7 bitbake: fetch2: npm: Create config npmrc in environment instantiation
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.

Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.

(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Stefan Herbrechtsmeier
d2d40e7332 bitbake: fetch2: npm: Support npm archives with missing search directory mode
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.

(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Ross Burton
c3b3d7c14f bitbake: fetch2/gitsm: remove the 'nugget' SRCREV caching
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.

A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball.  If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org.  edk2-firmware will then unpack and build
successfully.

I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before.  The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.

However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.

The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases.  Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.

(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Ross Burton
07996f492b bitbake: fetch2: clarify the command-no-found error message
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.

Add "not found" to clarify what happened.

(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 16:45:05 +01:00
Richard Purdie
c4fa4f9735 bitbake: fetch2/svn: Allow peg-revision functionality to be disabled
Sometimes the peg revision functionality we use in the svn fetcher is
not the correct option. Add a parameter nopegrevision which can be added
to disable this behaviour.

[YOCTO #6258]

(Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-21 23:44:08 +01:00
Richard Purdie
435bce9bf9 bitbake: fetch2/git: Use os.rename instead of mv
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.

(Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-21 23:44:08 +01:00
Valentin Danaila
a4c8d8e21a bitbake: fetch2/s3: allow to switch profile from environment variable
Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles
  and switch between profiles as export of the AWS_PROFILE environment variable

(Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58)

Signed-off-by: Valentin Danaila <drlv85@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-17 07:26:24 +01:00
Richard Purdie
626e949292 bitbake: fetch2/git: Avoid races over mirror tarball creation
There is a potential race over the mirror tarballs where a partial git repo
could be extracted causing fetcher failures if the tarball is being rewritten
whilst another build accesses it.

Create the mirror tarball atomically to avoid this.

[YOCTO #14441]

(Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-17 07:26:23 +01:00
Daniel Ammann
4fb4a6e229 bitbake: fetch2/wget: Enable ftps
The fetcher would fail with:
Could not find a fetcher which supports the URL: ftps://...

(Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-16 09:51:14 +01:00
Richard Purdie
433cff7297 bitbake: fetch2: Add recursion guard
Users sometimes put ${S} references in ${SRC_URI} without realising this can be
problematic. Improve the error messages if they accidentally do.

[YOCTO #11593]

(Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-11 22:39:19 +01:00
Scott Weaver
4939a41982 bitbake: bitbake: fetch2: fix premirror URI when downloadfilename defined
When downloadfilename is defined in a recipe's SRC_URI and PREMIRRORS is also
defined using the same URI, the downloadfilename is appended to the mirror
URI and it should not be.

[YOCTO #13039]

(Bitbake rev: 8a3ff9f3eaf19d4258eb070c5dc230dface269b2)

Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-07 21:54:39 +01:00
Enrico Scholz
e8182a794d bitbake: fetch2/wget: fix 'no_proxy' handling
The urllib.request.ProxyHandler constructor only reads the $http_proxy
+ $https_proxy environment variables.

$no_proxy is evaluated later when the url is opened.

It is therefore not sufficient to just construct the proxy handler in
the

|        with bb.utils.environment(**newenv):

context, but the 'opener.open(r)' call must also be made there.

(Bitbake rev: 076baf4fbd328d247508fd399866a397eb34f67e)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-23 08:30:54 +01:00
Ross Burton
1e2e9a84d6 bitbake: fetch2/wget: fetch securely by default
The days of broken certificates are behind us now, so instead of always
passing --no-check-certificate to wget, don't pass it by default and
instead only pass it BB_CHECK_SSL_CERTS = "0".

[ YOCTO #14108 ]

(Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-12 06:28:01 +01:00
Ross Burton
ad507bd5c4 bitbake: fetch2/wget: ensure all variables are set when calling urllib
Instead of just exporting the proxy variables when calling into urllib,
use bb.utils.environment() to export all of the known variables that are
needed for proper connectivity.

Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can
find the certificates in buildtools environments

(Bitbake rev: 116637b0e9aabae7f680b102dbf3577b8a58f049)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-12 06:28:01 +01:00
Ross Burton
3483841352 bitbake: fetch2: expose environment variable names that need to be exported
There is a list of environment variable names that need to be exported
when executing external commands, such as 'http_proxy'.  To avoid
duplication, make this a top-level variable.

Also add SSL_CERT_FILE, which is used by OpenSSL to locate the
certificate bundle. This is needed in buildtools environments where the
default path isn't valid.

(Bitbake rev: 5c2cf57fd11d91f749a3b562f6f0a863f15013ed)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-12 06:28:01 +01:00
Richard Purdie
7f0f1179eb bitbake: fetch/tests/toaster: Override conversion fixups
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.

(Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-06 06:34:58 +01:00
Richard Purdie
2d7cf6c056 bitbake: doc/lib: Update to use new override syntax containing colons
This runs the overrides conversion script in OE-Core over the bitbake code
base including the docs. A handful of things were excluded in toaster
and for the Changelog file.

(Bitbake rev: 47f8d3b24fd52381bf3b41e2f55a53e57841344c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:10 +01:00
Adam Romanek
5dce2f3da2 bitbake: fetch2/s3: allow to use credentials from environment variables
Previously access to AWS S3 was expected to be preconfigured and
credentials to be stored in ~/.aws/credentials. With this change
one can use Bitbake s3 fetcher without AWS credentials stored
permanently as above, just with them exported as the following
environment variables:
 - AWS_ACCESS_KEY_ID,
 - AWS_SECRET_ACCESS_KEY.
 - AWS_DEFAULT_REGION.

(Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48)

Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-11 23:01:25 +01:00
Adam Romanek
6cd8032e35 bitbake: fetch/git: run gc in foreground to avoid race with tar
It looks like git gc can interrupts the package creation when
BB_GENERATE_MIRRORS_TARBALL is in use.

Log excerpts:

tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output:
tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it
tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.

(Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877)

Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-10 22:43:41 +01:00
Scott Weaver
cbb8744735 bitbake: fetch2: add check for empty SRC_URI hash string
No error was being reported when the hash string was set to empty.
For example: SRC_URI[md5sum] = ""

On a related note (not a bug):
Because whitespace in the string will result in a checksum mismatch, the error
message was updated to make it a little clearer why the error was thrown.
For example: SRC_URI[md5sum] = " " or
SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1"

Now creates a message like this:
File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5
checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected

[YOCTO #14232]

(Bitbake rev: a13510d0028e234ea2f4744b0d0c38558395c70f)

Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-06 23:18:26 +01:00
Alexander Kanavin
37792f1410 bitbake: fetch2/wget: when checking latest versions, consider all numerical directories
Previously the regex was maching x.y, but wasn't matching x, which is a problem
e.g. here:
https://download.gnome.org/sources/epiphany/
(the new gnome version scheme adds 40-series at the end).

(Bitbake rev: c03101576f447263ea38e8464210d3a3a2c27226)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-14 07:57:57 +01:00
Harald Brinkmann
e155a81ab6 bitbake: fetch/svn: Fix parsing revision of SVN repos with redirects
svn was printing a message when encountering HTTP redirects.
This confused the revision parser.

(Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146)

Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:08 +01:00
Przemyslaw Gorszkowski
6da327c788 bitbake: fetch/s3: Add progress handler for S3 cp command
Adds progress support for fetchers from S3.

(Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Niels Avonds
d12a06cd49 bitbake: fetch/gitsm: Fix crash when using git LFS and submodules
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.

This issue was introduced in this
commit: 977b7268bf

[YOCTO #14283]

(Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)

Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Henning Schild
2cb6ce7788 bitbake: fetch/git: add support for disabling shared clones on unpack
By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour.

Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.

With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.

The switch is a global bitbake switch and will affect all git urls.
Build systems that need "noshared" most likely need it for everything
they do with git.

(Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Alejandro Hernandez Samaniego
0f84d24df8 bitbake: bitbake: Add Azure Storage fetcher implementation
Allows bitbake to fetch from an Azure Storage account.

        The fetcher submodule is compatible with the az:// URI protocol, its
        functionality is based on bitbakes wget fetcher, superior in performance
        to using a propietary tool like azcopy which can handle cloud storage
        account operations with more functionality (that we dont need in a fetcher)
	but less compatibility.

        A sample URI uses can be defined in the following way:
	SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz"

        This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.:

        SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n"

        PREMIRRORS_prepend = "\
            git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
        "

        Can also be used with non-public access Azure Storage accounts/containers via a
        Shared Access Signature by declaring the AZ_SAS variable which will be
        automatically used by the fetcher:

        AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature"

(Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Alejandro Hernandez Samaniego
255fdb9f74 bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatus
If an exception is raised when running host python code, the fetcher
    immediately crashes, this might be temporary depending on the servers
    reliability.

    Catch the exception when the connection was reset and try once again
    to fetch the data.

    File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto
         0665:        if self._timeout_occurred:
         0666:            raise OSError("cannot read from timed out object")
         0667:        while True:
         0668:            try:
     *** 0669:                return self._sock.recv_into(b)
         0670:            except timeout:
         0671:                self._timeout_occurred = True
         0672:                raise
         0673:            except error as e:
    Exception: ConnectionResetError: [Errno 104] Connection reset by peer

(Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 17:47:56 +00:00
Joshua Watt
75f87db413 bitbake: logging: Make bitbake logger compatible with python logger
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.

Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.

[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Matt Hoosier
45c02843d1 bitbake: fetch/git: download LFS content too during do_fetch
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.

[YOCTO #14191]

(Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c)

Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:10:46 +00:00
Ross Burton
5d81a91861 bitbake: fetch2: handle empty elements in _param_str_split
_param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws
an exception if a lone semicolon is used as the value doesn't split into two
items.

Fix by checking that the result of the first split has content.

(Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Richard Purdie
5920b3f129 bitbake: fetch2/perforce: Fix localfile to include ud.module
As reported by Katu Txakur, the output depends on the module path
set so this needs to be accounted for in localfile.

(Bitbake rev: 9861ed37bb1c5d09c3b4852d2a252e3f3c86ab14)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Christophe Priouzeau
9f12323826 bitbake: fetch2/wget: Update user-agent
With the usage of enterprise proxy, the user-agent defined are
too old and refused by proxy configuration. Updating to something
more modern is desirable.

(Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400)

Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com>
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Joey Degges
10c69538c0 bitbake: fetch/git: Fix usehead for non-default names
The usehead url parameter for git repositories causes bitbake to use
whatever commit the repository HEAD is pointing to if the repository
happens to have the name 'default'. This is the default name so in many
cases it works just fine, but if a different name is specified with the
url parameter 'name=newName' then it will fail to parse the recipe with
an error along the lines of:

ERROR: ExpansionError during parsing /path/to/my/recipe.bb
Traceback (most recent call last):
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init:
    >        ud.setup_revisions(d)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions:
             for name in self.names:
    >            self.revisions[name] = srcrev_internal_helper(self, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'):
         if srcrev == "AUTOINC":
    >        srcrev = ud.method.latest_revision(ud, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d, name)
                 return rev
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'):
             raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
    >            (ud.unresolvedrev[name], ud.host+ud.path))
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo

Let's fix this by setting the unresolved rev of _all_ repository names
to 'HEAD' when the usehead url parameter is specified. Update the
currently failing test, test_local_gitfetch_usehead_withname, to now
expect success.

This change preserves existing behavior that allows usehead to be
overridden by a valid looking revision if one happens to be specified
instead of AUTOREV.

(Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c)

Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Richard Purdie
4db8652250 bitbake: Revert "fetch2: use relative symlinks for anything pulled from PREMIRRORS"
This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb.

Unfortunately this caused problems where incorrect paths were found in paths
and files were not being found. This was reported on the mailing list but no
invetigation or resolution was found. Revert until we can figure out
what was wrong.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-09 12:36:22 +00:00
Hombourger, Cedric
53217b1121 bitbake: fetch2: use relative symlinks for anything pulled from PREMIRRORS
try_mirror_url() creates a symlink named as the original file to make
everything look like files specified in SRC_URI were downloaded from
their original location. The link is however created as an absolute
reference, this makes DL_DIR non-relocatable. This also causes issues
with the Isar project since it bind mounts DL_DIR to /downloads to
perform some of its build tasks in a chrooted environment (rendering
all symbolic links from DL_DIR invalid). Modify ensure_symlink() to
take an optional "relative" argument and have that function use
os.path.relpath() to produce a relative symlink.

(Bitbake rev: 481e66ea8fc2fc91903127d66b0f1b0fe86baedb)

Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
Cc: Chris Larson <Chris_Larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:26:12 +00:00
Alexander Kanavin
0a2b76064c bitbake: lib/bb/fetch2/__init__.py: drop _PYTHON_SYSCONFIGDATA_NAME unsetting
With introduction of python3targetconfig class in core this is no longer
needed.

(Bitbake rev: 47b64cfacd7c498ef9ed5486d117f2d69a39f225)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 10:50:18 +00:00
Richard Purdie
bd98ef572d bitbake: fetch2/git: Document that we won't support passwords in git urls
This keeps coming up, document why we don't do this and the alternatives.

(Bitbake rev: aded964eed4ce5a725ed1ab477efabc86b1aa481)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-03 08:21:10 +00:00
Mert Kirpici
41ea50dcc8 bitbake: fetch2: add zstd support to unpack
This commit adds unpack support to fetched files with extensions:
'.tar.zst', '.tzst' and '.zst'

(Bitbake rev: 4348743a7e36f3a495e2dc48f83989d9481060f2)

Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-03 08:21:10 +00:00
Charlie Davies
b50b6007e5 bitbake: bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url
This commit replaces the instances where escaped double quotes
are used to support SRC_URI url containing spaces with the more
pythonic shlex.quote().

(Bitbake rev: 4f9ba9c794de55bea0343267467bddea99844374)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-17 12:36:14 +01:00
Leif Middelschulte
b1dff146a6 bitbake: fetch2: fix handling of \ in file:// SRC_URI
Using backslashes in file:// URIs was broken.
Either the resolver would fail or the subsequent `cp` command.
Try to avoid this by putting the filenames into quotes.

Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=8161

(Bitbake rev: aa857fa2e9cf3b0e43a9049b04ec4b0b3c779b11)

Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-25 08:32:02 +01:00
Charlie Davies
9ef2bc360d bitbake: bitbake: fetch/git: add support for SRC_URI containing spaces in url
Microsoft's TFS VCS system allows for spaces in a git repository url.
An example of a valid url is:

ssh://tfs-my-company.org:22/tfs/My Projects/FooBar

This commit adds support for such urls by implementing two changes.
Firstly, when bitbake makes a git command line call the url is
surrounded by quotes so that the url, regardless of spaces, is
treated as one argument. Secondly, additional parsing of various
filepath variables, which are based off of the url, are now
completed with any spaces in the url replaced with underscores.

(Bitbake rev: eb38b6f0935763f7ba19e5618f376fcae1dac41a)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-23 20:55:53 +01:00
Richard Purdie
1560a4b0cb bitbake: fetch2: Drop globbing supprt in file:// SRC_URIs
Globbing in file:// urls is terminally broken. Currently when its used, the
file checksum code is basically bypassed. This means changes to the source
files don't change the task checksum, the task doesn't rebuild when the
inputs change and things generally break.

To make globbing work generically, we'd have to scan the file system for
all possible matches to the glob and log whether they exist or not. We can't
simply log the files which exist, we have to also know which files could
later exist and influence the choice of file so we know when to reparse.

For a simple file://xxx/*, this could be done but for bigger patterns,
it becomes much more problemtic. We already support file://xxx/ in urls.

So, lets decide we'll not support globs in file://urls. Worse case users
can put files in a directory and reference that, moving files into place
if needed.

Remove all the glob special cases (see the comments if anyone doesn't
believe this is terminally broken) and error to the user if they have
such urls.

(Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26 09:05:38 +01:00
Frazer Clews
abc6f864b9 bitbake: lib: fix most undefined code picked up by pylint
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.

(Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5)

Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
e4df6ef7f5 bitbake: fetch2/wget: Remove buffering parameter
The buffering parameter was removed in python 3.1 and made default
so we can clean up the code. This removes weird looking double
exceptions when connections fail.

(Bitbake rev: 06b7bafbd18a47c8db2f7b943dc535c65df176bf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-24 09:08:14 +01:00
Richard Purdie
61931d14df bitbake: fetch2: Change git fetcher not to destroy old references
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.

We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.

I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.

(Bitbake rev: 820ab886e79eea516560c0c008e4cf059c6e11a3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-12 11:53:00 +01:00
Alexandru N. Onea
b7006f50c7 bitbake: perforce: add local path handling SRC_URI options
This patch implements three new SRC_URI options for the perforce
fetcher, namely:

* module
* remotepath

The options are intended to provide the user more control over the
downloaded file paths by allowing the user to specify how much of the remote
path should be preserved locally.

The changes in this patch are backwards compatible, i.e. if none of the
introduced options is specified, the default (old) behavior is enforced.

(Bitbake rev: aab228822d2f221c01337dd57d7582c51ce9a505)

Signed-off-by: Alexandru N. Onea <onea.alex@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-25 10:24:02 +01:00