Always use a temporary file for the user config 'NPM_CONFIG_USERCONFIG'
because npm otherwise failed if configs and npmrc aren't set:
double-loading config "/dev/null" as "global", previously loaded as "user"
(Bitbake rev: 9f272ad7f76c1559e745e9af686d0a529f917659)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Quote destdir in run chmod command to support special characters in
package name and to avoid syntax error for packages like
'@(._.)/execute'.
(Bitbake rev: a701dfce3f0e74b4d7c687eeda83fe9c8e7240b1)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a parameter that limits sending Basic authentication in the
Authorization header to only the first host and not any that we're
redirected to. Ignoring potential security concerns, temporary AWS URLs
will reject any request that includes authentication details in both the
query parameters (from the redirect) and in the Authorization header.
Temporary AWS URLs are now being used for release assets from private
Github repositories. According to the previous discussion linked below,
they're also in use by bitbucket.
See also:
https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/
(Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0)
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a parameter `striplevel` to the SRC_URI to strip NUMBER leading
components (levels) from file names on extraction.
For example, if the archive `archive.tar.gz` contains `some/file`,
the SRC_URI `https://.../archive.tar.gz;subdir=other;striplevel=1`
will extract `some/file` to `other/file`.
This is useful to extract archives to a specified directory instead of
the original root component of the archive. The feature is required for
the npm support. The npm package contents should reside in a subfolder
inside a npm archive (usually it is called package/). npm strips one
directory layer when installing the package.
(Bitbake rev: aa4926e5d9c92f33b4434e2da709ff0bf3049f5b)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tar command and its arguments are repeated for many archive types in
the unpack function. Unify the common parts in a variable to prepare
further extension.
(Bitbake rev: a08e57c9eaec1d9740a96149bf4843e576da4e5c)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using "" as a target for .replace() is a really bad idea as it duplicates the replacement
for every character in the string. Add a testcase which triggered this and correct the
code to return the correct result.
(Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been requests to better document the contraints of fetcher
design and operation. This README attempts to start that.
(Bitbake rev: d9cda7835816ecd5a60f0575f6ce832ec9c6aced)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:
socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null
and pointing SSTATE_MIRRORS in OE at that address.
Adding a timeout to the checkstatus calls of 15s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.
[YOCTO #13716]
(Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.
[YOCTO #13823]
(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.
The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.
This patch was based on issues highligthted by a report from AWS Codeguru.
(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On master, tell the users they need to update their urls for github.
(Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.
(Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is much uncertainty around what tools and hosting providers will
do about default git branch naming in the future. To help ensure we
can handle the various scenarios, we will make branch names required in
SRC_URI. To start that process, show users a warning if it isn't set.
This may also allow us to change the default at some point in the future.
(Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.
Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.
We could consider dropping destsuffix as a parameter as some future point.
Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.
(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following commit to fix [Yocto #13039] causes regression of
the behavior of PREMIRRORS.
"bitbake: fetch2: fix premirror URI when downloadfilename defined"
Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
as an example.
SRC_URI = "\
http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
...
"
With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.
The commit above tries to avoid fetching from invalid url such as:
https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
regression.
This patch fixes the above regression. For Yocto #13039, it now tries
to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.
(Bitbake rev: 96c30007dc0b32eee2b15771daec7948bc9bfd97)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npmsw(and maybe other fetchers) may try to download same files simulatenously.
if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.
In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.
(Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If multiple npmsw fetchers are trying to download the same npm file, one of them
can try to download the file while other is calling verify. npmsw methods gets
called without holding the lock, which causes race conditions in fetching and
verification etc. Lock the lockfile before calling proxy fetcher methods.
(Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The checkstatus semantics are unconventional, so document them.
(Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.
Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.
(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.
(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.
A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball. If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org. edk2-firmware will then unpack and build
successfully.
I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before. The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.
However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.
The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases. Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.
(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.
Add "not found" to clarify what happened.
(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes the peg revision functionality we use in the svn fetcher is
not the correct option. Add a parameter nopegrevision which can be added
to disable this behaviour.
[YOCTO #6258]
(Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.
(Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles
and switch between profiles as export of the AWS_PROFILE environment variable
(Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58)
Signed-off-by: Valentin Danaila <drlv85@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a potential race over the mirror tarballs where a partial git repo
could be extracted causing fetcher failures if the tarball is being rewritten
whilst another build accesses it.
Create the mirror tarball atomically to avoid this.
[YOCTO #14441]
(Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher would fail with:
Could not find a fetcher which supports the URL: ftps://...
(Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4)
Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users sometimes put ${S} references in ${SRC_URI} without realising this can be
problematic. Improve the error messages if they accidentally do.
[YOCTO #11593]
(Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When downloadfilename is defined in a recipe's SRC_URI and PREMIRRORS is also
defined using the same URI, the downloadfilename is appended to the mirror
URI and it should not be.
[YOCTO #13039]
(Bitbake rev: 8a3ff9f3eaf19d4258eb070c5dc230dface269b2)
Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The urllib.request.ProxyHandler constructor only reads the $http_proxy
+ $https_proxy environment variables.
$no_proxy is evaluated later when the url is opened.
It is therefore not sufficient to just construct the proxy handler in
the
| with bb.utils.environment(**newenv):
context, but the 'opener.open(r)' call must also be made there.
(Bitbake rev: 076baf4fbd328d247508fd399866a397eb34f67e)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The days of broken certificates are behind us now, so instead of always
passing --no-check-certificate to wget, don't pass it by default and
instead only pass it BB_CHECK_SSL_CERTS = "0".
[ YOCTO #14108 ]
(Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of just exporting the proxy variables when calling into urllib,
use bb.utils.environment() to export all of the known variables that are
needed for proper connectivity.
Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can
find the certificates in buildtools environments
(Bitbake rev: 116637b0e9aabae7f680b102dbf3577b8a58f049)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a list of environment variable names that need to be exported
when executing external commands, such as 'http_proxy'. To avoid
duplication, make this a top-level variable.
Also add SSL_CERT_FILE, which is used by OpenSSL to locate the
certificate bundle. This is needed in buildtools environments where the
default path isn't valid.
(Bitbake rev: 5c2cf57fd11d91f749a3b562f6f0a863f15013ed)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.
(Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This runs the overrides conversion script in OE-Core over the bitbake code
base including the docs. A handful of things were excluded in toaster
and for the Changelog file.
(Bitbake rev: 47f8d3b24fd52381bf3b41e2f55a53e57841344c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously access to AWS S3 was expected to be preconfigured and
credentials to be stored in ~/.aws/credentials. With this change
one can use Bitbake s3 fetcher without AWS credentials stored
permanently as above, just with them exported as the following
environment variables:
- AWS_ACCESS_KEY_ID,
- AWS_SECRET_ACCESS_KEY.
- AWS_DEFAULT_REGION.
(Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48)
Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It looks like git gc can interrupts the package creation when
BB_GENERATE_MIRRORS_TARBALL is in use.
Log excerpts:
tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output:
tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it
tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
(Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877)
Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No error was being reported when the hash string was set to empty.
For example: SRC_URI[md5sum] = ""
On a related note (not a bug):
Because whitespace in the string will result in a checksum mismatch, the error
message was updated to make it a little clearer why the error was thrown.
For example: SRC_URI[md5sum] = " " or
SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1"
Now creates a message like this:
File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5
checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected
[YOCTO #14232]
(Bitbake rev: a13510d0028e234ea2f4744b0d0c38558395c70f)
Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
svn was printing a message when encountering HTTP redirects.
This confused the revision parser.
(Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146)
Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds progress support for fetchers from S3.
(Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.
This issue was introduced in this
commit: 977b7268bf
[YOCTO #14283]
(Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)
Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour.
Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.
With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.
The switch is a global bitbake switch and will affect all git urls.
Build systems that need "noshared" most likely need it for everything
they do with git.
(Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102)
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allows bitbake to fetch from an Azure Storage account.
The fetcher submodule is compatible with the az:// URI protocol, its
functionality is based on bitbakes wget fetcher, superior in performance
to using a propietary tool like azcopy which can handle cloud storage
account operations with more functionality (that we dont need in a fetcher)
but less compatibility.
A sample URI uses can be defined in the following way:
SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz"
This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.:
SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n"
PREMIRRORS_prepend = "\
git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
"
Can also be used with non-public access Azure Storage accounts/containers via a
Shared Access Signature by declaring the AZ_SAS variable which will be
automatically used by the fetcher:
AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature"
(Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
