After commit d591d7633fe8 ("fetch/git: Rework tag parameter handling"),
update the description of the tag= parameter for the Git fetcher.
(Bitbake rev: 85b31a55d114a1430868233d56573b470fef8908)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also, drop an obsolete S = WORKDIR/git note.
(Bitbake rev: 3884b386cda649030f59e3040dd08e6aa2ec50cf)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AZ_SAS token should be prefixed with a question mark. Add a sanity check for
this and fix the documentation.
[YOCTO #15882]
(Bitbake rev: 22011765202514600314732b97f1bb938e21f585)
Signed-off-by: Robbin Van Damme <robbinvandamme@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable is implemented in OE-Core, and therefore
only documented in the Yocto Project manual.
(Bitbake rev: 1019998e2f8682c8f2f13320fdb0de1a9a23e854)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using tags with the git fetcher may cause surprising behaviour. There
are reasons for this, document them.
(Bitbake rev: 56224da378ab63526d44fd7a70bcfd2cffe245cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fetcher allows BitBake to fetch from a Google Cloud Storage
bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and
expects it to be installed, configured, and authenticated prior
to use.
If accepted, this patch should merge in with the corresponding oe-core
patch titled "Add GCP fetcher to list of supported protocols".
Some comments on the patch:
There is also documentation for the fetcher added to the User
Manual. I'm still not completely sure about the recommends_checksum()
being set to True. As I've noted in the mailing list, it will throw
warnings if the fetcher is used in recipes without specifying a
checksum. Please let me know if this is intended behavior or if it
should be modified.
Here is how this fetcher conforms to the fetcher expectations
described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no
network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher
only downloads to the DL_DIR in the same way that other fetchers,
namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this
context
e) Yes, the fetcher output is deterministic because it is downloading
tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy
variable to test whether the Python API respected the proxy. It
appears that it did as I could see traffic passing through the
proxy. I also did some searching online and found posts indicating
that the Google Cloud Python APIs supported the classic Linux proxy
variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading
it if it does.
h) Not applicable, BitBake already knows which version it wants and
the version infomation is encoded in the filename. The fetcher has no
concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests
for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any
tools during parse time. Please correct me if I'm wrong.
(Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b)
Signed-off-by: Emil Ekmečić <eekmecic@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Namespace in this context means a branch, a tag, etc., clarify
it in the description. Also, fix a typo "a any", replace with
plain "any".
This patch is based of feedback on new applied patch
d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata")
(Bitbake rev: b4999425c812b25cb359d5163d11e3c1b030dc28)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every
single object in the remote repository. This works poorly with gitlab
and github, which use the remote git repository to track its metadata
like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/*
and refs/keep-around/* and they all contain massive amount of data that
are useless for the bitbake build purposes. The amount of useless data
can in fact be so massive (e.g. with FDO mesa.git repository) that some
proxies may outright terminate the 'git fetch' connection, and make it
appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such
that it only fetches refs/heads/* and refs/tags/* . Avoid using negative
refspecs as those are only available in new git versions.
Per feedback on the ML, Gerrit may push commits outsides of branches or
tags during CI runs, which currently works with the 'nobranch=1' fetcher
parameter. To retain this functionality, keep fetching everything in case
the 'nobranch=1' is present. This still avoids fetching massive amount of
data in the common case, since 'nobranch=1' is rare. Update 'nobranch'
documentation.
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
(Bitbake rev: d32e5b0ec2ab85ffad7e56ac5b3160860b732556)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL expected by git fetcher when the SSH protocol is used differs
from the one given by Git servers to be used with the `git clone`
command.
Add a note making this specificity known to users.
Cc: Quentin Schulz <foss+yocto@0leil.net>
(Bitbake rev: 81c09d434aed13fcfc4ba426ae6a0b849447eb5f)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a parameter that limits sending Basic authentication in the
Authorization header to only the first host and not any that we're
redirected to. Ignoring potential security concerns, temporary AWS URLs
will reject any request that includes authentication details in both the
query parameters (from the redirect) and in the Authorization header.
Temporary AWS URLs are now being used for release assets from private
Github repositories. According to the previous discussion linked below,
they're also in use by bitbucket.
See also:
https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/
(Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0)
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This syntax is obsolete, update to the correct modern version.
(Bitbake rev: 744e9a4c2b6f44116435feb62ac64ff256c752e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Document that S has to be set to "${WORKDIR}/git" in order for the
recipe to work if the git fetcher is fetching the main source of code
for the recipe.
(Bitbake rev: c394f34437686c2f57a27f0c06d7aeb3268bfd41)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The days of broken certificates are behind us now, so instead of always
passing --no-check-certificate to wget, don't pass it by default and
instead only pass it BB_CHECK_SSL_CERTS = "0".
[ YOCTO #14108 ]
(Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a variable has a glossary entry and some rST files write about those
variables, it's better to point to the glossary entry instead of just
highlighting it by surrounding it with two tick quotes.
The script that is used to do the replacement of ``FOO`` by :term:`FOO`
is the following Python code:
import re
from pathlib import Path
from runpy import run_module
import contextlib
import io
import sys
re_term = re.compile(r'variables.html#term-([a-zA-Z_0-9]*)')
terms = []
new_terms = set()
with contextlib.redirect_stdout(io.StringIO()) as f:
run_module('sphinx.ext.intersphinx', run_name='__main__')
objects = f.getvalue()
match = re_term.search(objects)
while match:
if match.group(1):
terms.append(match.group(1))
match = re_term.search(objects, match.end())
for rst in Path('.').rglob('*.rst'):
with open(rst, 'r') as f:
content = "".join(f.readlines())
for term in terms:
content = re.sub(r'``({})``(?!.*\s+[~=-]{{{:d},}})'.format(term, len(term)), r':term:`\1`', content)
with open(rst, 'w') as f:
f.write(content)
This script takes one argument as input: an objects.inv which can be
gotten from doc/_build/html/objetcs.inv after running `make html`.
Note that this excludes from replacement terms that appear in section
titles as it requires refs to be changed too. This can be automated too
if need be but right now it looks a bit confusing to have an anchor link
(for sections) also have a term/reference link in it. I am not sure this
is desired today.
(Bitbake rev: aba88f40c47133ed9bc999e0298aca3bc8490912)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This simplifies paragraphs ending with a colon and followed
by code insertion.
Automatically substituted through the command:
sed -i -z "s/:\n\s*::/::/g" file.rst
This generates identical HTML output.
(Bitbake rev: 51c80fc3497eecc8e50194fe1ff8069b59f03eda)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- This replaces instances of ": ::" by "::", which
generates identical HTML output
(Bitbake rev: fd8ce4dcaff3aae395f9945fb0a3be54905e1727)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code for this was removed in 2016 in commit
e659a3b0c2771679057ee3e13cd42e6c62383ff2. Nobody seems to have missed it
so remove the documentation so we match the code.
[YOCTO #13449]
(Bitbake rev: 76bf42ea41a28b19d0377c2e548b0a59119fdf67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is based on the comment added in revision
aded964eed4ce5a725ed1ab477efabc86b1aa481.
(Bitbake rev: 082683da089115d8b6f71f221cabb41ac401f733)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Review all pages, and fix up for formatting which was not covered by
pandoc, such as some links and code block sections.
(Bitbake rev: d99760cc687cc9c24d6d9a1c49fa094574476e0c)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we converted the list of variables into a Sphinx glossary, the
automatic conversion from Pandoc does not produce proper links. We fix
them up using a Python regexp.
Similarly some http links were not converted correctly, and can also
be fixed up with a regexp.
This patch was generated by running the following regexp:
line = re.sub("` <(https?://.*)>`__",
"\\1",
line)
line = re.sub("`+(\w+)`* <#var-bb-\\1>`__",
":term:`\\1`",
line)
(Bitbake rev: 57300955beb200c81762880f44f1a499451b7432)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is autogenerated pandoc to generate an inital set
of reST files based on DocBook XML files.
A .rst file is generated for each .xml files in all manuals with this
command:
cd <manual>
for i in *.xml; do \
pandoc -f docbook -t rst --shift-heading-level-by=-1 \
$i -o $(basename $i .xml).rst \
done
The conversion was done with: pandoc 2.9.2.1-91 (Arch Linux).
Also created an initial top level index file for each document, and
added all 'books' to the top leve index.rst file.
The YP manuals layout is organized as:
Book
Chapter
Section
Section
Section
Sphinx uses section headers to create the document structure.
ReStructuredText defines sections headers like that:
To break longer text up into sections, you use section headers. These
are a single line of text (one or more words) with adornment: an
underline alone, or an underline and an overline together, in dashes
"-----", equals "======", tildes "~~~~~~" or any of the
non-alphanumeric characters = - ` : ' " ~ ^ _ * + # < > that you feel
comfortable with. An underline-only adornment is distinct from an
overline-and-underline adornment using the same character. The
underline/overline must be at least as long as the title text. Be
consistent, since all sections marked with the same adornment style
are deemed to be at the same level:
Let's define the following convention when converting from Docbook:
Book => overline === (Title)
Chapter => overline *** (1.)
Section => ==== (1.1)
Section => ---- (1.1.1)
Section => ~~~~ (1.1.1.1)
Section => ^^^^ (1.1.1.1.1)
During the conversion with pandoc, we used --shift-heading-level=-1 to
convert most of DocBook headings automatically. However with this
setting, the Chapter header was removed, so I added it back
manually. Without this setting all headings were off by one, which was
more difficult to manually fix.
At least with this change, we now have the same TOC with Sphinx and
DocBook.
(Bitbake rev: 6bf6c8d63787aed7624793c24af3fa603b5ac961)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
