It is helpful to have timestamps on the ping failures so that they
can be matched against the bitbake logs. It is also useful to understand
how long the server takes for form a reply verses when it is sent.
(Bitbake rev: 65969a7a8f5ae22c230431d2db080eb187a27708)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* it might be a bit confusing as it shows number of threads before making
the decision to start more tasks and also it can show only a few tasks
running, but not because of pressure when there just aren't many tasks
left or wait for their dependencies to be finished first
* example output:
NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads
NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads
(Bitbake rev: b0b114f31f20c5fcde31e6c308937ad4102dfe0a)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable is implemented in OE-Core, and therefore
only documented in the Yocto Project manual.
(Bitbake rev: 1019998e2f8682c8f2f13320fdb0de1a9a23e854)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using tags with the git fetcher may cause surprising behaviour. There
are reasons for this, document them.
(Bitbake rev: 56224da378ab63526d44fd7a70bcfd2cffe245cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no reason to have the else clause in this try block, as it can
be moved into the try block, which is clearer.
(Bitbake rev: 5625849e9327fc71a38eea00d4506f80abc11bc6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We noticed some users were seeing very slow ping response times which
caused 'server timeout' issues. There were some function calls in runCommand
which could be slow such as the inotify callback processing.
Mark up the ping command such that it doesn't need configuration information,
it is allowed on a readonly server and specifically skip the inofity processing
too since ping would never need that.
This will hopefully resolve various ping timeout issues that were being reported.
(Bitbake rev: 0fc821a22f2b49cbd336d9658d92942c0d733be1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have a contributor guide combining various wiki pages, point at
that.
(Bitbake rev: fb19d647697d56e7554722abb5f4903c774d4213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value "Image Name" in bitbake events was missing for certain builds. Update 'buildinfohelper' to extract the
image name elsewhere in this circumstance and not crash.
[YOCTO #13191]
(Bitbake rev: 703792c48c818025163de9c2f35f6ac815500607)
Signed-off-by: Kieran McNulty <Kieran.McNulty@windriver.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you import a build directory, Toaster still adds openembedded-core,
meta-poky and meta-yocto-bsp to the newly created project. Toaster
should only be including in the project the layers that it imported.
[YOCTO #13764]
(Bitbake rev: e73c4d7685a3bd6b806a8f1a3600a3a86266f0b6)
Signed-off-by: Kieran McNulty <Kieran.McNulty@windriver.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the clone target directory exists and is a valid git repo, but the
git directory is not a child, it needs to be erased and re-cloned. One
example of how this can happen is if a clone creates the directory, but
then fails to actual clone and make it a git repository. This left-over
directory can be particularly problematic if the download directory is a
descent of some top layer git repo (e.g. the default with poky), as the
commands that operate on the remote later will then mangle the layers
git repository instead of the download git repo.
(Bitbake rev: 2117db3146ce38bb4a6e2df40b6cd2ab11b514d5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Explicitly pass down the lfs parameter from the source repo URI to
submodules. Add smudge skip to final checkout phase to make sure we
don't access the network here. Everything should have been fetched and
setup from the lfs logic in the git fetcher at this point.
(Bitbake rev: 1f8f21fe024b391d3a6670c64b839db0eee083c3)
Signed-off-by: Randolph Sapp <rs@ti.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Error occured while running bitbake on cephfs:
WARNING: The free inode of path is running low (-0.001K left)
ERROR: Immediately halt since the disk space monitor action is "HALT"!
(Bitbake rev: 95088b447f563c5e1d9630e6acb32787b5ebed9c)
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds Python wrappers around the xattr API from libc and the ACL API from
libacl.
(Bitbake rev: 538011256964d0253f8e3ab7ff1d6fd62c7c2f89)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* with latest bitbake I'm seeing very low number of bitbake tasks
executed in parallel, probably due to pressure regulation
show the values this is based on in the note
* also simplify a bit by counting the pressure and exceeds signs
only once
(Bitbake rev: 21c17968f801e406ef7f328656587fadd9ef7f5d)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add new functions to return some of the get_srcrev data in new and different
ways. We need two different forms of the data, one is a string to inject into
PKGV, the other is the full revisions as a string to include in hash computations
so that the hash changes when the input revisions change.
This allows us to clean up and simplify the code in OE-Core and move the
version information from PV to PKGV.
(Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
file checksums are part of the data checksummed
to generate the task hash. The list of file checksums
was not ordered.
In this commit we make sure the task hash checksum takes
a list of checksum data that is ordered by unique file name
thus guaranteeing reproducibility.
(Bitbake rev: 5293a1b36eeb89f57577cb709ec7f293909039a1)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a --mc argument to `bitbake-layers show-appends` so that users can
filter appends for a specific multiconfig (instead of always showing the
default configuration)
(Bitbake rev: f4dcbf114554c829467623b5587314d16ebdf3eb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The debug in the comments was out of date. It is still useful so update the
code sample to the new code needed.
(Bitbake rev: fa2724069ea7028939d816cb5ccd0e7c1bed09a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current calculation is not correct because if tdiff is less than
1.0, it's not taken into consideration when calculating the current
pressure.
Also, make it clear that the 1.0s is the psi accumulation cycle,
which might be changed in the future. We have this cycle because it
could largely avoid the 0 result issue, that is, if the interval
between checks are too small, the result might be 0. With this
accumulation logic, which has been there but let's make it clear,
this 0 result problem could be mitigated.
(Bitbake rev: 95fa8fb5fb4d5a72e79b11d69792613bfd494e72)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fetcher allows BitBake to fetch from a Google Cloud Storage
bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and
expects it to be installed, configured, and authenticated prior
to use.
If accepted, this patch should merge in with the corresponding oe-core
patch titled "Add GCP fetcher to list of supported protocols".
Some comments on the patch:
There is also documentation for the fetcher added to the User
Manual. I'm still not completely sure about the recommends_checksum()
being set to True. As I've noted in the mailing list, it will throw
warnings if the fetcher is used in recipes without specifying a
checksum. Please let me know if this is intended behavior or if it
should be modified.
Here is how this fetcher conforms to the fetcher expectations
described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no
network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher
only downloads to the DL_DIR in the same way that other fetchers,
namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this
context
e) Yes, the fetcher output is deterministic because it is downloading
tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy
variable to test whether the Python API respected the proxy. It
appears that it did as I could see traffic passing through the
proxy. I also did some searching online and found posts indicating
that the Google Cloud Python APIs supported the classic Linux proxy
variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading
it if it does.
h) Not applicable, BitBake already knows which version it wants and
the version infomation is encoded in the filename. The fetcher has no
concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests
for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any
tools during parse time. Please correct me if I'm wrong.
(Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b)
Signed-off-by: Emil Ekmečić <eekmecic@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
process.signal_received is a list for signum and not iterable,
change a suitable method to handle sig.
(Bitbake rev: bfc53b190bd2530c2bfcea0690127d7eff620f45)
Signed-off-by: Yang Xu <yang.xu@mediatek.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runtaskdep data in siginfo files was written out with full paths to
the bb files, matching bitbake's internal "unique key" ID for recipes/tasks.
When originally implemented this made sense.
Over time, the main use for the data in siginfo files has become to match
against other siginfo files to debug changes of hash calcuations. The
recipename data is not useful for this as the siginfo filenames use PN
instead which can often be derived from the recipe filename but not always.
It is time to throw away the 'tid' data format and switch over the use a
hybrid PN form which includes the multiconfig. That can be easily stripped
off in the find_siginfo code in oe-core.
The other purpose of having a sortable dependency ID is retained and the
multiconfig needs to be included to allow the taskhashes to be processed
and calculated correctly. PN is meant to be unique between recipes, only
one would ever be built so using PN in this location is fine.
The one risk of this change is there isn't any compatibility to the old
format. I'm not convinced we should spend time complicating the code
with it. This change will change the taskhashes everywhere so the only
mixing of old and new siginfo files will be either through hash equivalence
or through users using the tool against old and new info files manually
which will give some weird output but it should be clear they're in
different formats as there would be large paths from the old files not
present in the new ones.
We have options to add backwards compatibility if some issue is found
to need that.
(Bitbake rev: 637933e2e5a59228a8d17aae4160551cab5f2f61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test_python_snippet_function_reference to not require the 'func'
flag, now that we know the real function will be returned for context
functions without the flag.
(Bitbake rev: 83f41281ec3d9b4327ffc8e2312e1fb8f53cbf02)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We previously checked for the existence of the 'func' flag to determine
if we should avoid looking up in the metadata. This was done to ensure
the user gets the function for 'def' python functions rather than their
string contents. We can sidestep the metadata lookup and check our
function context directly, instead.
(Bitbake rev: 6cac1eac51efa9a54e8457f60ea1ea0e604c50b7)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test makes sure that '${@eval()}' calls the eval builtin, even if an 'eval' variable is defined in the metadata.
(Bitbake rev: e9150447738a48f772240874b3512b08e982b19b)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids the need to hardcode a list of python builtins. This also slightly changes behavior, in a case like `${@eval("3")}`, this will ensure we always call the builtin, even if the metadata has an 'eval' variable defined.
(Bitbake rev: 9976ae50677b333d646ca0fd395468bd2301d03f)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that any change to the presence of builtins in inline
python execs will be noticed.
(Bitbake rev: ee22d3d51c60db2da97422b2be1e42239b7a2324)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a test for an issue seen long ago, to avoid regressions, where a
reference to a def'd function in the metadata would return the string
value from the metadata rather than the function in inline python.
(Bitbake rev: 9f7cb22febd557817c164e25a93f5660e9c06358)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes a few errors when the bitbake indent plugin is reloaded:
1) Define functions with "!" so that vim doens't issue a warning when
they are replaced
2) Rename GetPythonIndent -> GetBBPythonIndent to prevent potential
conflict with other plugins
(Bitbake rev: b7109acb96e416e3c537b6b51f7c1fec2ca89371)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.
(Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19)
Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using multiconfig to target baremetal pieces of the system and building
corresponding toolchains for them results in hundreds and hundreds of
"Deferring %s after %s" and "Deferred task %s now buildable".
To clean up the output and to reduce risk of missing important warnings,
convert these notice messages to debug messages.
(Bitbake rev: 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Denys Dmytriyenko <denys@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is currently hard to tell when bitbake is throttling task execution
due to system pressure changes. Add notes to the console output to make
this clearer, only generating output when the values change.
(Bitbake rev: a6056599922fb2fe3f54c5c86ac7ea604f469adc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is unclear from the server logs when parsing starts. We know that timeouts
sometimes happen when parsing but it is unclear where in the code the delays
are from. Adding this debug message to the server log should help narrow
that down.
(Bitbake rev: a5c145f436d68f090b113cfb9b82857adc95b546)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To learn more about the server timeout issues, be clear in the error
message about which command is showing the timeout. It is currently
unclear if this is the original command or a ping to the server.
(Bitbake rev: ac3cd866274f67b29eff89e393132bdabf76dbfd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"fetch2/npmsw: Add support for the new format of the shrinkwrap file"
added support for the new format shrinkwrap files but this regressed
our tests which still use the old format.
Similar to how npm handles this, support both for now until we can
migrate our tests.
(Bitbake rev: 9941b480a0e2a8b57f2ed069cd583f2784394a2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Packages like @(._.)/execute causes problems because they generate names
that are not supported by yocto
(Bitbake rev: b2e1be67d2acca27451bed59874bc1c2a7ec44a6)
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Npm is a package manager that has its own manner to handle installation of packages.
But it is not yocto friendly, for instance NPM fetch dependencies in the middle of compilation.
The shrinkwrap file changed its format over npm versions, but npm does not version
this file, so we can use it properly.
The actual changes make NPM depencies work with the actual shrinkwrap format.
(Bitbake rev: 19b9f7f0f451a636f3fdcdc1bb283ab431ede612)
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are several multiconfigs in play for example a non-multiconfig with
a task with one hash and then three multiconfigs for the same task, different
architectures but the same hash (different to the non-mc), the three mcs
will be deferred until after the non-mc task but then will all run together
and race against each other.
Change the code to re-enable deferred tasks one at a time. This way, if they do
race, they won't run in parallel against each other.
(Bitbake rev: 9523e28658ad7fb446645b590608dfac2812afd3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To support running the bitbake self-tests using pytest, add a
pytest.ini This enables these tests to be run using the following
command from the bitbake directory.
pytest
For now, testpaths only includes the bitbake selftests.
Add a note to the README.
(Bitbake rev: 79bc14003000ab439262272dae6a99961f7c14c7)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, pytest captures all stdout and exposes it using its built-in
fixtures (capsys, caplog etc), so stdout does not support getvalue().
To support running tests using both unittest and pytest, use assertLogs
to capture logging and assert on the log output instead.
(Bitbake rev: 2d28caa01bab9540d2bbaf713ae3e5c563d003f5)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git config returns an error code if user.email or user.name are not set,
so bb.process.run raises an ExecutionError, failing the test.
Improve the logic by catching the ExectionError and using this to set a
default value.
Change a direct invocation of 'git init' to use self.git_init so that
it follows the code path above.
Remove the related comment from the README now that git sets up the user
details automatically.
(Bitbake rev: 0c7079d0609a13713b890a9675abd9fba032f199)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
