Handles the log messages from the bitbake server with the specific
logger that the event originated from. This allows hierarchical logging
configurations to work as expected.
(Bitbake rev: 9624d42133e024fd044d0d089c7017ed53eed874)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Converts the default domain variable to a dictionary where the keys are
the logging domains and the values are the logging level (instead of the
debug count). This makes it easier to deal with the logging domains and
the awkward conversion from a list to a dictionary only needs to be done
once when logging is initialized. Finally, other code has been written
that already assumes this variable is a dictionary, see:
f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker")
(Bitbake rev: f32a8bc7ff7a0b0750b6934a96f5d48391b1383a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked temporarily so that the .gitmodules file can be examined.
(Bitbake rev: 3987db953e414255ce278bc25a5f6cec0f2a30c7)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PF is already captured, but it's a pain to try to parse out PN and PV,
which can be helpful to have available.
(Bitbake rev: 2885034f42a1dce7586e081a0461b8932a3593bf)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously --runonly=do_task would give a misleading error like:
ERROR: Could not find any tasks with the tasknames ['do_task'] to run
within the recipes of the taskgraphs of the targets...
The problem is that BitBake tried to find "do_do_task". So teach it to
only add the do_ prefix if it's not already there.
(Bitbake rev: 694904bde980606dc67c201da61f4fb685679b17)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for the Tinfoil cache adaptor to be bound to a specific
multiconfig and invoke the appropriate commands for that multiconfig
instead of the default. The cooker adapter now creates a cache adapter
for each multiconfig specified in BBMULTICONFIG so that each multiconfig
is present.
(Bitbake rev: d515481681dca4a0bc733c6ad8a8498a82e3d9b9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cooker had a multiconfig parameter for the findProviders() and
findBestProviders() API, but it was being ignored.
(Bitbake rev: ea0b68ac2b77676ed1c63f0ee1ae5d300f2b4696)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a multiconfig selection parameter to the findProviders command.
This allows a client to find the providers for a specific multiconfig
instead of the base configuration.
(Bitbake rev: 95bb446137734865f3fe04fe97681d90250deb11)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example a tinfoil connection edits the datastore, a subsequent
connection can be "corrupted" by those changes. By setting the parse
status of the caches as False at exit, the behaviour becomes the same
as a newly setup server as a new data store is setup.
This avoids problems in tests when BB_SERVER_TIMEOUT is set as the
server is properly reset between connections.
[YOCTO #13812]
(Bitbake rev: e66759106e21da2b34a6cdec7aa681ad2204da54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake will list many checksums for a recipe when none are
present, encouraging users to add them all to a recipe. We don't
need/want them all.
We used to show md5 and sha256 but given the concerns about md5,
switch to showing just sha256 going forward which seems like the
sensible one to standardise upon.
There will be no change to existing recipe functionality.
(Bitbake rev: 47f0c849ed13ba554d9523b926d92405e8251702)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The update codepath would trigger:
Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment
Fix this so the code functions as intended in both fetch and update cases.
[YOCTO #13798]
(Bitbake rev: 16c4e930ff37ea6eac2ac0cb2197908ce3a1cc53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Documentation says:
"path_spec": A specific directory in which to checkout the specified
svn module.
but existing svn fetcher uses "module" always as path of checked out
svn-module, regardless whether path_spec was given or not.
(Bitbake rev: 75223644ab9bc94fc268f1bab775e66c4188f279)
Signed-off-by: Jens Rehsack <sno@NetBSD.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It printed duplicated ones when there are multiple similar recipes in
differrent layers, for example, if python-lockfile in different layers,
and there is no python3-lockfile:
$ bitbake python3-lockfile
ERROR: Nothing PROVIDES 'python3-lockfile'. Close matches:
python-lockfile
python-lockfile
python3-aiofiles
Remove the duplicated ones to fix the problem.
(Bitbake rev: 5612192cec9f467e2ab5a86482cb34876d198bc6)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing, reset the loghandler when finished, else the messages
can be misleading.
(Bitbake rev: 7af80cd1dd577b05d39a3cc5d5c547a2549e39df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While usually a programming error, the behavior can cause a Parser
instance to eventually gobble up a significant amount of memory,
greatly affecting system performance. Try to avoid getting into
that situation and alert the user about what they attempted to do.
(Bitbake rev: 01bf0912eef5700d61c6e3c9138cb4b6825ee782)
Signed-off-by: Kyle Russell <bkylerussell@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mark any keys used to cache the srcrevs for a recipe as "dontcache" if
BB_DONT_CACHE is set for the recipe. Remove any such keys upon the
next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will
make sure the srcrev is updated as expected if ${AUTOREV} is used.
(Bitbake rev: ba093a38539960e645e994a66ed7872a604c00a9)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This seems to have been broken for a very long time. Now it also works
regardless of BB_SRCREV_POLICY.
(Bitbake rev: ffd663a8e07e2e39e8ca2d2493f4f98037c5f9e4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This matches the other bb.command.Command* events and without it,
running `bitbake --revisions-changed` will hang indefinitely if there
are changed revisions.
(Bitbake rev: 40520d229c8ea51ee9784184ab5d13a82dd1eb61)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing recipes its apparent the memory usage of bitbake rises linearly
with number of recipes parsed. It shouldn't.
Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
dependency information left behind in siggen was the culprit. Add a new
method to allow us to drop this information. We don't need it after the recipe
has been parsed and hashes calculated (at runtime its different but only the
currently executing task would be in memory).
This should give signficant memory usage improvements for bitbake and that
in turn should help speed on more constrained systems, as well as when used in
multiconfig environments.
(Bitbake rev: 5d98d8e39bba42f458532b1eef3619f2321d8a2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If locked sigs are in use this function makes little sense, need to
avoid generating mismatch warnings.
(Bitbake rev: 27ad9c1d468fba858a4adeb56b605227b415ae0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add unihash cache of values to speed up cache lookup.
This avoids the overhead of the disk based check functions.
(Bitbake rev: 5c9cc45b60904a1c355db9bf9c4495f1b25aca37)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cache can grow huge since any previously used hash is
retained in the cache. This change moves to use one hash per task
which improves the speed of the functions considerably. Currently
performance is an issue, as are very large cache files and cache
load time.
By moving to a single hash per task, the shorted filename as a key
is no longer usable as the same recipe has multiple variants for
the same filename so this has to change.
(Bitbake rev: ed764e7fcf04b6d0ba6b4cac7415b1ee8f492865)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed the deprecated methods as it will only cause problems later on,
and since warn() just calls warning(), it shouldn't change anything
(Bitbake rev: a194f275235f22411cb2368f06a44f61ceb6a0f3)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit adds a new npmsw fetcher that fetches every npm dependencies
described in a npm shrinkwrap file:
https://docs.npmjs.com/files/shrinkwrap.json.html
The main package must be fetched separately:
SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \
npmsw://${THISDIR}/npm-shrinkwrap.json"
Since a separation has been created between the package and its
dependencies, the package can also be fetched with a non npm fetcher
without impacting the general behavior:
SRC_URI = "git://github.com/foo/bar.git;protocol=https \
npmsw://${THISDIR}/npm-shrinkwrap.json"
(Bitbake rev: f5223be54450bf20e0bfbd53b372a7748a44b475)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit refactors the npm fetcher to improve some points and fix
others:
- The big change is that the fetcher is only fetching the package
source and no more the dependencies. Thus the npm fetcher act as the
other fetchers e.g git, wget. The dependencies will be handled later.
- The fetcher only resolves the url of the package using 'npm view' and
then forwards it to a proxy fetcher.
- This commit also fixes a lot of issues with the package names (exotic
characters, scoped packages) which were badly handled.
- The validation files - lockdown.json and npm-shrinkwrap.json - are no
longer used by the fetcher. Instead, the downloaded tarball is
verified with the 'integrity' and 'shasum' provided in the 'npm view'
of the package [1][2].
1: https://docs.npmjs.com/files/package-lock.json#integrity
2: https://www.w3.org/TR/SRI
(Bitbake rev: 0f451cdc43130d503ada53ed1b4fc5a24943f6ef)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using a download filename with characters which can be interpreted
by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the
filename fixes the issue.
(Bitbake rev: ed652dce5200161068eccdbfaaaefde33136eb09)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the done condition to a
proxy fetcher.
(Bitbake rev: ee3a2545e99e6e99559a72bcda64797ae674ec71)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "try_mirrors" functions to
a proxy fetcher.
(Bitbake rev: 462c9a2b368a1720da276310b1d5d0423b7cefea)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "verify_donestamp" and
"update_stamp" functions to a proxy fetcher.
(Bitbake rev: f7612c0704b4252bba5157ce9a94d8888c6d0760)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit enables the "sha1", "sha384" and "sha512" hash functions in
the supported checksum list. This allows to use more SRC_URI checksums
functions for a url:
SRC_URI[sha1sum] = "..."
SRC_URI[sha384sum] = "..."
SRC_URI[sha512sum] = "..."
The npm fetcher needs this to support subresource integrity:
https://www.w3.org/TR/SRI/
(Bitbake rev: cd80a646aa841b71e68282bb8d11194abb5df0e4)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit refactors the way checksums are verified to be more generic.
The support of new hash functions is now limited to the update of the
CHECKSUM_LIST variable.
(Bitbake rev: debd9eeaf5638755d8956b2d65b904fe02826966)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently these tests rely upon multiple uptream webservers which may change
or be unavailable. Add local copies of the test data, copy the httpserver
from OE-Core (used for testing there) and run these tests against a local
server instead.
(Bitbake rev: d5a4a352723258b4d499d3a51f340109c4f36f60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To avoid races, SQLTable::__setitem__ needs an exclusive lock for the
entire transaction, not just the INSERT/UPDATE part.
(Bitbake rev: feb43e7c30f5bfab75d718896c45df621810d06f)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From a SRCREV caching point of view, there is no reason to treat the
following upstreams as different:
SRC_URI = "git://github.com/file/file.git"
SRC_URI = "git://github.com//file/file.git"
(Bitbake rev: 425e21c14955dd38868c6e97637df3bbe0f89fac)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior to this change, two different recipes pulling from the same
exact repo could get a different SRCREV during a single parse session.
This was originally observed using git. For git at least, it still
allows recipes to pull from the same repo, but with different branches
or tags, since the form of the srcrev cache key for git is:
"git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name]
Where the 'unresolvedrev' part is the branch or tag name.
(Bitbake rev: 6c938e6fd29beebe09b32be839dae008fe6491d2)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given a start expression, bb.utils.get_referenced_vars returns the
referenced variable names in a quasi-BFS order (variables within the
same level are ordered aribitrarily).
For example, given an empty data store:
bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d)
returns either ["A", "B", "C"], ["A", "C", "B"], or another
permutation.
If we then set A = "${F} ${G}", then the same call will return a
permutation of [A, B, C] concatenated with a permutation of [F, G].
This method is like a version of d.expandWithRefs().references that
gives some insight into the depth of variable references.
(Bitbake rev: 076eb5453ca35b8b75b8270efb989d5208095b27)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should give performance improvements to functions using this list of
tasks (sets are used for most of the other code for this reason, not sure
why this wasn't a set in the first place).
(Bitbake rev: f5daef68703481a3c243dfecc7de404e6ebfdbb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating a new progress bar (using BBProgress), a colon was
appended to the supplied message. However, when updating the message,
no colon was appended.
Change this so that the colon is instead part of the widgets that make
up the progress bar so that it does not matter when and how the
message is updated, it always displays the same.
(Bitbake rev: 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of the hash equivalence server, the progress bar
for "Checking sstate mirror object availability" is shown repeatedly
while the tasks are being executed. If the footer is not hidden then,
it will be moved up one line every time, creating a messy interface.
(Bitbake rev: 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When there are many watched files, keeping track of them using lists
is suboptimal. Using sets improves the performance considerably.
(Bitbake rev: 1e96df260e47d160dbd36bfc92c31ef06266f662)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
