The update codepath would trigger:
Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment
Fix this so the code functions as intended in both fetch and update cases.
[YOCTO #13798]
(Bitbake rev: 16c4e930ff37ea6eac2ac0cb2197908ce3a1cc53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When bitbake-layers fetch-layerindex clones the repositories, these are
full clones. Allow the user to specify '-s' and do shallow clones
instead for faster downloads.
(Bitbake rev: a0c8b27675a590d9deeb3cbc462c0eb0e113cf3b)
Signed-off-by: Jan-Simon Moeller <dl9pf@gmx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Documentation says:
"path_spec": A specific directory in which to checkout the specified
svn module.
but existing svn fetcher uses "module" always as path of checked out
svn-module, regardless whether path_spec was given or not.
(Bitbake rev: 75223644ab9bc94fc268f1bab775e66c4188f279)
Signed-off-by: Jens Rehsack <sno@NetBSD.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It printed duplicated ones when there are multiple similar recipes in
differrent layers, for example, if python-lockfile in different layers,
and there is no python3-lockfile:
$ bitbake python3-lockfile
ERROR: Nothing PROVIDES 'python3-lockfile'. Close matches:
python-lockfile
python-lockfile
python3-aiofiles
Remove the duplicated ones to fix the problem.
(Bitbake rev: 5612192cec9f467e2ab5a86482cb34876d198bc6)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing, reset the loghandler when finished, else the messages
can be misleading.
(Bitbake rev: 7af80cd1dd577b05d39a3cc5d5c547a2549e39df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While usually a programming error, the behavior can cause a Parser
instance to eventually gobble up a significant amount of memory,
greatly affecting system performance. Try to avoid getting into
that situation and alert the user about what they attempted to do.
(Bitbake rev: 01bf0912eef5700d61c6e3c9138cb4b6825ee782)
Signed-off-by: Kyle Russell <bkylerussell@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mark any keys used to cache the srcrevs for a recipe as "dontcache" if
BB_DONT_CACHE is set for the recipe. Remove any such keys upon the
next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will
make sure the srcrev is updated as expected if ${AUTOREV} is used.
(Bitbake rev: ba093a38539960e645e994a66ed7872a604c00a9)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This seems to have been broken for a very long time. Now it also works
regardless of BB_SRCREV_POLICY.
(Bitbake rev: ffd663a8e07e2e39e8ca2d2493f4f98037c5f9e4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This matches the other bb.command.Command* events and without it,
running `bitbake --revisions-changed` will hang indefinitely if there
are changed revisions.
(Bitbake rev: 40520d229c8ea51ee9784184ab5d13a82dd1eb61)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing recipes its apparent the memory usage of bitbake rises linearly
with number of recipes parsed. It shouldn't.
Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
dependency information left behind in siggen was the culprit. Add a new
method to allow us to drop this information. We don't need it after the recipe
has been parsed and hashes calculated (at runtime its different but only the
currently executing task would be in memory).
This should give signficant memory usage improvements for bitbake and that
in turn should help speed on more constrained systems, as well as when used in
multiconfig environments.
(Bitbake rev: 5d98d8e39bba42f458532b1eef3619f2321d8a2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When currently specified, the branch is used to verify the versioning of
the meta layer, but the master branch is checked out. This change
allows for the branch to be specified. Now it is easy to specify all
of the meta layers being added are of the same version, without having
to do it in each individual git tree. Also, it will error if there are
branches without a matching version. Finally, this allows for meta
layer git trees without a master branch.
(Bitbake rev: 4ec49f42f327068890e7aad8553d7f282e2ffaa1)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Minor rework of the patch to use the layerBranch actual_branch since
the layerindex referenced branch may be different then the overall
release branch.
Also adjust the patch to use the default git checkout branch instead of
master if no branch was specified. (Some repositories don't have a
master branch.)
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If locked sigs are in use this function makes little sense, need to
avoid generating mismatch warnings.
(Bitbake rev: 27ad9c1d468fba858a4adeb56b605227b415ae0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add unihash cache of values to speed up cache lookup.
This avoids the overhead of the disk based check functions.
(Bitbake rev: 5c9cc45b60904a1c355db9bf9c4495f1b25aca37)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cache can grow huge since any previously used hash is
retained in the cache. This change moves to use one hash per task
which improves the speed of the functions considerably. Currently
performance is an issue, as are very large cache files and cache
load time.
By moving to a single hash per task, the shorted filename as a key
is no longer usable as the same recipe has multiple variants for
the same filename so this has to change.
(Bitbake rev: ed764e7fcf04b6d0ba6b4cac7415b1ee8f492865)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed the deprecated methods as it will only cause problems later on,
and since warn() just calls warning(), it shouldn't change anything
(Bitbake rev: a194f275235f22411cb2368f06a44f61ceb6a0f3)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit adds a new npmsw fetcher that fetches every npm dependencies
described in a npm shrinkwrap file:
https://docs.npmjs.com/files/shrinkwrap.json.html
The main package must be fetched separately:
SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \
npmsw://${THISDIR}/npm-shrinkwrap.json"
Since a separation has been created between the package and its
dependencies, the package can also be fetched with a non npm fetcher
without impacting the general behavior:
SRC_URI = "git://github.com/foo/bar.git;protocol=https \
npmsw://${THISDIR}/npm-shrinkwrap.json"
(Bitbake rev: f5223be54450bf20e0bfbd53b372a7748a44b475)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit refactors the npm fetcher to improve some points and fix
others:
- The big change is that the fetcher is only fetching the package
source and no more the dependencies. Thus the npm fetcher act as the
other fetchers e.g git, wget. The dependencies will be handled later.
- The fetcher only resolves the url of the package using 'npm view' and
then forwards it to a proxy fetcher.
- This commit also fixes a lot of issues with the package names (exotic
characters, scoped packages) which were badly handled.
- The validation files - lockdown.json and npm-shrinkwrap.json - are no
longer used by the fetcher. Instead, the downloaded tarball is
verified with the 'integrity' and 'shasum' provided in the 'npm view'
of the package [1][2].
1: https://docs.npmjs.com/files/package-lock.json#integrity
2: https://www.w3.org/TR/SRI
(Bitbake rev: 0f451cdc43130d503ada53ed1b4fc5a24943f6ef)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using a download filename with characters which can be interpreted
by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the
filename fixes the issue.
(Bitbake rev: ed652dce5200161068eccdbfaaaefde33136eb09)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the done condition to a
proxy fetcher.
(Bitbake rev: ee3a2545e99e6e99559a72bcda64797ae674ec71)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "try_mirrors" functions to
a proxy fetcher.
(Bitbake rev: 462c9a2b368a1720da276310b1d5d0423b7cefea)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is necessary to introduce proxy fetchers and do not modify
the behavior of existing fetchers.
This commit allows fetchers to forwards the "verify_donestamp" and
"update_stamp" functions to a proxy fetcher.
(Bitbake rev: f7612c0704b4252bba5157ce9a94d8888c6d0760)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit enables the "sha1", "sha384" and "sha512" hash functions in
the supported checksum list. This allows to use more SRC_URI checksums
functions for a url:
SRC_URI[sha1sum] = "..."
SRC_URI[sha384sum] = "..."
SRC_URI[sha512sum] = "..."
The npm fetcher needs this to support subresource integrity:
https://www.w3.org/TR/SRI/
(Bitbake rev: cd80a646aa841b71e68282bb8d11194abb5df0e4)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit refactors the way checksums are verified to be more generic.
The support of new hash functions is now limited to the update of the
CHECKSUM_LIST variable.
(Bitbake rev: debd9eeaf5638755d8956b2d65b904fe02826966)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently these tests rely upon multiple uptream webservers which may change
or be unavailable. Add local copies of the test data, copy the httpserver
from OE-Core (used for testing there) and run these tests against a local
server instead.
(Bitbake rev: d5a4a352723258b4d499d3a51f340109c4f36f60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To avoid races, SQLTable::__setitem__ needs an exclusive lock for the
entire transaction, not just the INSERT/UPDATE part.
(Bitbake rev: feb43e7c30f5bfab75d718896c45df621810d06f)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From a SRCREV caching point of view, there is no reason to treat the
following upstreams as different:
SRC_URI = "git://github.com/file/file.git"
SRC_URI = "git://github.com//file/file.git"
(Bitbake rev: 425e21c14955dd38868c6e97637df3bbe0f89fac)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior to this change, two different recipes pulling from the same
exact repo could get a different SRCREV during a single parse session.
This was originally observed using git. For git at least, it still
allows recipes to pull from the same repo, but with different branches
or tags, since the form of the srcrev cache key for git is:
"git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name]
Where the 'unresolvedrev' part is the branch or tag name.
(Bitbake rev: 6c938e6fd29beebe09b32be839dae008fe6491d2)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given a start expression, bb.utils.get_referenced_vars returns the
referenced variable names in a quasi-BFS order (variables within the
same level are ordered aribitrarily).
For example, given an empty data store:
bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d)
returns either ["A", "B", "C"], ["A", "C", "B"], or another
permutation.
If we then set A = "${F} ${G}", then the same call will return a
permutation of [A, B, C] concatenated with a permutation of [F, G].
This method is like a version of d.expandWithRefs().references that
gives some insight into the depth of variable references.
(Bitbake rev: 076eb5453ca35b8b75b8270efb989d5208095b27)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should give performance improvements to functions using this list of
tasks (sets are used for most of the other code for this reason, not sure
why this wasn't a set in the first place).
(Bitbake rev: f5daef68703481a3c243dfecc7de404e6ebfdbb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating a new progress bar (using BBProgress), a colon was
appended to the supplied message. However, when updating the message,
no colon was appended.
Change this so that the colon is instead part of the widgets that make
up the progress bar so that it does not matter when and how the
message is updated, it always displays the same.
(Bitbake rev: 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of the hash equivalence server, the progress bar
for "Checking sstate mirror object availability" is shown repeatedly
while the tasks are being executed. If the footer is not hidden then,
it will be moved up one line every time, creating a messy interface.
(Bitbake rev: 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When there are many watched files, keeping track of them using lists
is suboptimal. Using sets improves the performance considerably.
(Bitbake rev: 1e96df260e47d160dbd36bfc92c31ef06266f662)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These messages spam the logs for no good reason, they were useful for debugging
a particular problem long ago but are distracting noise now. Disable them.
(Bitbake rev: 1a9247c468cf09da60e5d396ccb81e950841c99e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hack the hashserv to allow extra data to be injected into the hashserv
method. This allows OE-Core to handle cases where there are multiple
sstate objects for the same taskhash, e.g. native/cross objects based
upon BUILD_ARCH or the host distro (when uninative isn't used).
This has been tested and proven to be very effective. We will likely
rework the code to improve how this is handled but for now this
improves automated builds until we can get to that refactoring and
more invasive changes.
(Bitbake rev: 0a09b0fa03d1afc08037964dc63a18ef7cff9c78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A corner case was identified where tasks with valid stamps from previous
builds need to be accounted for in the new sanity test in the migration
code. Add a variable to track such completed tasks to ensure the sanity
test works correctly.
(Bitbake rev: d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen a number of mystery failures where task B would run despite
task A, its dependency not having run. An example would be do_compile
when do_unpack didn't run.
This has been tracked down to this code block. In theory it shouldn't
trigger however it can and has due to bugs elsewhere. When it does, it
causes significant weird failures and possible build corruption.
Change the code to abort the build. This avoids any chance of corruption
and should ensure the issues get reported, putting an end to the weird
build failures.
There may be some cases where this triggers and it shouldn't, we'll work
through those as they arise and are identified.
(Bitbake rev: 7a92b7f58ab187eddfe550bd6fb687240c7b11bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Regardless of whether we remapped the hash on the server or not, we need
to have bitbake work as if we did as we need to match how the stamp files
look.
This change resolves build failures where tasks were rerunning when they
shouldn't.
(Bitbake rev: 40928f6991436cf687821015324483b205abfcb1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.
Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"
(Bitbake rev: 923aff060d8aba8456979c35b16d300ba7c13ff9)
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to ensure the correct working directory is used for this
tar command to succeed.
(Bitbake rev: e1a2a0e29a75dead3426ae083387181ad52da91f)
Signed-off-by: Christian Herzig <cherzig@gauselmann.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have bb.debug(2, xxx) messages in cooker which are useful for debugging
but have really bad effects on performance, 640,000 calls on recent profile
graphs taking tens of seconds.
Rather than commenting out debug which can be useful for debugging, don't
create events for debug log messages from cooker which would never be seen.
We already stop the messages hitting the IPC but this avoids the overhead
of creating the log messages too, which has been shown to be signficiant
on the profiles. This allows the code to perform whilst allowing debug
messages to be availble when wanted/enabled.
(Bitbake rev: f04cd931091fb0508badf3e002d70a6952700495)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This doesn't save much time but does make the profile counts for the
function more accurate which is in itself useful.
(Bitbake rev: d446fa89d206fbc6d098215163c968ea5a8cf4a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
