If no eligible providers are found then an integer zero is returned.
This causes the following error, in two possible places in taskdata.py,
when the return value is used in a list comprehension:
[snip]
eligible = [p for p in eligible if not p in self.failed_fns]
TypeError: 'int' object is not iterable
[\snip]
Fix by returning the variable eligible itself, of type list.
(Bitbake rev: 217c4b436b588a6a47aeaddf61531711ad3fca67)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a Dockerfile for build the reference hash equivalence server in a
container
(Bitbake rev: d9a0a88db5888039bfbb6ce5129b43350f79b1c1)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows metadata to rely upon BB_DEFAULT_UMASK.
(Bitbake rev: 969ac64adab236ce2d5196bcc294005a497913ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently each task has to have a umask specified individually. This
is leading to determinism issues since it is easy to miss specifying
this for an extra task.
Add support for specifing the default task umask globally which
simplifies the problem.
(Bitbake rev: 3e664599fd54a8a37ce587022fcbce5ca26f2ed3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.
This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.
See [YOCTO #13071] for detailed bug information.
Add mc target name as a prefix to event handler name so there won't be
two different handlers with the same name. Add internal __BBHANDLERS_MC
variable to have the handlers lists per machine.
(Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When adding a layer, parse error can occur, raising BBHandledException.
Catch this and error, aborting the layer add to meet user expectations.
[YOCTO #14054]
(Bitbake rev: ceddb5b3d229b83c172656053cd29aeb521fcce0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
From tinfoil, if you edit bblayers.conf and break it, then call
parseConfiguration (e.g. by adding a bad layer with bitbake-layers),
the system doens't show any parse error yet it should.
Add in a call to the updateCache function so that things really
are reparsed when requested.
Partially fixes [YOCTO #14054]
(Bitbake rev: e655f9361b9c3b77906b8e06b5cc76bc5180640e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If shell function name starts with 'python' or 'fakeroot' parser wrongly
assumes it's python/fakeroot function.
[YOCTO #14204]
Use regex lookahead assertions to check if 'python' expression is
followed by whitespace or '(' and if 'fakeroot' is followed by
whitespace.
(Bitbake rev: b07b226d5d1b3acd3f76d8365bc8002293365999)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get-outhash message can be sent via the get_outhash client method.
This works in a similar way to the get message but looks up a db entry
by outhash rather than by taskhash. It is intended to be used as a
read-only form of the report message.
As both handle_get_outhash and handle_report use the same query string
we can factor this out.
(Bitbake rev: dc19606ada29a4d8afde4fcecd8ec986b47b867e)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new get-outhash message to perform a read-only query against an
upstream server (if present) when a reported taskhash/outhash
combination is not found in the current database. If a matching entry is
found upstream it is copied into the current database so it can be found
by future queries.
(Bitbake rev: 2be4f7f0d2ccb09917398289e8140e1467e84bb2)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Short form arguments are added for convenience.
(Bitbake rev: 921199a4923ce383b27e23c9b7e34eb785c8bae3)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hashserv server already implements support for pulling hash data
from another "upstream" server. Add the -u/--upstream argument to the
bitbake-hashserv app to expose this functionality to users.
(Bitbake rev: 8de510f1de35e581bcd5858edf23619c6a4cf923)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The -r/--readonly argument is added to the bitbake-hashserv app. If this
argument is given then clients may only perform read operations against
the server. The read-only mode is implemented by simply not installing
handlers for write operations, this keeps the permission model simple
and reduces the risk of accidentally allowing write operations.
As a sqlite database can be safely opened by multiple processes in
parallel, it's possible to start two hashserv instances against a single
database if you wish to export both a read-only port and a read-write
port.
(Bitbake rev: 492bb02eb0e071c792407ac3113f92492da1a9cc)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing occasional issues where builds fail as events were written from the
worker children in the form <event>partial data<event>full event</event>.
This causes failures as bitbake server can't parse that and exits. This could
be due to short writes to the worker event pipe which we weren't checking. Check
this and loop accordingly. Also add some asserts to detect other potential causes.
Thanks to Joshua Watt for help in spotting the issue.
[YOCTO #14181]
(Bitbake rev: a9451746a4bd7ccedf4c72cd03ad4ff0ab0143aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server returns an empty response ("null" in json), this cannot
be iterated to check for the presence of the "chunk-stream" key.
(Bitbake rev: bf75370bcd6d02ed08cd959eec6190196b792515)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect function was previously removed from the hashserv Client
class but the bitbake-hashclient app was not updated. The client is
connected during hashserv.create_client() anyway so not separate connect
call is needed.
(Bitbake rev: 99bdb236bceeffa0083a0fa529280b217c1d310d)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.
| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot
(Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.
[YOCTO #14191]
(Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c)
Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To exercise the previous patch, add a test case.
(Bitbake rev: efdf6d5c4cd8155bc54dc667346f58dccb138e07)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
_param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws
an exception if a lone semicolon is used as the value doesn't split into two
items.
Fix by checking that the result of the first split has content.
(Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As reported by Katu Txakur, the output depends on the module path
set so this needs to be accounted for in localfile.
(Bitbake rev: 9861ed37bb1c5d09c3b4852d2a252e3f3c86ab14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the usage of enterprise proxy, the user-agent defined are
too old and refused by proxy configuration. Updating to something
more modern is desirable.
(Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400)
Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com>
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The usehead url parameter for git repositories causes bitbake to use
whatever commit the repository HEAD is pointing to if the repository
happens to have the name 'default'. This is the default name so in many
cases it works just fine, but if a different name is specified with the
url parameter 'name=newName' then it will fail to parse the recipe with
an error along the lines of:
ERROR: ExpansionError during parsing /path/to/my/recipe.bb
Traceback (most recent call last):
File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init:
> ud.setup_revisions(d)
File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions:
for name in self.names:
> self.revisions[name] = srcrev_internal_helper(self, d, name)
File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'):
if srcrev == "AUTOINC":
> srcrev = ud.method.latest_revision(ud, d, name)
File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud, d, name)
return rev
File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'):
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
> (ud.unresolvedrev[name], ud.host+ud.path))
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo
Let's fix this by setting the unresolved rev of _all_ repository names
to 'HEAD' when the usehead url parameter is specified. Update the
currently failing test, test_local_gitfetch_usehead_withname, to now
expect success.
This change preserves existing behavior that allows usehead to be
overridden by a valid looking revision if one happens to be specified
instead of AUTOREV.
(Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c)
Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add tests for fetching a URL with the usehead parameter set and a
non-default name set. We currently expect the local version of this test
to fail since there is a bug in the usehead implementation that breaks
for non-default names.
(Bitbake rev: a2345110f217fac429f6ec15f699c87c39531e7c)
Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The test `test_gitfetch_usehead' exercises a way to override the usehead
feature by setting SRCREV. It may not be obvious that this is what is
being exercised here so let's add some comments to document the expected
behavior.
(Bitbake rev: 1cd998c19101e3b093e81c126b3048c5d56058b0)
Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the local only usehead test to the FetcherLocalTest class so it
will be run when BB_SKIP_NETTESTS=yes since it does not require
network access.
Rename the usehead tests to better match the new organization.
(Bitbake rev: 137cfa13d5319bc91c3e5fe6c7062cb8c8484d64)
Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A list of functions that now has a docstring.
* vercmp_string
* explode_dep_versions
* prunedir
* prune_suffix
* to_boolean
* contains_any
* export_proxies
See [YOCTO #9725] for details.
(Bitbake rev: b61ba4a18693a9e553d2a93161feb0bcc1c82384)
Signed-off-by: Milan Shah <mshah@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The provided packages by a skipped recipe are supposed to be listed in
SkippedPackage.rprovides, which is used when generating a meaningful
error message when a build fails because of a skipped package.
Previously this variable only contained the contents of ${RPROVIDES}.
However, most recipes don't define RPROVIDES, they define
RPROVIDES_<pkg> for each package they provide. Additionally, the recipe
provides the packages in PACKAGES without them being included in
${RPROVIDES}.
Before this change, having a runtime dependency on a skipped non-recipe
package would result in a build error stating that the build failed
because the package was skipped, but without providing any reason for
why it was skipped.
(Bitbake rev: efd026c26a377b826a49b945a8212bf7de8a480a)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get_hash() function is used to decide if the base configuration has changed
and hence whether a reparse is required. The vardepvalue flag's value was not
expanded but it is often used in contexts like:
METADATA_REVISION = "${@base_detect_revision(d)}"
METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"
which in it's unexpanded form means reparsing doesn't happen when it should
as the data appears unchanged. Update get_hash to expand the values of
vardepvalue so reparsing works as expected. This avoids basehash mismatch
errors such as the one recently caused by using METADATA_REVISION in poky.conf's
DISTRO_VERSION variable. The issue there could be exposed by a recipe using
DISTRO_VERSION with the sequence:
bitbake os-release
<change the revision of the metadata with a dummy commit>
bitbake os-release -C install
which was caused because METADATA_REVISION changed but the metadata didn't reparse.
(Bitbake rev: 26ccf1575aef2d6e2d7717d3bd10b1ed0d5a777d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If using a YAML file in BB_LOGCONFIG, the unsafe loader is used and this
causes a runtime warning:
YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated,
as the default Loader is unsafe. Please read https://msg.pyyaml.org/load
for full details.
As log configuration YAML is relatively simple we can just use
safe_load().
(Bitbake rev: 71aaac9efa69abbf6c27d174e0862644cbf674ef)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes the bug were long paths would break Unix domain socket clients
(for real this time; the previous attempt was missing os.path.basename).
Adds some tests to prevent regressions
(Bitbake rev: 77790e3656048eff5cb1a086c727d86d32773b68)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb.
Unfortunately this caused problems where incorrect paths were found in paths
and files were not being found. This was reported on the mailing list but no
invetigation or resolution was found. Revert until we can figure out
what was wrong.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Restores a fix for unix domain socket path length limits when using the
synchronous hash equivalence client that was accidentally removed when
the async client was added.
Unfortunately, it's much more difficult to fix the same problem when
using the async client directly due to the interaction of chdir() and
async code, but this will at least restore the old behavior in the
synchronous case.
(Bitbake rev: 53e85022a8b1c8f407c9418260c59beffb96f0f9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit [991f92b4d bb.ui: delete __init__.py to make bb.ui a
namespace package] caused `bitbake -h' failed
...
$ bitbake -h
Traceback (most recent call last):
File "/buildarea/raid5/hjia/community/poky/bitbake/bin/bitbake", line 35, in <module>
sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/cookerdata.py", line 27, in __init__
self.options, targets = self.parseCommandLine(argv or sys.argv)
File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/main.py", line 297, in parseCommandLine
options, targets = parser.parse_args(argv)
File "/usr/lib64/python3.6/optparse.py", line 1387, in parse_args
stop = self._process_args(largs, rargs, values)
File "/usr/lib64/python3.6/optparse.py", line 1431, in _process_args
self._process_short_opts(rargs, values)
File "/usr/lib64/python3.6/optparse.py", line 1536, in _process_short_opts
option.process(opt, value, values, self)
File "/usr/lib64/python3.6/optparse.py", line 785, in process
self.action, self.dest, opt, value, values, parser)
File "/usr/lib64/python3.6/optparse.py", line 807, in take_action
parser.print_help()
File "/usr/lib64/python3.6/optparse.py", line 1647, in print_help
file.write(self.format_help())
File "/usr/lib64/python3.6/optparse.py", line 1635, in format_help
result.append(self.format_option_help(formatter))
File "/usr/lib64/python3.6/optparse.py", line 1615, in format_option_help
result.append(OptionContainer.format_option_help(self, formatter))
File "/usr/lib64/python3.6/optparse.py", line 1061, in format_option_help
result.append(formatter.format_option(option))
File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/main.py", line 54, in format_option
valid_uis = list_extension_modules(bb.ui, 'main')
File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/main.py", line 72, in list_extension_modules
pkgdir = os.path.dirname(pkg.__file__)
AttributeError: module 'bb.ui' has no attribute '__file__'
...
This reverts commit 991f92b4d15b0571b6a540964e5216d1b9728539
(Bitbake rev: 22b8c53205f8915b33d1e0ad6a666dcacc01491d)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From sphinx-build man page:
-j N build in parallel with N processes where possible
(special value "auto" will set N to cpu-count)
(Bitbake rev: e0d08058d849cb28cd70ade0cb35863a91019353)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the selected version if not available, bitbake will happily attempt
to build something else. This should be a loud warning not a small note.
(Bitbake rev: 78cd63285713fde59506eb2e71a7b7ee59a594ff)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
try_mirror_url() creates a symlink named as the original file to make
everything look like files specified in SRC_URI were downloaded from
their original location. The link is however created as an absolute
reference, this makes DL_DIR non-relocatable. This also causes issues
with the Isar project since it bind mounts DL_DIR to /downloads to
perform some of its build tasks in a chrooted environment (rendering
all symbolic links from DL_DIR invalid). Modify ensure_symlink() to
take an optional "relative" argument and have that function use
os.path.relpath() to produce a relative symlink.
(Bitbake rev: 481e66ea8fc2fc91903127d66b0f1b0fe86baedb)
Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
Cc: Chris Larson <Chris_Larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removes the keys() function from the MulticonfigCache. This appears to
be a leftover from before the class inherited from collections.Mapping,
is now unnecessary, and was outright incorrect.
(Bitbake rev: 5f37b6d2829fcac1f16602d9697f8bfbcb65ff62)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for an upstream server to be specified. The upstream server
will be queried for equivalent hashes whenever a miss is found in the
local server. If the server returns a match, it is merged into the
local database. In order to keep the get stream queries as fast as
possible since they are the critical path when bitbake is preparing the
run queue, missing tasks provided by the server are not immediately
pulled from the upstream server, but instead are put into a queue to be
backfilled by a worker task later.
(Bitbake rev: e6d6c0b39393e9bdf378c1eba141f815e26b724b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for create a client that operates using Python asynchronous
I/O.
(Bitbake rev: cf9bc0310b0092bf52b61057405aeb51c86ba137)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With introduction of python3targetconfig class in core this is no longer
needed.
(Bitbake rev: 47b64cfacd7c498ef9ed5486d117f2d69a39f225)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to the calls to sys.stderr.write() and sys.exit() in exception
handling in case of sphinx_rtd_theme not being installed, the
following exception is raised by Python due to the fact that sys
module not being imported.
(Bitbake rev: 5ddf0e5bee0de59d07295fc5693e20b1a0380fde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
