The intent of the code was to catch one kind of error, it was actually swallowing
all exceptions and looping indefinitely. Fix it to work as intended.
This explains some mystery hangs we've been seeing.
(Bitbake rev: d89358c7b8aa69f12b8c384c4fdb493782633494)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are typos in the fixture files that need to be fixed to correctly
check out the 'thud' branch.
[YOCTO #13064]
(Bitbake rev: 03a2bc00a070794452aa1df0fd75a338bdccc47d)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the bitbake server recieved multiple connections, it currently closes
ones it can't handle (while its dealing with another). This is rather
antisocial behaviour which causes clients to quickly run through their
retries and abort.
Instead, queue any other connections until the current one is closed. This
way the client can decide when it wants to stop waiting for the server. If the
client is gone by the time we handle it, we handle that gracefully.
This also fixes a number of bugs in the connection handling where connections
which did drop early were badly handled causing tracebacks in the logs.
Also, handle queue incomming connections in a loop to ensure that the main
client handling doesn't starve that piece of the system.
This code was stress tested by running 50 connection attempts in parallel at
once, ensuring the code correctly handled them.
(Bitbake rev: 220193dc38c4e78cb7cf36132a3a5b499a35bc8a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we set a timeout for the socket, it can return EWOULDBLOCK
if a signal or other event happens to wake up even if we don't timeout.
If this happens, retry the connection, else we simply see it quickly
loop through the retries and abort the connection in a very short
interval.
(Bitbake rev: c2000651a200530ba08161207ade5eea8bbeec43)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current value of 2 seconds has shown to be short in
wider testing.
(Bitbake rev: 469cc520593ba52775a373faad03072b7af05dba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing hangs in oe-selftest where server startup and shutdown are
racing. The assumption was a connect would timeout however no timeout is
set which can leave processes hanging. Set a short timeout for
the connection to avoid this.
(Bitbake rev: 833d95f538c007c27c6eb8d8f2f97094dc2b1a41)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake
server_connection = bb.server.process.connectProcessServer(sockname, featureset)
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 490, in connectProcessServer
if command_chan_recv:
UnboundLocalError: local variable 'command_chan_recv' referenced before assignment
(Bitbake rev: 257ed88590883f46beec0164749e45733e67954e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible the read may return a smaller number of characters. Remove
the possibility by using a single character to signal the server is ready.
(Bitbake rev: 7fb2d6bed06439e59a81dd91798d886ee0a72e99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing slow startup in bitbake, add some timeing debug messages so
the logs are more useful for debugging when its slow.
(Bitbake rev: 92f61dfe409da2fdd7c609cf1125878aad273d04)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On loaded production systems we've seen bitbake server take over
40s to start up. Increase the timeout to 90s which tries to avoid
failures in selftests.
The delays come from setting up the inotify watches (31s) so can't
really be avoided.
After 5s delay we now warn the user we're waiting for 90s so the
interactive exeperience shouldn't be much changed and its very
unlikely the user would see that anyway.
(Bitbake rev: 492a5c1b32bee1f5d7978954ee0ebdf4fccdf56f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an indirect multiconfig dependency exists, such as:
A depends on B, B has a multiconfig dependency to C,and our build
target is A, the multiconfig dependency to C is not processed on
time, hence no providers are added for it, causing an exception in
the runqueue because the dependency does exist in it.
Call add_unresolved() for all available multiconfigs before processing
providers for multiconfig dependencies, detecting mcdepends on time so
providers for them can be added correctly.
(Bitbake rev: 25b585b981cc7e4ed48b0f7c89a075486fa1eb2b)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing issues where the server doesn't start with no logs as to why. Allow
the server to print the last 60 log lines just in case this shows us something useful
about what is failing.
(Bitbake rev: 1351978585b76262cb104f3d609d79c184ee5d2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 log lines may not capture any full traceback, increase the number of
lines to 60 which covers most tracebacks.
(Bitbake rev: 2626ff964c0a5726037e539cfd07027aded0b7a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #12898]
There might be no bitbake-cookerdaemon.log, print a message for debugging.
(Bitbake rev: 9c3c965b9d90617e8aa2cf439019534a35a7bcb1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This simplifies the code and makes it easier to read but has the
same functionality.
(Bitbake rev: b829d1b5eb486786cae088f6927530433a7e08e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The colors specified for use with bitbake-diffsigs were adapted for a
dark theme, e.g., by setting the background color to black, which made
it look very bad when used with a light theme.
To make it look good both with a dark or a light theme, it is better
to drop the background color. It is also better to leave out the color
altogether for the title and just use bold. Finally, dropping bold for
the red and green texts indicating removed/added values better matches
other colorized diff implementations as, e.g., git diff.
(Bitbake rev: 35769412de6387f55e5efa4ebc59a8d1e3c8870f)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Small tweak to ensure these items are printed into the log since there
is other logging code which looks for the header and this makes it clearer
the server did start but is slow somewhere in startup.
(Bitbake rev: 9b15f265643af8bd85af87164ee5cab0bf58ccbb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN
repositories from which tarballs are generated. These repositories were
protected from concurrent update with a lock on the tarballs. However,
the tarballs are specific to the SRCREV and module checked out (many
tarballs can come from the same repository), meaning a repository could
be modified concurrently if two recipes checked out two different
SRCREVs or modules from it in parallel. This caused errors like the
following:
ERROR: Fetcher failure: Fetch command failed with exit code 1, output:
svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked.
svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked.
Fix it by adding a per-repository lock that's independent of the module
and SRCREV.
(Bitbake rev: cddd27fb66bf82465cf8eb5012cb8e3c088625a5)
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de>
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if there are no setscene tasks, the disk monitor isn't started.
Move the startup code to somewhere to ensure it always is started. This
issue would partially explain occasional selftest failures.
(Bitbake rev: 7e76237c930d354ee7eb37d247d71b4f4fbf7993)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also use assetIn instead of assertTrue which aids debugging failures.
(Bitbake rev: 7e9c0bffb95ca92cab61deeeef18f45f1b4edb6f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When multiconfig is enabled the cooker adds providers
for all the targets to be built on all the multiconfig
variables that were set, regardless if there is a dependency
to it or not.
This causes an issue when a certain target is incompatible
with one or more of the multiconfigs, e.g. the target is not
in COMPATIBLE_MACHINE for one of the MACHINEs being built,
causing the cooker to error out since no providers can be
found for that certain target on that multiconfig.
This patch modifies the behavior to only look for PROVIDERS
for a target on the multiconfig that was selected to be built,
PROVIDERS are then looked for in other multiconfigs only when
there is a defined dependency to them.
[YOCTO #12985]
(Bitbake rev: 8bd519b04c7d6ee4cf626d5e04110c3759488d28)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code checks to see if shallow is either disabled or the tarball is
missing, but the else block tries to print the tarball filename, and
this attribute doesn't exist at all when shallow is disabled. Handle the
two cases separately to give sane errors for both cases without the
exception:
Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow'
(Bitbake rev: b0ad961064ef6ecd77fab3435e5652c4ae284426)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use % operator to ensure that the dependency is substituted into the error
message correctly.
(Bitbake rev: 8e587386b0b3a0ed11edf71dfbe9fb508f60d0b3)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support users who want to specify a custom ssh to Git using an environment
variable. (This lets build systems work without requiring users to
configure Git explicitly.)
(Bitbake rev: db46fb2755a4b1033ab60051ce511cc9dd7e34fb)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Git does not require the module and target path to be the same in the
.gitmodules file. This incorrect assumption was being made previously
causing various unpack failures.
An example .gitmodule showing this issue:
[submodule "plugins/WaveShaper/Libs/inih"]
path = plugins/wolf-shaper/Libs/inih
url = https://github.com/pdesaulniers/inih.git
The unpack function also needed to work in a loop on the overall
submodules_queue. Before it could have missed items that were not in the
primary repository.
(Bitbake rev: 5a7009c204f4d2254e3b2d83ad9319ac23f1cf4d)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Submodules by definition refer to a specific commit, not branch. If we don't
ignore the branch, then any commits on a submodule on a branch different then
the original module will trigger a failure that the commit is not on the
branch.
(Bitbake rev: fdc1dbf96f153b496de52acd8263366a1ff303ad)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a variable has a _remove applied to it but that variable is in turn
'renamed' through OVERRIDES, the removal gets lost with the current code.
TEST = "foo"
TEST_someval = "bar"
TEST_someval_remove = "bar"
OVERRIDES = "someval"
currently gives "bar" for TEST but should give "".
This fixes the code to track the removal and adds a test case to ensure this
doesn't regress again.
(Bitbake rev: 8f55010c18057be040f073d8bcb4c5c2c311d809)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds some basic tests for task checksums to ensure that the
checksums:
* change when variables change
* change when active _remove operators are present
* don't change when the _remove operators are not active
* change when an active contains() expression is present
* dont' change a contains() expression isn't active
There is a lot of other functionality which should be added to this
test but its a start.
(Bitbake rev: 5463c16e3619d324aed137f47f93f0997a227d29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By creating a standalone function, we can add better functional testing
of this code.
(Bitbake rev: 796a20d24dc18479de1975a37b9e52a5ac75c73f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently remove operations are not being accounted for in the task
checksums. This is a fairly serious oversight and needs to be fixed.
To do so, we need internal data from getVarFlag combined with the
expanded variable data so that only "active" remove operators are
accounted for in the task checksum. We can get this from the new
optional removes attribute in the returned parser object.
The code can then use the data on active remove operators to account
for the removals in task checksum but only when the removal is active.
We have to be careful here not to reference any expanded data since this
may for example contain build paths. This means we can only map back
and reference the unsplit (and hence unexpanded) remove string which may
expand to multiple removal values.
[YOCTO #12913]
(Bitbake rev: 57d2ee17ae83a139a37081eb082e6184fa883581)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The contents of the expand_cache is meant to match the return value of
getVarFlag() but the implementation was mostly in expandWithRefs(). If
an incorrect key was passed to expandWithRefs(), or a variable was only
partially expanded with no remove processing, the cache could become
corrupted.
Move the code to getVarFlag making the data lifecycle very clear, meaning
other calls to expandWithRefs() cannot corrupt the cache.
The expand_cache reset code needs to be moved ahead of any remote data
connectors too, since the expand_cache is now on the local side of the
connection.
(Bitbake rev: a039052f9b680eae53f3f12b7381b945f1d69253)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This brings _remove handling into line with _append/_prepend with regard
to the parsing flag to getVarFlag.
This is an internal flag and the only times this is used is through getVar
during renameVar operations and when processing ?= operations to see if
a variable is set. In either case we don't need to process remove operations.
Therefore take the minor speedup and skip processing for parsing=True.
[YOCTO #10945]
(Bitbake rev: 6d19eb3226b59922c0f888e33b28443635151501)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling getVarFlag with flag=None makes no sense, don't do it. Bitbake
used to silently ignore this, it now warns so avoid the warning.
(Bitbake rev: a68de8ace62eaba23856bfb301efbbe1824322aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Provide more information in the case the sources are not found in the
unpack step.
(Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A user friendly error is throw when neither the clonedir nor
fullshallow exist. Without the check, a difficult to interpret error
is throw from within the fetch command.
(Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For better readability and future use, we extract the function which
ensures that a given symlink exists.
(Bitbake rev: 5e69ca56533666a097bb23d09ab673e5c862051c)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The mapping of the URLs to the local shallow tarballs is not obvious. For
easier understanding, we add this tests to explicitly showing the mapping.
(Bitbake rev: 05f2ac8e19a027d103921b5ae0070db609a83042)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The mapping of the URLs to the local tarballs is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.
(Bitbake rev: c604d9402b4c9ad7eb5c794ab24f2f348d9719a9)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The mapping of the URLs to the local directory is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.
(Bitbake rev: 5f92682389fee437d6df2ff7718c571b7444e179)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some recent changes broke SRCPV and workaround was introduced
to avoid circular dependency if SRCPV is in PV.
However there is still the same error if SRCPV is in PR.
(Bitbake rev: 05ee4845f925b8528a7ce9cffb4bae425b8fa1e9)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
