Only the env variables which were added to the datastore were being
updated. We need to update the whole copy, we just only trigger a
reparse if any of the variables making it into the datastore change.
This avoids a bug where variables such as DISPLAY in a new UI context
would break under memory resident bitbake.
(Bitbake rev: 4bb71b627767297269e762b414443e15e28bfac4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should not be using sys.argv inside the server to decide which targets
the user added on the commandline. There might not even be a commandline.
Thankfully the targets variable is easily accessible in this context
and contains this exact data we want.
(Bitbake rev: 5b12bf30bccdd00262e74964223220c649040be4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to have a new python process forked off an original doesn't work
out well and ends up having race issues. To avoid this, exec() a new
bitbake server process. This starts with a fresh python interpreter
and resolves various atexit and other multiprocessing issues once
and for all.
(Bitbake rev: 9501dd6fdd7a7c25cbfa4464cf881fcf8c049ce2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dump info into the logs if there are extra threads left at process exit
time.
(Bitbake rev: 1c9496797b753e67351bd5cb98ef2b8e9435d51e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes to configuration.env were not updating BB_ORIGENV, fix this.
(Bitbake rev: c5fbd8452f87e0a2d234eaf27d0450aacdeb8891)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.
This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.
The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.
(Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sock object isn't used client side so we can just created it in
the server process and save passing around the fd/object.
(Bitbake rev: ee5d2c92dcce89ccb701e028ffc6419eb315f5ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.
The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.
(Bitbake rev: 3caa43b665604475d2c87ba505efb0b9fca9c2e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the eventlog is not handled correctly for memory resident
bitbake. Fix this by allowing adpations to configuration changes.
(Bitbake rev: f7d2c9116116659ea42260a3bb96dca100aadae7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were some options which were not being passed to the server. This
was breaking, particularly in memory resident bitbake mode.
Add the missing options to the correct place to ensure the server is
updated correctly.
(Bitbake rev: 5dc178e7fae3ca8558146e385a05f5d96a021777)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing debug/verbose/debug_domains around, pass the
computed output of these. Ensure that the cooker sets the levels
to match the levels currently set in the UI and generally try and
make it easier to understand what the code is doing.
(Bitbake rev: f0535beecc692a6213be2a22f9eef5956450ecf8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
(Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its becomming clear the upstream server doesn't like this, drop these
two urls from the test, not sure we need them here anyway.
(Bitbake rev: ab2ef942dc21f9639793c972f2e546edf9444783)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hongxu Jia reported a problem where the bb_cache files were not always being
written out correctly. This was due to the sync thread being terminated
prematurely.
Whilst the preceeding changes mean the exit handler for this thread is now
correctly called since we switch to using sys.exit() instead of os._exit(),
this write can happen after we drop the bitbake lock, leading to potential
races. Avoid that headache by adding in explicit thread join() calls before
we drop the lock (which atexit or Finalize can't do).
(Bitbake rev: afd1900939f7b042297558f4cb01f50f3a299267)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Each run through the parser would leak a thread from the queue created to
shut the parser down. Close this down correctly and clean up the code flow
slightly whilst in the area, making sure this thread does shut down correctly
(we don't care if it loses data).
(Bitbake rev: 51ba35e9bbd4da8b5f3b3b5b8213cb634a6b0f06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buffering parameter was removed in python 3.1 and made default
so we can clean up the code. This removes weird looking double
exceptions when connections fail.
(Bitbake rev: 06b7bafbd18a47c8db2f7b943dc535c65df176bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this fix, this example basepath (a):
virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile
would get incorrectly "cleaned" into:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb
When searching backwards in `a` trying to isolate the 'virtual:xyz' to add
it to the end of the string, we need to consider `a` still has the recipe
path and taskname. So stoping the rsplit after only 1 split is not enough.
We want to reach the second ':' from the end.
This way, we obtain:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native
reviewed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
(Bitbake rev: d193d93422a0ad62aa35b5d4ca5da8d422f72180)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This shuts up both the DEBUG and EXIT handlers.
Also, remove an unneeded "set -o errtrace" (i.e. set -E),
since we no longer use an ERR trap.
(Bitbake rev: 89e851fa0403d1e98aeed69990101e3f84f0b283)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Leverage the comments that emit_var writes and the backtrace that
the shell func writes to generate an additional metadata-relative
backtrace. This will help the user troubleshoot shell funcs much
more easily.
Example:
| WARNING: /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955:171 exit 1 from 'exit 1'
| WARNING: Backtrace (BB generated script):
| #1: myclass_do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 171
| #2: do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 166
| #3: actually_fail, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 153
| #4: my_compile_extra, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 155
| #5: do_compile, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 141
| #6: main, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 184
|
| Backtrace (metadata-relative locations):
| #1: myclass_do_something, /home/laplante/repos/oe-core/meta/classes/myclass.bbclass, line 2
| #2: do_something, autogenerated, line 2
| #3: actually_fail, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 36
| #4: my_compile_extra, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 38
| #5: do_compile, autogenerated, line 3
ERROR: Task (/home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 542 tasks of which 541 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
(Bitbake rev: ae1aa4ea79826c32b20e1e7abdf77a15b601c6f2)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the possibility data is being lost from the logfile due to
data buffering. Add in a couple of extra flush calls to ensure data
is being written out before the lock file is dropped.
Possible fix for [YOCTO #14000]?
(Bitbake rev: 596ea229a87d26a8e970c7ee77179519ad081fef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Further cleanup, just pass these settings directly.
(Bitbake rev: ac2284357f1fc7044dac9c146fad218fc9906412)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.
(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We already call bb.daemonize.createDaemon() in BitBakeServer so the extra
multiprocessing.Process() appears to be totally unneeded and just an extra layer
of forking which confuses things. Remove it.
(Bitbake rev: d214e55c45f9733b3289138feec0ae3361a4a48b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make it easier for users to debug shell task failure by including
some breadcrumbs in the emitted .run file that (hopefully) points
to the .bb/.bbclass file where the shell function was defined.
Unfortunately this won't work with functions with _append
or _prepends, since BitBake wipes the filename/lineno information.
This shouldn't be too hard to fix; for now, you'll just see
comments like this for such functions:
[YOCTO #7877]
(Bitbake rev: 9747211cbb45401cbf4dd0409e9c80c648a178c6)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is in prepartion for introducing the log-colorizer bbclass into poky.
(Bitbake rev: 889a873d71a6543efb71a0eb4ea6632c9f17175d)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also fixes DummyMultiStageProcessProgressReporter calling the wrong super __init__
(Bitbake rev: 7a1b4a7e4fffe54afe8d1d7e169ff558ad8c92d9)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just use the |resolve| function which already takes care of it.
(Bitbake rev: 91b809a0902ffd42be4edf7f0a7d25e6d354d822)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
UI control can happen via the xmlrpc connection. Account for this when timing
out UI connections. This was causing issues for toaster on systems where it
couldn't parse the metadata within the timeout.
(Bitbake rev: fa85a8263971c25e67fa3b421c686a90e46acd87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're only meant to be doing UI connection timeouts on the first connection
but haveui changes for each connection. We need to add a specific variable
to track this correctly and get the intended behaviour.
(Bitbake rev: e7c387c2e2fb2cc3ca1dc9d2029362909c326d72)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes the findSigInfo command to accept a recipe specified with the
multiconfig prefix
(Bitbake rev: 379951b6417eacbafc92ac1413ae1358bafdddfb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"bitbake mc:arm:bash mc:arm:busybox"
works but
"bitbake multiconfig:arm:bash multiconfig:arm:busybox"
does not. The reason is the list is modified whilst iterating.
Don't do that.
[YOCTO #13607]
(Bitbake rev: cd041a78d96e656438d93fb1e288080b8a6fe8bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a traceback if an invalid multiconfig is referenced in the bitbake
commandline and tweaks the message to make it more understanable.
(Bitbake rev: f31d7d0ad57b0ecc2ae06ed4b547c98df2aaa1a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
deltask currently supports only one task to delete but it would be useful
if it could support a variable which gets expanded to allow flexibility
in the metadata.
This is simple to support in bitbake and is how other directives such
as inherit operate so adjust the parser/code to handle that. It means
that syntax like:
EXTRA_NOPACKAGE_DELTASKS = ""
deltask ${EXTRA_NOPACKAGE_DELTASKS}
is now allowed.
(Bitbake rev: 883d926120833c85a16dcf60425dd7af7699046a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if all recipes in a layer are skipped, there are warnings that the
BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this
for skipped recipes.
The current code is hard to understand, not least as it passes variables
which functions modify by reference rather than giving a return value.
Update calc_bbfile_priority() to return values rather than modifying them.
Refactor the code to try and make it clearer what its doing and fix the
skipped recipe issue by passing in the list of parsed files.
The function is complicated by the need to not rerun regex matching more
than we ever have to which complicates the flow, it would be easier if we
just reran operations multiple times.
(Bitbake rev: 969cb27b4d978551817612ff4558bec81cfb655c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its bb.note or logger.info, this avoids a backtrace.
(Bitbake rev: 82c534ca1a1313de067b0d79c79857e89fa2764a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If server startup is broken for some reason (e.g. lockfile issues)
and no UI connection is made, the server will just sit inifinitely
waiting.
Add a timeout upon startup in the non-memory resident case so that
such infinite waits are avoided. In the memory resident case, the
server wouldn't have shut down in the first place or will timeout
according to configuration.
Since any race may mean the socket file is no longer present, ensure
the unlink doesn't fault upon exit, thus ensuring any hashequiv or
PRServ is removed from memory, allowing all processes to exit
cleanly in such scenarios.
(Bitbake rev: 39888b750df12478e8bdea6727cca112dce1df85)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing rare occasional races on the autobuilder as if two server
processes have the lockfile at the same time. We need to be extremely
careful this does not happen.
I think there is a potential race in this shutdown code since we delete
the lockfile, then call unlockfile() which also tries to delete it.
This means we may remove a lock file now held by another process if we're
unlucky. Since unlockfile removes the lockfile when it can, just rely on
that and remove any possible race window.
An example cooker-deamonlog:
--- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 ---
Started bitbake server pid 2266
Entering server connection loop
Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
Processing Client
Connecting Client
Running command ['setFeatures', [2]]
Running command ['updateConfig', XXX]
Running command ['getVariable', 'BBINCLUDELOGS']
Running command ['getVariable', 'BBINCLUDELOGS_LINES']
Running command ['getSetVariable', 'BB_CONSOLELOG']
Running command ['getSetVariable', 'BB_LOGCONFIG']
Running command ['getUIHandlerNum']
Running command ['setEventMask', XXXX]
Running command ['getVariable', 'BB_DEFAULT_TASK']
Running command ['setConfig', 'cmd', 'build']
Running command ['getVariable', 'BBTARGETS']
Running command ['parseFiles']
--- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 ---
Started bitbake server pid 8252
--- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 ---
Started bitbake server pid 13278
Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}]
Running command ['getRecipes', '']
Running command ['clientComplete']
Processing Client
Disconnecting Client
No timeout, exiting.
Exiting
where it looks like there are two server processes running which should not be.
In that build there was a process left sitting in memory with its bitbake.sock file
missing but holding the lock (not sure why it wouldn't timeout/exit).
(Bitbake rev: e1a7c1821483031b224a1570bfe834da755219cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
this allows shell statements like '; ;' to pass the parser.
As it may be bad code but still valid enough to execute
(Bitbake rev: b7732b1b5085bea73e17d112e1bd9ac3d4dc34fb)
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.
We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.
I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.
(Bitbake rev: 820ab886e79eea516560c0c008e4cf059c6e11a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're running into this timeout on loaded autobuilders in situations
where things should otherwise succeed. Log a note in these
cases and continue to try for longer.
(Bitbake rev: e567743e70f426786ae54dcb5ab550748d9266e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding back gtk objects import.
Fix bug introduce when adding validation on gtk import.
(Bitbake rev: 765be5ef60668f8a1cfbcba248f4995725807196)
Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running bitbake -g -u taskexp without having gi python module or
and invalid gtk version, bitbake fails with a stack trace.
In case of import or version error, bitbake should exit with an error
message instead of a stack trace.
(Bitbake rev: 2a2c507f239b047f34765312df4168030e38b90d)
Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The autobuilder can end up using build/../ syntax which is an issue
if the build directory is cleaned. Avoid this by using normpath()
on the file path passed in.
(Bitbake rev: 41988fec47eb196ab7195a75330a6d98de19101b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In rare cases the pickled data from a task contains "</event>" which
causes backtrace. This can be reproduced with something like:
do_unpack_prepend () {
bb.warn("</event>")
}
There are several solutions but the easiest is to catch this exception
and look for the next marker instead as this should be the only way such
an unpickle error could occur.
This fixes rare exceptions seen on the autobuilder.
Also add in other potential exceptions listed in the pickle manual
page so that better debug is obtained should there be an error in
this code path in future. exitcode doesn't need the same handling
since we control what is in that data field and it could never contain
</exitcode>
(Bitbake rev: 5ada512d6f9cbbdf1172ff7818117c38b12225ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that handled exceptions from the hash equivalence client was
raising an exception itself because hashserv.client wasn't imported
(Bitbake rev: a76290dfc6f34ff9f6efdb13a6db74b6b4759daf)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
