There is the possibility data is being lost from the logfile due to
data buffering. Add in a couple of extra flush calls to ensure data
is being written out before the lock file is dropped.
Possible fix for [YOCTO #14000]?
(Bitbake rev: 596ea229a87d26a8e970c7ee77179519ad081fef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Further cleanup, just pass these settings directly.
(Bitbake rev: ac2284357f1fc7044dac9c146fad218fc9906412)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.
(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We already call bb.daemonize.createDaemon() in BitBakeServer so the extra
multiprocessing.Process() appears to be totally unneeded and just an extra layer
of forking which confuses things. Remove it.
(Bitbake rev: d214e55c45f9733b3289138feec0ae3361a4a48b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make it easier for users to debug shell task failure by including
some breadcrumbs in the emitted .run file that (hopefully) points
to the .bb/.bbclass file where the shell function was defined.
Unfortunately this won't work with functions with _append
or _prepends, since BitBake wipes the filename/lineno information.
This shouldn't be too hard to fix; for now, you'll just see
comments like this for such functions:
[YOCTO #7877]
(Bitbake rev: 9747211cbb45401cbf4dd0409e9c80c648a178c6)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is in prepartion for introducing the log-colorizer bbclass into poky.
(Bitbake rev: 889a873d71a6543efb71a0eb4ea6632c9f17175d)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also fixes DummyMultiStageProcessProgressReporter calling the wrong super __init__
(Bitbake rev: 7a1b4a7e4fffe54afe8d1d7e169ff558ad8c92d9)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just use the |resolve| function which already takes care of it.
(Bitbake rev: 91b809a0902ffd42be4edf7f0a7d25e6d354d822)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
UI control can happen via the xmlrpc connection. Account for this when timing
out UI connections. This was causing issues for toaster on systems where it
couldn't parse the metadata within the timeout.
(Bitbake rev: fa85a8263971c25e67fa3b421c686a90e46acd87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're only meant to be doing UI connection timeouts on the first connection
but haveui changes for each connection. We need to add a specific variable
to track this correctly and get the intended behaviour.
(Bitbake rev: e7c387c2e2fb2cc3ca1dc9d2029362909c326d72)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes the findSigInfo command to accept a recipe specified with the
multiconfig prefix
(Bitbake rev: 379951b6417eacbafc92ac1413ae1358bafdddfb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"bitbake mc:arm:bash mc:arm:busybox"
works but
"bitbake multiconfig:arm:bash multiconfig:arm:busybox"
does not. The reason is the list is modified whilst iterating.
Don't do that.
[YOCTO #13607]
(Bitbake rev: cd041a78d96e656438d93fb1e288080b8a6fe8bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a traceback if an invalid multiconfig is referenced in the bitbake
commandline and tweaks the message to make it more understanable.
(Bitbake rev: f31d7d0ad57b0ecc2ae06ed4b547c98df2aaa1a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
deltask currently supports only one task to delete but it would be useful
if it could support a variable which gets expanded to allow flexibility
in the metadata.
This is simple to support in bitbake and is how other directives such
as inherit operate so adjust the parser/code to handle that. It means
that syntax like:
EXTRA_NOPACKAGE_DELTASKS = ""
deltask ${EXTRA_NOPACKAGE_DELTASKS}
is now allowed.
(Bitbake rev: 883d926120833c85a16dcf60425dd7af7699046a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if all recipes in a layer are skipped, there are warnings that the
BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this
for skipped recipes.
The current code is hard to understand, not least as it passes variables
which functions modify by reference rather than giving a return value.
Update calc_bbfile_priority() to return values rather than modifying them.
Refactor the code to try and make it clearer what its doing and fix the
skipped recipe issue by passing in the list of parsed files.
The function is complicated by the need to not rerun regex matching more
than we ever have to which complicates the flow, it would be easier if we
just reran operations multiple times.
(Bitbake rev: 969cb27b4d978551817612ff4558bec81cfb655c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its bb.note or logger.info, this avoids a backtrace.
(Bitbake rev: 82c534ca1a1313de067b0d79c79857e89fa2764a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If server startup is broken for some reason (e.g. lockfile issues)
and no UI connection is made, the server will just sit inifinitely
waiting.
Add a timeout upon startup in the non-memory resident case so that
such infinite waits are avoided. In the memory resident case, the
server wouldn't have shut down in the first place or will timeout
according to configuration.
Since any race may mean the socket file is no longer present, ensure
the unlink doesn't fault upon exit, thus ensuring any hashequiv or
PRServ is removed from memory, allowing all processes to exit
cleanly in such scenarios.
(Bitbake rev: 39888b750df12478e8bdea6727cca112dce1df85)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing rare occasional races on the autobuilder as if two server
processes have the lockfile at the same time. We need to be extremely
careful this does not happen.
I think there is a potential race in this shutdown code since we delete
the lockfile, then call unlockfile() which also tries to delete it.
This means we may remove a lock file now held by another process if we're
unlucky. Since unlockfile removes the lockfile when it can, just rely on
that and remove any possible race window.
An example cooker-deamonlog:
--- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 ---
Started bitbake server pid 2266
Entering server connection loop
Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
Processing Client
Connecting Client
Running command ['setFeatures', [2]]
Running command ['updateConfig', XXX]
Running command ['getVariable', 'BBINCLUDELOGS']
Running command ['getVariable', 'BBINCLUDELOGS_LINES']
Running command ['getSetVariable', 'BB_CONSOLELOG']
Running command ['getSetVariable', 'BB_LOGCONFIG']
Running command ['getUIHandlerNum']
Running command ['setEventMask', XXXX]
Running command ['getVariable', 'BB_DEFAULT_TASK']
Running command ['setConfig', 'cmd', 'build']
Running command ['getVariable', 'BBTARGETS']
Running command ['parseFiles']
--- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 ---
Started bitbake server pid 8252
--- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 ---
Started bitbake server pid 13278
Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}]
Running command ['getRecipes', '']
Running command ['clientComplete']
Processing Client
Disconnecting Client
No timeout, exiting.
Exiting
where it looks like there are two server processes running which should not be.
In that build there was a process left sitting in memory with its bitbake.sock file
missing but holding the lock (not sure why it wouldn't timeout/exit).
(Bitbake rev: e1a7c1821483031b224a1570bfe834da755219cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
this allows shell statements like '; ;' to pass the parser.
As it may be bad code but still valid enough to execute
(Bitbake rev: b7732b1b5085bea73e17d112e1bd9ac3d4dc34fb)
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.
We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.
I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.
(Bitbake rev: 820ab886e79eea516560c0c008e4cf059c6e11a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're running into this timeout on loaded autobuilders in situations
where things should otherwise succeed. Log a note in these
cases and continue to try for longer.
(Bitbake rev: e567743e70f426786ae54dcb5ab550748d9266e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding back gtk objects import.
Fix bug introduce when adding validation on gtk import.
(Bitbake rev: 765be5ef60668f8a1cfbcba248f4995725807196)
Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running bitbake -g -u taskexp without having gi python module or
and invalid gtk version, bitbake fails with a stack trace.
In case of import or version error, bitbake should exit with an error
message instead of a stack trace.
(Bitbake rev: 2a2c507f239b047f34765312df4168030e38b90d)
Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The autobuilder can end up using build/../ syntax which is an issue
if the build directory is cleaned. Avoid this by using normpath()
on the file path passed in.
(Bitbake rev: 41988fec47eb196ab7195a75330a6d98de19101b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In rare cases the pickled data from a task contains "</event>" which
causes backtrace. This can be reproduced with something like:
do_unpack_prepend () {
bb.warn("</event>")
}
There are several solutions but the easiest is to catch this exception
and look for the next marker instead as this should be the only way such
an unpickle error could occur.
This fixes rare exceptions seen on the autobuilder.
Also add in other potential exceptions listed in the pickle manual
page so that better debug is obtained should there be an error in
this code path in future. exitcode doesn't need the same handling
since we control what is in that data field and it could never contain
</exitcode>
(Bitbake rev: 5ada512d6f9cbbdf1172ff7818117c38b12225ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that handled exceptions from the hash equivalence client was
raising an exception itself because hashserv.client wasn't imported
(Bitbake rev: a76290dfc6f34ff9f6efdb13a6db74b6b4759daf)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hash equivalence client and server can occasionally send messages
that are too large for the server to fit in the receive buffer (64 KB).
To prevent this, support is added to the protocol to "chunkify" the
stream and break it up into manageable pieces that the server can each
side can back together.
Ideally, this would be negotiated by the client and server, but it's
currently hard coded to 32 KB to prevent the round-trip delay.
(Bitbake rev: e27a28c1e40e886ee68ba4b99b537ffc9c3577d4)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ASSUME_PROVIDED can take regexs however the current way of handling
this in code is suboptimal. It means that you can add something like:
DEPENDS += "texinfo-nativejunk-that-does-not-exist"
and if texinfo-native is in ASSUME_PROVIDED, no error will occur.
Update the code to only treat something as a regex if a start or end
anchor character is present (which wouldn't be valid in a recipe name).
[YOCTO #13893]
(Bitbake rev: 3d72e23109990970fbb1086923277af752168b4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch implements three new SRC_URI options for the perforce
fetcher, namely:
* module
* remotepath
The options are intended to provide the user more control over the
downloaded file paths by allowing the user to specify how much of the remote
path should be preserved locally.
The changes in this patch are backwards compatible, i.e. if none of the
introduced options is specified, the default (old) behavior is enforced.
(Bitbake rev: aab228822d2f221c01337dd57d7582c51ce9a505)
Signed-off-by: Alexandru N. Onea <onea.alex@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds a basic implementation of a progress handler for the
perforce fetcher, based on the number of files to be downloaded and the
output behavior of the p4 print command used in the fetcher
implementation.
(Bitbake rev: f0582292bf79b0988048683dfd086aa3b9787344)
Signed-off-by: Alexandru N. Onea <onea.alex@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add an inverse mode for BBFILES_DYNAMIC.
Entries where the layername is prefix with '!' will act in inverted mode
including given glob mask in absence of the referenced layer.
This is useful to backfill recipes from potentially missing layer.
(Bitbake rev: bfd729cfe06351ccd0071004e23af1d04546c723)
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Upstream is unavailable, breaking tests. Switch to a YP mirror since
if we can't reach that there are bigger problems. This should remove
a source of intermittent failures on the autobuilder.
(Bitbake rev: 232471083d6c574c7ada6320f9379ad7d7862a9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The layout of the cache data hasn't changed, but the cache has now been
split into different files for multiconfig. If a user pulls in these
changes, it's possible that their base cache will still contain the
combined multiconfig cache entries, which are now unexpected and
generate errors like:
Unexpected multiconfig: foo
Bumping the version fixes this since the old cache data won't be
considered valid anymore.
(Bitbake rev: 1082188ce633ec6891c961d726f584b3f1259941)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value of TERM is leaking into OE-Core postinst-useradd-${PN} scripts,
which in turn can optionally be monitored by buildhistory. Prune the value in
order to make the OE-Core buildhistory output more deterministic.
(Bitbake rev: 0d5cdd0c0d65f2f81c3af0f3767fee86c4142c3a)
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the bitbake.srcrev nugget is not present for the commit we're
interested in we should not just bail out and say that an update is
needed. Instead we can recursively walk through the submodules and check
for the presence of the required commits.
(Bitbake rev: cfc78316309556bec487ef0a5a9205e41f1be86f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This prevents multiple bitbake.srcrev entries being created when a
repository has more than one submodule. It also ensures that the
bitbake.srcrev entry is not added unless all submodules are correctly
fetched.
(Bitbake rev: 860ff1193fe53f04696d41635a720c2d1f29fa7f)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The virtualfn variable is not defined, the filename is what should be
shown instead.
(Bitbake rev: 1f9d2c21db3a1ad2ab13dfebd2f8e9a7c3682ee2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a test to validate that mcdepends causes the dependent tasks to
build, and also that a change in the dependent task causes the dependee
task to re-execute.
(Bitbake rev: f2062c41693f9f684bdaf2df0a2a08b7f3871026)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passing all the data caches to the task hashing functions allows them to
correctly account for mcdepends in task signatures. This allows tasks to
be correctly re-run when a mcdepends changes.
By default, the legacy behavior is maintained for derived signature
generators by passing a special proxy object that can either be used to
access all multiconfigs or the legacy behavior. If a derived signature
generator is updated, it can set the supports_multiconfig_datacaces
property to instruct bitbake it deals with multiconfigs properly.
[YOCTO #13724]
(Bitbake rev: 8ff9203de4fce9c104c2987d86980c9f34036b97)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is expected that load_cachfile() returns an integer indicating how
many entries were loaded from the cache. In the event the cache needs to
be rebuilt, 0 must be returned to prevent python from attempting to add
an None and an integer together.
(Bitbake rev: 3459d98fbc280637ecb36961bda8436818ee51e5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All of the other multiconfig splitting functions are located in runqueue
so move the function to split a pn/fn there also so that its easier to
see them all together. Fixes a case where the findBestProvider() command
wasn't working for multiconfig because it was looking for a prefix of
"multiconfig:" instead of the newer "mc:"
(Bitbake rev: 325827af66434affc2da460cc8b9a5c460e38056)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a test to validate that multiconfigs can independently mask off
recipes by setting BBMASK. See the test description for further
information about how the test works.
(Bitbake rev: 513fc2dddf13d5e344162c26d89d2dde2fe85634)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Instead, each
cache now only parses the files for it's multiconfig.
(Bitbake rev: 3c5c7346adf4ca7ec761c08738f12401ba75b7c8)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improves the logging of Cache objects by prefixing the log messages with
the multiconfig name of the cache, so as to distinguish between multiple
instances of the class. Also adds a more log messages.
(Bitbake rev: 74fd10b33c66f4142d6eff6531200f7620a06ae0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
