Commit Graph

5511 Commits

Author SHA1 Message Date
Richard Purdie
b24605d8a4 bitbake: server/process: Fix note reference -> info
Its bb.note or logger.info, this avoids a backtrace.

(Bitbake rev: fb80389d2ce62e15e354c6ea288c9a4bbe0040f6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-21 16:57:48 +01:00
Richard Purdie
e0ef31f1d2 bitbake: server/process: Ensure UI-less servers don't sit in infinite loops
If server startup is broken for some reason (e.g. lockfile issues)
and no UI connection is made, the server will just sit inifinitely
waiting.

Add a timeout upon startup in the non-memory resident case so that
such infinite waits are avoided. In the memory resident case, the
server wouldn't have shut down in the first place or will timeout
according to configuration.

Since any race may mean the socket file is no longer present, ensure
the unlink doesn't fault upon exit, thus ensuring any hashequiv or
PRServ is removed from memory, allowing all processes to exit
cleanly in such scenarios.

(Bitbake rev: 81cad9b8c4df15218d1a419c6b8e3ac73b54600c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 39888b750df12478e8bdea6727cca112dce1df85)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
Richard Purdie
2add0d0cea bitbake: server/process: Fix a rare lockfile race
We're seeing rare occasional races on the autobuilder as if two server
processes have the lockfile at the same time. We need to be extremely
careful this does not happen.

I think there is a potential race in this shutdown code since we delete
the lockfile, then call unlockfile() which also tries to delete it.

This means we may remove a lock file now held by another process if we're
unlucky. Since unlockfile removes the lockfile when it can, just rely on
that and remove any possible race window.

An example cooker-deamonlog:

 --- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 ---
Started bitbake server pid 2266
Entering server connection loop
Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
Processing Client
Connecting Client
Running command ['setFeatures', [2]]
Running command ['updateConfig', XXX]
Running command ['getVariable', 'BBINCLUDELOGS']
Running command ['getVariable', 'BBINCLUDELOGS_LINES']
Running command ['getSetVariable', 'BB_CONSOLELOG']
Running command ['getSetVariable', 'BB_LOGCONFIG']
Running command ['getUIHandlerNum']
Running command ['setEventMask', XXXX]
Running command ['getVariable', 'BB_DEFAULT_TASK']
Running command ['setConfig', 'cmd', 'build']
Running command ['getVariable', 'BBTARGETS']
Running command ['parseFiles']
 --- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 ---
Started bitbake server pid 8252
 --- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 ---
Started bitbake server pid 13278
Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}]
Running command ['getRecipes', '']
Running command ['clientComplete']
Processing Client
Disconnecting Client
No timeout, exiting.
Exiting

where it looks like there are two server processes running which should not be.
In that build there was a process left sitting in memory with its bitbake.sock file
missing but holding the lock (not sure why it wouldn't timeout/exit).

(Bitbake rev: de919782f488a83b80d7c40896bf5b2596f1f65f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

(cherry picked from commit e1a7c1821483031b224a1570bfe834da755219cc)

Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
Konrad Weihmann
c15296aa63 bitbake: pyshyacc: allow double COMMA statements
this allows shell statements like '; ;' to pass the parser.
As it may be bad code but still valid enough to execute

(Bitbake rev: 85a499f8915dd6596d56808e6e1c536015e4eb34)

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b7732b1b5085bea73e17d112e1bd9ac3d4dc34fb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
Richard Purdie
ef3209ba8f bitbake: fetch2: Change git fetcher not to destroy old references
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.

We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.

I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.

(Bitbake rev: 6e527ea72b0816bf5f454473faf6523c365daa18)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 820ab886e79eea516560c0c008e4cf059c6e11a3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
Richard Purdie
3ef26fead4 bitbake: server/process: Increase timeout for commands
We're running into this timeout on loaded autobuilders in situations
where things should otherwise succeed. Log a note in these
cases and continue to try for longer.

(Bitbake rev: 344d3006a243f5f7d8f551556bab721b2ac0a2aa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e567743e70f426786ae54dcb5ab550748d9266e4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
Chris Laplante
03e20850af bitbake: ui/teamcity: don't use removed logging classes
Allows the TeamCity frontend to be used again.

(Bitbake rev: cea130b238e874bae30e13b3242a00eaff7f853b)

Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c5477ba79fcad4a887808dd0df9cfe3554e2c17a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
David Khouya
62de572029 bitbake: lib/ui/taskexp: Fix missing Gtk import
Adding back gtk objects import.

Fix bug introduce when adding validation on gtk import.

(Bitbake rev: 44e24fffc568dfee054b5feff2d9b382b9ce4bb5)

Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 765be5ef60668f8a1cfbcba248f4995725807196)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
David Khouya
38a756daa9 bitbake: lib/ui/taskexp: Validate gi import
When running bitbake -g -u taskexp without having gi python module or
and invalid gtk version, bitbake fails with a stack trace.

In case of import or version error, bitbake should exit with an error
message instead of a stack trace.

(Bitbake rev: 45e32f2651d35bf7a315af95cfe0b1070e038c9e)

Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2a2c507f239b047f34765312df4168030e38b90d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-19 16:22:12 +01:00
Richard Purdie
c587a49adc bitbake: msg: Avoid issues where paths have relative components
The autobuilder can end up using build/../ syntax which is an issue
if the build directory is cleaned. Avoid this by using normpath()
on the file path passed in.

(Bitbake rev: 1d3892d93ee9688d69f5bb5ea6aef8b1152bd1e3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 41988fec47eb196ab7195a75330a6d98de19101b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02 16:11:40 +01:00
Richard Purdie
b020384e85 bitbake: runqueue: Avoid unpickle errors in rare cases
In rare cases the pickled data from a task contains "</event>" which
causes backtrace. This can be reproduced with something like:

do_unpack_prepend () {
    bb.warn("</event>")
}

There are several solutions but the easiest is to catch this exception
and look for the next marker instead as this should be the only way such
an unpickle error could occur.

This fixes rare exceptions seen on the autobuilder.

Also add in other potential exceptions listed in the pickle manual
page so that better debug is obtained should there be an error in
this code path in future. exitcode doesn't need the same handling
since we control what is in that data field and it could never contain
</exitcode>

(Bitbake rev: 6d780fe3a111adbf3f3d2dda22d5a0787b195b62)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5ada512d6f9cbbdf1172ff7818117c38b12225ca)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02 16:11:40 +01:00
Joshua Watt
cbe8754af2 bitbake: siggen: Fix error when hash equivalence has an exception
The code that handled exceptions from the hash equivalence client was
raising an exception itself because hashserv.client wasn't imported

(Bitbake rev: a7fc2ae3fd40b7de36f7f9b3da64a610a698b4ed)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a76290dfc6f34ff9f6efdb13a6db74b6b4759daf)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02 16:11:40 +01:00
Joshua Watt
6ebf01bfd4 bitbake: hashserv: Chunkify large messages
The hash equivalence client and server can occasionally send messages
that are too large for the server to fit in the receive buffer (64 KB).
To prevent this, support is added to the protocol to "chunkify" the
stream and break it up into manageable pieces that the server can each
side can back together.

Ideally, this would be negotiated by the client and server, but it's
currently hard coded to 32 KB to prevent the round-trip delay.

(Bitbake rev: 1a7bddb5471a02a744e7a441a3b4a6da693348b0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e27a28c1e40e886ee68ba4b99b537ffc9c3577d4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02 16:11:40 +01:00
akuster
b6e0f5889e bitbake: test/fetch: change to better svn source
fixes:
svn: warning: W175002: Unexpected HTTP status 504 'Gateway Timeout' on '/openembedded/bitbake/!svn/vcc/default'
svn: E205011: Failure occurred processing one or more externals definitions

picked pcre2

[Yocto #13948]

(Bitbake rev: 767aa9316603a1c92c9c433a0c11ae98089bbc3a)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1483d17108da02f5d615e83403d5fd6288ca957c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02 16:11:40 +01:00
Richard Purdie
0045df8f9b bitbake: tests/fetch: Switch from git.infradead.org to a YP mirror
Upstream is unavailable, breaking tests. Switch to a YP mirror since
if we can't reach that there are bigger problems. This should remove
a source of intermittent failures on the autobuilder.

(Bitbake rev: 83296870bede70e31bdf6e73683bcc30681023fc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-22 20:53:28 +01:00
Richard Purdie
5e44568d90 bitbake: data_smart: Handle hashing of datastores within datastores correctly
If there is a datastore within a datastore (e.g. BB_ORIGENV) then
get-hash() doesn;t correclty handle the contents using the memory
address instead of the contents.

This is a patch from dominik.jaeger@nokia.com which addresses
this problem. Its been low priority since we don't include
BB_ORIGENV anywhere this would cause an issue as standard.

[YOCTO #12473]

(Bitbake rev: 1a8bcfc1eb89ccff834ba68fb514330b510976a2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 13:41:29 +01:00
Richard Purdie
9062a225aa bitbake: cache: Fix performance problem with large numbers of source files
Some companies are using large numbers of patch files in SRC_URI.
Rightly or wrongly that exposes a performance problem where the code
does not handle the large string manipulations in a way which works
efficienty in python.

This is a modified version of a patch from z00539568
<zhangyifan46@huawei.com153340508@qq.com which addresses the performance
problem. I modified it to use a more advanced regex, retain the "*" check
and cache the regex.

[YOCTO #13824]

(Bitbake rev: c07f374998903359ed55f263c86466d05aa39b68)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 13:41:29 +01:00
Paul Barker
d03a6d33de bitbake: fetch2/wget: Set User-Agent when checking status of a URL
When a website is behind a CDN like Cloudflare there may be a "Browser
Integrity Check" or other test applied to requests before they are
allowed through to the server. Downloading via wget passes these tests
as headers are set appropriately, however the Python urllib module may
fail these tests unless additional headers are set. This causes
Wget.checkstatus() to fail where Wget.download() would actually succeed.

For Cloudflare in particular a valid User-Agent is needed, it's easy to
add this to the headers in Wget.checkstatus(). The user agent string is
copied from Wget._fetch_index().

(Bitbake rev: 4679d3cdb9cdf23f3962aa61c599ad7474591f9f)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 13:41:29 +01:00
Hongxu Jia
14c7608311 bitbake: tinfoil: fix config_data mess up insane check while parsing multiple recipes
Since commit [tinfoil: Simplify remote datastore connections][1] and
[tinfoil: Add back ability to parse on top of a datastore][2] applied,
bitbake run command parseRecipeFile with param config_data.dsindex rather
than config_data.

While calling tinfoil.parse_recipe_file() with one config_data (with the same
config_data.dsindex) to parse multiple recipes, it will mess up insane check.

It broke update_layer.py on layerindex, here are the simplified steps:
[snip]
t= bb.tinfoil.Tinfoil()
t.prepare()
data = bb.data.createCopy(t.config_data)
fn = "path_to/oe-core/meta/recipes-graphics/images/core-image-clutter.bb"
t.parse_recipe_file(fn, appends=False, config_data=data)
fn = "path_to/oe-core/meta/recipes-graphics/packagegroups/packagegroup-core-x11-base.bb"
t.parse_recipe_file(fn, appends=False, config_data=data)

|  File "path_to/oe-core/meta/classes/insane.bbclass", line 1303,
in __anon_1304__path_to_oe_core_meta_classes_insane_bbclass
|    bb.fatal("Fatal QA errors found, failing task.")
[snip]

In above failure, RDEPENDS is assigned `${PACKAGE_INSTALL} ${LINGUAS_INSTALL}
${IMAGE_INSTALL_DEBUGFS}' in core-image-clutter.bb, but it broke insane check
on packagegroup-core-x11-base.bb

>From commit [remotedata: enable transporting datastore from the client to
the server][3], it create a new DataSmart to save receive_datastore's remote_data
Similarly, make a copy of config_data(with different config_data.dsindex) could
fix the issue.

[1] http://git.openembedded.org/bitbake/commit/?id=85e03a64dd0a4ebe71009ec4bdf4192c04a9786e
[2] http://git.openembedded.org/bitbake/commit/?id=4618da2094189e4d814b7d65672cb65c86c0626a
[3] http://git.openembedded.org/bitbake/commit/?id=784d2f1a024efe632fc9049ce5b78692d419d938

(Bitbake rev: a3074807974536e370289c25fddcb9ad93cbc137)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 13:41:29 +01:00
Richard Purdie
70ee928f8f bitbake: tinfoil: Add back ability to parse on top of a datastore
This option was removed recently as we didn't realise the layerindex
relies upon it. Add back the API which it turns out can be supported
as long as we assume the datastore passed in is a remote datastore
which it usually would be unless created locally.

(Bitbake rev: 4618da2094189e4d814b7d65672cb65c86c0626a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:45:34 +01:00
Richard Purdie
9435b6cf10 bitbake: data/siggen: Don't expand ignored variables
If a variable is in the signature whitelist, we'd currently expand
it, then later ignore the data. This is problemtic for code which
has effects when expanded, recently source date epoch in OE-Core
for example.

We don't actually need to do this, if we pass the whitelist into
the earlier function it can avoid the expansion. This also also
give a small performance boost since we avoid running code in some
cases.

[YOCTO #13581]

(Bitbake rev: f483ee4a869fb1dafbe4bdf2da228cdaa40b38bd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:45:34 +01:00
Richard Purdie
40407d28f0 bitbake: bitbake: Update to version 1.46
(Bitbake rev: 76396230731432b38fdcb25ad27bb84065bc89e5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:43:38 +01:00
Richard Purdie
5c323ef6e5 bitbake: fetch2: Fix urldata_cache key issues
Upon inspection its clear the way the keys for this cache were being handled
would break it and cause the cache to never be used. Fix this.

(Bitbake rev: 9a5dd1be63395c76d3fac2c3c7ba6557fe47b442)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-30 12:33:50 +01:00
Paul Eggleton
abee806f11 bitbake: toaster: fix for import build directory with var refs in BBLAYERS
Update importing a build directory to support where bblayers.conf
sets BBLAYERS to a value that includes a variable reference e.g.:
  BBLAYERS = "${TOPDIR}/../meta \
              ${TOPDIR}/../meta-selftest"

[YOCTO #13707]

(Bitbake rev: 5bd29d448a31c132afd6fc0127029e246759b87b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 20:02:36 +01:00
Reyna, David
4f959dff76 bitbake: toaster: FieldError loading Reverse runtime dependencies
When selecting reverse runtime dependencies, filter on simply 'depends_on'
instead of the obsolete 'depends_on__name' or 'depends_on__size'.

[YOCTO #13717]

(Bitbake rev: 085940c12619549be3fbd4ead9379ab0a97ac5bf)

Signed-off-by: Ahmed.Hossam <Ahmed.Hossam@opensynergy.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 20:02:36 +01:00
Reyna, David
3dc19bb3be bitbake: toaster: update to Dunfell
Toaster needs update to Dunfell

[YOCTO #13847]

(Bitbake rev: 387fcfb3ffe573715a1997d729237a49cc889e1b)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 20:02:36 +01:00
David Reyna
4dabdbe11d bitbake: toaster: migrate to Django-2.2
Toaster migration to Django-2.2. Django-1.x has been deprecated.

[YOCTO #13207]

(Bitbake rev: 9730f95686b2ac72cf1fa513c555f7c7787e2667)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 20:02:36 +01:00
Khem Raj
12fc1c5637 bitbake: Revert "fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY is "cache""
As per mailing list discussion, the cache policy was behaving correctly before
and wouldn't expect to update after the initial fetch even for AUTOREV.

This reverts commit ba093a38539960e645e994a66ed7872a604c00a9.

(Bitbake rev: 51f827911b7202de3e855e683fdbd732d7a84e09)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:24 +00:00
Richard Purdie
25f1f4644d bitbake: tinfoil: Add iterator support for DataStoreConnector
Some usages need to iterate the datastore. This is slow and not recommended
but support this.

(Bitbake rev: 38b4e330c6c2007e8ead878e1d9e9580d3b19c7b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Richard Purdie
d53958766e bitbake: tinfoil: Simplify remote datastore connections
The current approach to remote datastores used in tinfoil is breaking. For
example, adding a devupstream extension to a recipe with a git upstream,
making it the preferred version and then running "devtool modify" on it
causes get_srcrev() circular dependency issues. The problem is the override
handling in the datastore is broken.

This gets broken since remotedata:recieve_datastore() sets d.dict but doesn't
update d.overridedata (or d.inchistory or d.varhistory). We could play
whack-a-mole but the current implementation seems to be flawed to me. It
also doesn't cover, or only partially covers some datastore operations and
each needs new dedicated command API.

Instead, step back and reimplement the way the datastore connector works.

With this change, the datastore is either remote or local but the data is not
spread on two sides of the connection. All the API is proxied over the connection
by a single function for the datastore (and two to support variable history
and include history).

This code does not support using the datastore as a parameter to any data store
functions. We did have one case of that but its just bad code and can be
replaced.

The result is something which is much simpler and less invasive to the datastore
code itself, meaning its behaviour should be much more consistent. The existing
tests for the remote data no longer make any sense and are removed.

The one bug this code would have is if key/value pairs are returned over the IPC
and those values contained a DataSmart object since we don't recurse into return
values to find such things. Nothing appears to do that currently so lets worry
about it if its ever an issue. This change should simplfy a ton of other issues
and avoid a ton of other bugs so is a huge net gain.

Tested with bitbake's and OE's selftests.

(Bitbake rev: 85e03a64dd0a4ebe71009ec4bdf4192c04a9786e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Richard Purdie
89705a60f3 bitbake: data_smart: Don't pass unneeded datastore
The datastore is already available to this function internally so don't
also try and pass the datastore as a parameter. This is clearly broken
API when you look at the existing calls to it.

This then doesn't break the planned tinfoil data connector changes.

(Bitbake rev: af1654498ee5b47368a41dad2d2b7b6abc19ff15)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Richard Purdie
c0e79ee2e9 bitbake: tinfoil: Drop parse_recipe_file custom datastore support
This parameter is unused except for a single selftest and is problematic
with regard to fixing some other bugs. Remove it for now, if really needed
we could re-implement it in some other way in the future. Experience
tells us we likely don't want to support this kind of change to the metadata
anyway as its not as useful as it first sounds/appears.

(Bitbake rev: f2341dc46b6abfc08a4d01c7abdd3d4630577999)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Richard Purdie
ca5b3b4d1c bitbake: tinfoil: Update to match recent knotty console changes
This updates tinfoil to match recent changes to the logging code
in knotty.

(Bitbake rev: e67dfa4a4d0d63e4752655f25367582e5a95f1da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
bf7d68c812 bitbake: knotty/msg: Use logging.shutdown() instead of bb.msg.cleanupLogging()
The logging module provides a shutdown() function that does the same
thing in a much better way

(Bitbake rev: 970cd2fc4f0bbc93069dee5a15a608dd76081c67)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
5ce50b3ce7 bitbake: knotty: Treat verbconsole as a console output
The BitBake.verbconsole needs to be treated like a console output logger
(meaning that the TerminalFilter attaches an InteractConsoleLogFilter to
it), even if it's not directly attached to the root 'BitBake' logger.
First, assign a special "is_console" property to the relevant handlers,
then look for the property in the handlers from the configuration object
return by bb.msg.setLoggingConfig(). Finally, pass the list of all
handlers to the TerminalFilter object; it doesn't care about the
difference between console and errconsole, so pass all the relevant
handlers as a list.

This fixes cases where the console output was corrupted when messages
were sent to the 'BitBake.verbconsole' handler.

(Bitbake rev: 2010be588c74a99256df7b565a309c84c2973546)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
0dd9e5a84d bitbake: msg: Return config object
Returns the configuration object from setLoggingConfig(). This object
has a config dictionary that contains all of the created handlers,
filters and loggers, which makes it much easier to pull out items with
specific names.

(Bitbake rev: 93c98a85cb10d3795b3cebc9cd97214d0f4748e5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
b2d55ec1eb bitbake: knotty: Update hash equivalence logging
Updates hash equivalence logging so that the interesting VERBOSE
messages are always logged to the consolelog file so that issues in
individual user builds can be diagnosed. The autobuilder logging config
then updates this so that they also are shown on stdout, since the
consolelog file is not capture there.

In order to facilitate this, 2 new logging handlers were added,
"BitBake.verbconsole" and "BitBake.verbconsolelog". Neither of these
handlers are attached to anything by default, but they will log any
messages that wouldn't otherwise be logged by the normal console or
consolelog handlers. Users can attach whatever loggers the desire to
this handler to get them to appear on the console or in the consolelog,
as demonstrated by the autobuilderlog.json file.

(Bitbake rev: 766587c0baaaeb5cda3e9454395edbb70e33f756)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
5766cf50b0 bitbake: runqueue: Lower setscene complete logging level
Lowers the level of the log message when setscene tasks have completed.
This message can occur multiple times when hash equivalence is enabled,
since the runqueue switches between executing setscene tasks and normal
tasks. Since this is primarily of use when debugging hash equivalence,
use the hash equivalence logger at VERBOSE level.

[YOCTO #13813]

(Bitbake rev: 7dd5b3900622008ff34ec70d71c6e994f460a46f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
361ea84d56 bitbake: msg: Add helper to merge logging configs
Adds a function that can be used by UI front ends to merge logging
configuration fragments.

(Bitbake rev: e860d93b76c142fdae90226bc68e4fe6e7b6a9db)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
c8b2694cd4 bitbake: knotty: Add logging cleanup
Adds code to close all loggers when bitbake exits. This prevents
unclosed file ResourceWarnings. A form of this closing existed
previously, but was removed in the new logging code.

(Bitbake rev: b3f3779adf63c0d970462a558a6205da1d30c0ed)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
622ec78f7e bitbake: runqueue/siggen: Lower hash equivalence logging
Lowers the level at which hash equivalence messages are logged so as to
not annoy the majority of users. The autobuilder can use a custom
logging configuration to log these to a file for debugging (see
contrib/autobuilderlog.json)

[YOCTO #13813]

(Bitbake rev: 2ddb649ea31afe052f08e3969e36abf6fb515bc2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
554894287e bitbake: runqueue/siggen: Log hash equivalence with a different logger
Switches the hash equivalence logging to use a different logger so that
it can be easily filtered out with python's structured logging.

(Bitbake rev: 20bb29ef973e9c5483eb50a74550ea207637367b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
1c9874b9f3 bitbake: knotty: Setup logs with config helper
Sets up logging in knotty to use python's structured logging config and
the bb.msg.setLoggingConfig() helper to setup logging. This allows the
user to specify additional logging mechanism in a config file with
BB_LOGCONFIG

(Bitbake rev: 646a68a49364b50a42168b4b16308f7217eec0dc)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
300fd2a659 bitbake: knotty: Remove dependency on format variable
Passing around the log formatter variable was unnecessary since the log
levels of interest can be accesses as class members of
bb.msg.BBLogFormatter. Switching to do this will make using the
structured python logging much easier, since it can be difficult to
extract out the formatter for a specific handler.

(Bitbake rev: c1c867df24b4ef204027d485acac7c75c63f2bc0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
6b9eacb404 bitbake: lib/bb/msg: Add helper to set logging config
Adds a helper function to setup the structure logging information in
bitbake. The helper function takes a default configuration and an
optional path to a user config file that can be merged into the default.

(Bitbake rev: 14c98d36b74c1599b4649078170b6e72df79ba2a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
143685de1b bitbake: lib/bb/msg: Remove unused filters
Now that the filter are described using the python logging structure,
these classes are no longer needed.

(Bitbake rev: 2cb16e0c61609f3fb8a86530ddedf8ad0e69428e)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
9030edf4af bitbake: lib/bb/msg: Add filter utilities
Adds generic filter utilities that can be used to filter when the log
level is greater than or equal to a defined value, or below a defined
value, as well as a generic function to translate a string to a logging
level (or bitbake logging level)

(Bitbake rev: 8ebe8a7662200fad9b88be8b08376262a61c85c4)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
ae6e0e0a51 bitbake: knotty: Add commented logging_tree code
Adds a comment section that can be easily uncommented to enable dumping
the logging tree. This module is extremely useful for debugging issued
with logging configuration

(Bitbake rev: 30461310915f911b80f92e03df694af7c1eb1f46)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
4bbd4dc6b5 bitbake: lib/bb/msg: Add repr for BBLogFormatter
Adds a __repr__ function for BBLogFormatter. This allows it to get a
human readable string when printed using the logging_tree module

(Bitbake rev: 5ff962dca71f4ef16a3aa11b41a23f2227fe8c21)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
4c9a412999 bitbake: lib/bb/msg: Use log level instead of debug count
Passes around the actual logging level as the default log level variable
instead of the debug count. This makes it easier to deal with logging
levels since the conversion from debug count and verbose flag only has
to occur once when logging is initialized and after that actual log
levels can be used

(Bitbake rev: 41bd155faf7f65cb0727fcce972715769b26ca89)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00