Commit Graph

6109 Commits

Author SHA1 Message Date
Robert Yang
2cb6a75b5a bitbake: bitbake: server/process: Handle BBHandledException to avoid unexpected exceptions
The parseBaseConfiguration() raises bb.BBHandledException(), but
BitBakeServer() didn't handle it, so we always got unexpected exceptions when
there were errors. For example:

=== Case 1:
* Add "print "hello"' in base.bbclass' def oe_import() function
def oe_import(d):
    print "hello"
[snip]

$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Last 60 lines of server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
  File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 21
    print "hello"
                ^
SyntaxError: Missing parentheses in call to 'print'

<The first exception>
During handling of the above exception, another exception occurred:
<Tracebacks>

<The second exception>
During handling of the above exception, another exception occurred:
<Tracebacks>

<The third exception>
During handling of the above exception, another exception occurred:
<Tracebacks>
[snip]

Now it looks like:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 21:

The code lines resulting in this error were:
     0001:def oe_import(d):
 *** 0002:    print "hello"
     0003:    import sys
     0004:
     0005:    bbpath = d.getVar("BBPATH").split(":")
     0006:    sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
SyntaxError: Missing parentheses in call to 'print' (base.bbclass, line 21)

=== Case 2:
* Add 'HOSTTOOLS += "hello"' to conf/local.conf:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
<Tracebacks>
[snip]
During handling of the above exception, another exception occurred:
[snip]
<Tracebacks>
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

The error message is printed by bb.fatal() which raises bb.BBHandledException(),
but BitBakeServer() doesn't handle it, so we got it.

Now it looks like:
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

No unexpected exceptions anymore.

[YOCTO #13267]

(Bitbake rev: 6e6865e6371dbd31a136eae64cc5b1fa5f5bee33)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-16 10:05:32 +01:00
Richard Purdie
5f0d31ce46 bitbake: runqueue: Ensure data is handled correctly
This doesn't appear to have ill effects right now but there is a correctness
issue which this so fix it.

(Bitbake rev: a5e084a266f63c2fd370122327615e49beaeb94e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-15 13:32:06 +01:00
Richard Purdie
2fa5d98f52 bitbake: runqueue: Fix data corruption problem
This was overwriting data in the parent which caused all kinds of
odd/weird failures.

(Bitbake rev: 4c5aeb424247a9d0c907524ffacd9c61fcdc0852)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-15 13:32:06 +01:00
Richard Purdie
a3c53bff11 bitbake: tests/runqueue: Fix tests
There were paths being accidentally included in some of the hashserv tests. Remove
that and update the hashes so the tests work independently of paths.

(Bitbake rev: 6ddb9f09cb60c2354fa6a67cce412c4dc1e7dc2d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-15 08:43:22 +01:00
Richard Purdie
93e4dc5f11 bitbake: runqueue: Ensure we clear the stamp cache
When the task hashes change we need to ensure the stampcache is cleared out
else tasks don't rerun when they should as we're basing decisions on stale
cache data.

(Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
64d0493d97 bitbake: runqueue: Add missing setscene task corner case
We weren't marking this special case of setscene task as buildable leading to
runqueue task failures.

(Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
0993752e06 bitbake: runqueue: Add further debug information
Further testing shows we should test some extra datastructures to help pinpoint logic
errors more precisely. This adds some further data structure sanity checks.

(Bitbake rev: 83c4370b25c3a14cc946965c5c5f83ea28f488a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
0d73901c98 bitbake: runqueue: Drop debug statement causing performance issues
This debug statement could result in a long list of tasks which when
repeatedly sent over our IPC, slowed down the builds immensely. Remove
it in favour of other more targeted debugging added recently, bringing
back some lost performance, particularly on builds with large numbers
of tasks.

(Bitbake rev: 85fe627fdb6510f0942917964386fad9d8c479c8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
977a293f2f bitbake: runqueue: Fix event timing race
The event from the task notifiing of hash equivalency should only be processed
when the task completes. This can otherwise result in a race where a dependent
task may run before the original task completes causing various failures.

To make this work reliably, the code had to be restructured quite a bit.

(Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
3fb90d0fff bitbake: runqueue: Recompute holdoff tasks from scratch
The changed_setscene variable here is just odd and not needed. Worse,
it could prevent some tasks from being removed from the holdoff tasks
list. The list is being rebuilt and should work as intended just from
the other data, this is a leftover from previous versions of the code
as far as I can tell.

(Bitbake rev: 030b9f2b3ce6ed40e79304eb0ffee6c6613f43be)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
de06716c76 bitbake: runqueue: Improve scenequeue debugging
Whilst we had good runqueue failure mode debug, it hadn't adapted to the
scenequeue changes. Run the scenequeue sanity tests at the end of
a build and output the results regardless of whether all setscene tasks
completed or not. This *massively* improves the ability to debug runqueue
problems.

(Bitbake rev: b9b2177473c0b95a23bd519a201e1d2ba101c6c1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
d842d2507a bitbake: runqueue: Fix next_buildable_task performance problem
Looking at the profile information, a lot of time is being spent in
next_buildable_task. This is probably due to the generator expressions
not working well with the empty test.

The easiest way to improve things is to switch to using set manipulations.
We also don't need to update self.buildable the way the original code did
as we don't rely on that anywhere.

(Bitbake rev: 3bcf9ad4964b7e42d1a02ce231e9db42a81ead2a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
99db00b836 bitbake: runqueue: Wait for covered tasks to complete before trying setscene
If tasks are in the covered list of tasks for a given setscene task,
it needs to wait for those to complete before we can start.

(Bitbake rev: fdee640c26750b852eb68f5c80437377aa300ed8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
5f5bc81b3e bitbake: cooker: Improve hash server startup code to avoid exit tracebacks
At exit the hashserv code was causing tracebacks as join() wasn't
being called from the thread that started the process. Ensure that
the hashserver is started from the pre_serve hook which is the
final thread the cooker runs in. This avoids the traceback at the
expense of some horrific poking into data stores which will ultimately
need improving through a proper API.

(Bitbake rev: 05888700e5f6cba48a26c8a4c447634a28e3baa6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
abda938d66 bitbake: tests/runqueue: Add further hash equivalence tests
Add some extra hash equivalence runqueue tests based on recent scenarios
that caused problems during testing.

(Bitbake rev: 373b085ead992a725b2230ededd992b4c61a1a05)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
c86ae704d3 bitbake: runqueue: Improve setscene task handling logic
The previous tasks_covered and tasks_notcovered were basically unstable
data structures. We couldn't always tell whether tasks should be covered
or not when trying to repair the sturcture if sstate tasks reran.

In the end its simpler to throw the lists away and rebuild them based upon
current data rather than trying to patch it adhoc. This turns out to be
simpler and much more reliable and I've much more confidence in this code.

(Bitbake rev: 52ee2ba2c617d928569f5afa404925c8b6f317bc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Richard Purdie
18c109d6f5 bitbake: runqueue: Fix corruption issue
We need to copy this set, not modify the original else all kinds
of weird and bad things break, mostly from circular references.
We'll not go into how much sleep I lost tracking down the fallout
from this.

(Bitbake rev: 49927546d2b306830c98f6f9da4a6ad828f6a3a6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-14 17:28:23 +01:00
Mark Hatle
18b8e2e104 bitbake: layerindexlib: Fix parsing of recursive layer dependencies
[YOCTO #13447]

When running bitbake-layers layerindex-fetch from 'master', there is a
circular dependency between meta-oe and meta-python.  This triggered a maximum
recursion depth exception.

To fix the exception, as we walk down a branch (depth first search), we track
the layers we've already seen.  If we are about to recurse into a layer we've
already seen we report a warning and then stop recursion.

(Bitbake rev: d6155d095513be3f500d089c4ed4c4b89949d560)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-13 09:38:26 +01:00
Robert Yang
e1f589f044 bitbake: knotty: Fix for the Second Keyboard Interrupt
Fixed:
$ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p
Press the first Ctrl-C when the parsing process is at about 50%:

Keyboard Interrupt, closing down...

Then presss the second Ctrl-C:

  File "/path/to/bitbake/bitbake/lib/bb/ui/knotty.py", line 619, in main
    event = eventHandler.waitEvent(0.25)
  File "/path/to/bitbake/lib/bb/server/process.py", line 591, in waitEvent
    self.eventQueueNotify.wait(delay)
  File "/usr/lib/python3.5/threading.py", line 549, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.5/threading.py", line 297, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

Capture the second KeyboardInterrupt during stateShutdown is running can fix
the problem. There may be still tracebacks for the third KeyboardInterrupt, but
I'm leaning to not fix it since we aimed for supporting 2 KeyboardInterrupts
only.

(Bitbake rev: 8c26b451f22193ef1c544e2017cc84515566c1b8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-08 10:22:36 +01:00
Robert Yang
016b91b334 bitbake: cooker: Cleanup the queue before call process.join()
Fixed:
$ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p
Press *one* Ctrl-C when the parsing process is at about 50%, then the processes
are not exited:

Keyboard Interrupt, closing down...

Timeout while waiting for a reply from the bitbake server

It hangs at process.join(), according to:

https://docs.python.org/3.7/library/multiprocessing.html

Cleanup the queue before call process.join() can fix the problem.

(Bitbake rev: 3eddfadd19b2ce4c061861abf0c340e3825b41ff)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-08 10:22:36 +01:00
Richard Purdie
8fc23b8c1c bitbake: bitbake: Bump version to 1.43.1 for API changes
(Bitbake rev: f43778c2d19e70d4befd483b06aaf247fc65c799)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
fd79638046 bitbake: tests/runqueue: Add hashserv+runqueue test
Add a test which tests the runqueue adaptations for hash equivalency.

(Bitbake rev: 477321d0780df177c1582db119c2bb6795912fc6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
347c205602 bitbake: build/utils: Drop bb.build.FuncFailed
Its hard to see what this exception adds in the current codebase. The logfile
attribute is effectively ignored, the exception doesn't serve a defined
purpose and mostly seems to be worked around.

Remove it entirely. If this does cause output problems, we'll figure
out better ways to address those.

(Bitbake rev: cfeffb602dd5319f071cd6bcf84139ec77f2d170)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
4fbb862cdc bitbake: siggen: Clean up task reference formats
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.

This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.

Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.

(Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
40a5e193c4 bitbake: runqueue: Clean up BB_HASHCHECK_FUNCTION API
This function uses an old API which uses offsets into lists as a communication
mechanism. Update the API to use "tid" which is used universally in runqueue now.

We can also add kwargs support to the funciton definition to drop some of the
backwards compaiblility hoops we had to jump though with different function
argument combinations.

(Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
43d37a6eaf bitbake: hashserv: Switch from threads to multiprocessing
There were hard to debug lockups when trying to use threading to start
hashserv as a thread. Switch to multiprocessing which doesn't show the
same locking problems.

(Bitbake rev: be23d887c8e244f1ef961298fbc9214d0fd0968a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
d40d7e4385 bitbake: hashserv: Use separate threads for answering requests and handling them
Experience with the prserv shows that having two threads, one accepting
and queueing connections and one handling the requests leads to much
more reliable behaviour than having everything in a single thread.

(Bitbake rev: a03d60671a53d9ff70e07cc42fe35f6f8776dac2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
94d5c61c9a bitbake: prserv: Use a memory journal
We've seen PR Server timeouts on the autobuilder, this is likely from the
journal being blocked on disk IO generated by the build.

Since we're running with synchronous off, we may as well put the journal
into memory and avoid any IO related stalls.

(Bitbake rev: ee3fc6030e653f3244b065fc89aafd2a7c36ae04)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
f9f308250a bitbake: hashserv: Turn off sqlite synchronous mode
We're seeing performance problems with hashserv running on a normal build
system. The cause seems to be the large amounts of file IO that builds involve
blocking writes to the database. Since sqlite blocks on the sync calls, this
causes a significant problem.

Since if we lose power we have bigger problems, run with synchronous=off
to avoid locking and put the jounral into memory to avoid any write issues
there too.

This took writes from 120s down to negligible in my tests, which means
hashserv then responds promptly to requests.

(Bitbake rev: 7ae56a4d4fcf66e1da1581c70f75e30bfdf3ed83)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
ca04aaf7b5 bitbake: cooker/hashserv: Allow autostarting of a local hash server using BB_HASHSERVE
Its useful, particularly in the local developer model of usage, for
bitbake to start and stop a hash equivalence server on local port,
rather than relying on one being started by the user before the build.

The new BB_HASHSERVE variable supports this.

The database handling is moved internally into the hashserv code so that
different threads/processes can be used for the server without errors.

(Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
d9aafb8507 bitbake: runqueue: Improve determinism
Whilst this isn't strictly necessary, its helpful if the log output is
consistent and its also helpful if bugs either appear or don't appear
for a specific configuration. Ensuring the various iterations we make
are deterministic (sorted) helps with this.

(Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
7df31ff368 bitbake: runqueue: Enable dynamic task adjustment to hash equivalency
There is a compelling usecase for tasks being able to notify runqueue
that their "unihash" has changed. When this is recieved, the hashes of
all subsequent tasks should be recomputed and their new hashes checked
against existing setscene validity. Any newly available setscene tasks
should then be executed.

Making this work effectively needs several pieces. An event is added
which the cooker listen for. If a new hash becomes available it can
send an event to notify of this.

When such an event is seen, hash recomputations are made. A setscene
task can't be run until all the tasks it "covers" are stopped. The
notion of "holdoff" tasks is therefore added, these are removed from
the buildable list with the assumption that some setscene task will
run and cover them.

The workers need to be notified when taskhashes change to update their
own internal siggen data stores. A new worker command is added to do this
which will affect all newly spawned worker processes from that worker.

An example workflow which tests this code is:

Configuration:
BB_SIGNATURE_HANDLER = "OEEquivHash"
SSTATE_HASHEQUIV_SERVER = "http://localhost:8686"

$ bitbake-hashserv &
$ bitbake automake-native
$ bitbake autoconf-native automake-native -c clean
$ bitbake m4-native -c install -f
$ bitbake automake-native

with the test being whether automake-native is installed from sstate.

(Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
40eb5b344b bitbake: siggen: Convert to use self.unitaskhashes
Rather than metadata driven sqlite databases for communication, use
bitbake's unitaskhashes variable instead.

(Bitbake rev: a0d941c787cf3ef030d190903279d311bc05d752)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
de143e0db6 bitbake: siggen: Add new unitaskhashes data variable which is cached
We need to preserve unihash task hashes between runs. Use the new SimpleCache
class to create such a class within the signature generator which is loaded
at init time and saved when builds complete. The default is unpopulated but
metadata sig handlers can populate this cache.

(Bitbake rev: 1f326f2c29c2664a5daaeeb0c1fd332630efbdba)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
53bb939383 bitbake: runqueue: Improve scenequeue processing logic
Rather than a special copy of the data structure which we change, compute
the logic using set operations from other data we have. This means
we can add tasks back into the scenequeue without having to worry about
reversing operations on this variable with all the potential bugs that
might involve.

(Bitbake rev: b707d0cbc25fa336a1e95ff588f1ea37eee063eb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
5f56244092 bitbake: cache: Add SimpleCache class
This adds a simple version of the MultiProcessCache which can be used to
save and load cache data, useful for a new usecase we have in
sigdata/runqueue.

(Bitbake rev: 19a6e35600ae6d2d1bcecca6e68ab8c37674774e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Richard Purdie
db31374efa bitbake: siggen: Import unihash code from OE-Core
This code is closely tied with the hash server in bitbake and also means
we can't relibably test the hashserv runqueue functionality without OE
metadata. Moving this to bitbake as a MixIn class makes most sense
and encourages code collaboration and reuse as well as enabling easier
and more accurate testing of the APIs.

(Bitbake rev: 7bb79099a6c1b463d6ae9226c4cab5e76a965675)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Joshua Watt
9802b2e650 bitbake: hashserv: SQL Optimizations
Implements a number of optimizations to the SQL used in the hash
equivalence server:

 1) Two indexes are created for the two methods (method, taskhash and
    method outhash) by which rows are found in order to speed up the
    lookup
 2) An extra SELECT to lookup the just inserted row was removed. This
    SELECT is unnecessary since all of the information about the newly
    inserted row is already available.
 3) A uniqueness constraint was added to the table. This should allow
    the server to be multithreaded in the future since duplicate inserts
    can be detected (and ignored). This change requires bumping the
    database version to '2', since a uniqueness constraint can't be
    added to an existing table.
 4) Some comments are added to clarify the trick SELECT statement used
    when inserting new equivalent hashes

(Bitbake rev: 7aec8632e67b4f0ab7b72692c40a42f6926608c3)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Chris Laplante via bitbake-devel
1a4e4fb6b0 bitbake: fetch2/wget: avoid 'maximum recursion depth' RuntimeErrors when handling 403 codes
The code says that some servers respond with 403 codes when they really
mean 405 codes. But we still need to account for legitimate 403 codes.

Before this change, I noticed that sstate mirror checking was taking a
very long time when I purposely entered incorrect credentials into my
.netrc file for our sstate mirror. Instrumenting the code, I discovered
tracebacks like the following for every mirror access attempt:

    File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus
      fetcher.checkstatus()
    File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus
      ret = try_mirrors(self, self.d, ud, mirrors, True)
    File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors
      ret = try_mirror_url(fetch, origud, uds[index], ld, check)
    File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url
      found = ud.method.checkstatus(fetch, ud, ld)
    File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus
      opener.open(r)
    File "/usr/lib/python3.5/urllib/request.py", line 472, in open
      response = meth(req, response)
    File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
      'http', request, response, code, msg, hdrs)
    File "/usr/lib/python3.5/urllib/request.py", line 504, in error
      result = self._call_chain(*args)
    File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
      result = func(*args)
    File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
      unverifiable=True))
    File "/usr/lib/python3.5/urllib/request.py", line 472, in open
      response = meth(req, response)
    File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
      'http', request, response, code, msg, hdrs)
    File "/usr/lib/python3.5/urllib/request.py", line 504, in error
      result = self._call_chain(*args)
    File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
      result = func(*args)
    File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
      unverifiable=True))
    ...  (repeats until recursion depth is reached)

Solution is to make sure we only attempt the GET request once when handling 403/405 error codes.

(Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-27 22:46:09 +01:00
Mads Andreasen
bebeed310b bitbake: fetch2/npm: Use npm pack to download node modules instead of wget
Using npm pack to download the main node module and its dependencies
allow for the use of private npm modules and access to them via .npmrc

(Bitbake rev: e5eda3871893e4eadeb311aeb997e183675598f4)

Signed-off-by: Mads Andreasen <mads@andreasen.cc>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-27 22:46:09 +01:00
Daniel Ammann
d7a026d434 bitbake: toaster: Sync list of fs_types with oe-core
(Bitbake rev: 1dddfe3512b6390958abb91b21f074568ae4e8db)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-27 22:46:08 +01:00
Richard Purdie
ae5615000b bitbake: siggen: Fix handling of tainted sig files
The addition of some debugging code meant that comparisions between sig
files with a taint and without a taint weren't working. Tweak the logic
to avoid tracebacks if one side doesn't have a taint.

(Bitbake rev: f5ea06fc2b6713c9f8e85ecf7cb981ae9a84d896)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-19 16:20:28 +01:00
Robert Yang
b70993cf92 bitbake: cache: Create a symlink for current cachefile
So that people or other tools can easily know which one is being used, just
like what we did for run.do_task and log.do_task, otherwise, we have no way
to know it. I usually use "ls -t", but it isn't reliable since the one which
is being used may not the latest one.

(Bitbake rev: cf286dff653eed542bf347ca46234c224944d5b0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-19 08:45:08 +01:00
Richard Purdie
9881c532c8 bitbake: runqueue: Fix non setscene tasks targets being lost
If you specify both setscene and non-setscene tasks on the commandline, the
non-setscene tasks could be missed, e.g. "bitbake X:do_patch X:do_populate_sysroot"
and do_patch would fail to run.

Fix the problem in runqueue and add a testcase.

(Bitbake rev: 75292fdec5d9c0b5b3c554c4b7474a63656f7e12)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-18 12:16:19 +01:00
CHerzig@Gauselmann.de
3b6e8a2fe8 bitbake: fetch2/clearcase: Fix class import errors
(Bitbake rev: 9a5152fa4613a1164cbf2a0248460e75207b2624)

Signed-off-by: Christian Herzig <cherzig@gauselmann.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16 14:19:15 +01:00
Richard Purdie
efc22e7555 bitbake: tests/runqueue: Allow common sstate tasks to become valid
As the logic in bitbake improves, the logic in the tests needs to as well.

Afer we built a task for the first time, allow its setscene hash verification
status to change, mirroring what would happen in a multiconfig build.

(Bitbake rev: 27ec2e69ab3e32972caf8b072b2945736696d83d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16 13:53:17 +01:00
Richard Purdie
1069c36417 bitbake: runqueue: Optimise multiconfig with overlapping setscene
Currently if a multiconfig build contains different configurations which
have overlapping sstate artefacts, it will build them multiple times.
This is clearly suboptimal and not what users want/expect.

This adds code to detect this and stall all but one of the setscne tasks
so that once its built, it can be found by the other tasks.

We take care to iterate the multiconfigs in order so try and avoid
dependency loops. We also match on PN+taskname+taskhash since this is
what we know sstate in OE-Core would use. There are some tasks even within
a multiconfig which match hashes (mostly do_populate_lic tasks) but those
have a much higher chance of circular dependency so aren't work attempting
to optimise.

If a deadlock does occur the build will be slower but there is code to
unbreak such a deadlock so it hopefully doens't break anything.

Comments are injected into the test tasks so they have different task
hashes and a new test for this optimisation is added.

(Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16 13:53:17 +01:00
Richard Purdie
5333f31fc7 bitbake: runqueue: Abstract hash verification function
Pull the common pieces of the hash verification code into a single function
and reduce code duplication.

(Bitbake rev: d0c39e05cef841c6f29cc6c919df6cbf271a9bda)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16 13:53:17 +01:00
Richard Purdie
be564f82ca bitbake: runqueue: Whitespace fix
Fix some unwanted extra indentation.

(Bitbake rev: 460a5c2e3e1d72f2da16fbc96832fadc82e72c52)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16 13:53:17 +01:00
Richard Purdie
3b01998281 bitbake: runqueue: Simplify some convoluted logic
This was left from when task IDs complicated the code, simplify.

(Bitbake rev: ae36b5c693bb9f13c88199e78e3c31616852eafb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-16 13:53:17 +01:00