Commit Graph

5740 Commits

Author SHA1 Message Date
Richard Purdie
2f12a20935 bitbake: server/process: Handle error in heartbeat funciton in OOM case
We've seen cases where an OOM error causes bitbake server to hang:

9171 02:21:09.127810 Command Completed
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in <module>
    bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer
    server.run()
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run
    ret = self.main()
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main
    ready = self.idle_commands(.1, fds)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands
    bb.event.fire(heartbeat, self.cooker.data)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
    fire_class_handlers(event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
    ret = handler(event)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats
    write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval")
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data
    output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8')
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

We need to wrap the calls in the same high level wrapper as idle function calls
and trigger an exit upon an unhandled exception.

(Bitbake rev: 74042b5b89d5a170013fc1a327ce3a6530fbf7d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-18 23:53:15 +01:00
Armin Kuster
ed2e3b739a bitbake: hashserver/client.py: drop unused imports
(Bitbake rev: 9dc77266085c605b108641a9d76ac4dbdc064c34)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-14 07:57:57 +01:00
Armin Kuster
c0c16809ef bitbake: hashserv/server.py: drop unused imports
remove unused vars.

(Bitbake rev: 3287d28a506f67abd192799e61ef28e74ce7002d)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-14 07:57:57 +01:00
Alexander Kanavin
37792f1410 bitbake: fetch2/wget: when checking latest versions, consider all numerical directories
Previously the regex was maching x.y, but wasn't matching x, which is a problem
e.g. here:
https://download.gnome.org/sources/epiphany/
(the new gnome version scheme adds 40-series at the end).

(Bitbake rev: c03101576f447263ea38e8464210d3a3a2c27226)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-14 07:57:57 +01:00
Richard Purdie
49da1ed2dc bitbake: bitbake: Switch to post release version number 1.51.0
(Bitbake rev: 97a64d12f70eb02f1d35b4ffefb291b80ca8c425)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:08 +01:00
Harald Brinkmann
e155a81ab6 bitbake: fetch/svn: Fix parsing revision of SVN repos with redirects
svn was printing a message when encountering HTTP redirects.
This confused the revision parser.

(Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146)

Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:08 +01:00
Paul Barker
81b55a050d bitbake: prserv: Handle requests in main thread
The prserver process is cleanly separated from the main bitbake process
so requests can be handled in the main thread. This removes the need for
a request queue and a separate request handling thread.

(Bitbake rev: 6b09415bed6b5e7c12aaf39b677d9ef72844e233)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:07 +01:00
Paul Barker
d66a1d83f5 bitbake: prserv: Extract daemonization from PRServer class
The code to start the prservice process as a daemon is extracted out of
the PRServer class and simplified. This makes the PRServer class easier
to modernise as it no longer needs to worry about process management.

(Bitbake rev: 39c7c158c52157b18f5ccbbd673e3298e6402f52)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:07 +01:00
Paul Barker
802024fd2a bitbake: prserv: Use multiprocessing to auto start prserver
We can use the modern multiprocessing support in Python instead of
manually using fork to start the prserver process. To do this we need
to set up the signal handlers for the prserver process in the
work_forever function (which is now used as the main function for this
process).

The old code to start the prserver process using fork is not removed in
this commit as it is tightly intertwined with the daemonization code
which will be refactored in a following commit.

(Bitbake rev: b3da56240c0f92efab1c0b293738c35c0f1ee6ab)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:07 +01:00
Devendra Tewari
581233a798 bitbake: lib/bb: Add bb.utils.rename() helper function and use for renaming
os.rename can fail for example an incremental build in Docker fails with:

OSError: [Errno 18] Invalid cross-device link

when source and destination are on different overlay filesystems.

Rather than trying to fix every call site, add a wrapper in bb.utils
for renames. We can then handle cross device failures and
fall back to shutil.move. The reason os.rename is still used is
because shutil.move is too slow for speed sensitive sections of code.

[YOCTO #14301]

(Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b)

Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:04:59 +01:00
Richard Purdie
93aaa5e994 bitbake: runqueue: Handle deferred task rehashing in multiconfig builds
If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig
build), we need to ensure that the hash change propagates through to all the tasks
else the build will run multiple copies of the task, sometimes with oddly differing
results as the outhashes of native tasks built in differing locations can confuse
things.

(Bitbake rev: 2db571324f755edc4981deecbcfdf0aaa5a97627)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Richard Purdie
76891afd76 bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue
We were testing the validity of deferred tasks setscene status "up front" which
is very unlikely to succeed and leads to cache invalidation issues. With the
change to rebuild the deferred task list, this status becomes out of sync. The
result was tasks being executed when they should not have been leading to extra
work for the build unnecessarily.

Instead, don't process validity status for deferred tasks and assume their
data will become available. If it doesn't, this will now result in a build
error as the setscene task will fail and the main task will run instead.

In theory we could try and track the state changes in the deferred list and
re-test validity then but I'm not sure it is worth the effort when the other
code path and errors in setscene tasks will give a pretty good idea of what
is happening anyway.

(Bitbake rev: edcafac13b3b241b6687419e59018d21811507a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Joshua Watt
f3e1a668fb bitbake: knotty: Re-enable command line logging levels
The "-l" command line options to enable specific logging domains wasn't
working with the switch to structured logging because they were only
being used to set the legacy logging domains. Fix this by implementing
the logic to parse the user options into the logging configuration.

(Bitbake rev: 005fc7a8c588d0b0bca382469645cbf481ad8e30)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Przemyslaw Gorszkowski
6da327c788 bitbake: fetch/s3: Add progress handler for S3 cp command
Adds progress support for fetchers from S3.

(Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Przemyslaw Gorszkowski
a854068b52 bitbake: progress: LineFilterProgressHandler - Handle parsing line which ends with CR only
S3 commands need to handle different CR only line endings, update the handler
to cope with this.

(Bitbake rev: 3f7b9c1b429a4c68240e80832a8ef93ee210e5ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Paul Barker
ebb7adac03 bitbake: prserv: Add connect function
This function abstracts the setup of a PR service client connection so
that openembedded-core doesn't need to be updated any time the details
are changed.

(Bitbake rev: d892287b31f81b075983ba500be265f75b53df64)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Paul Barker
ad5c898808 bitbake: prserv: Drop unused dump_db method
(Bitbake rev: ecb7bf34eac02ff58dbc27b3768ceaf4adb1c9cd)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Paul Barker
ea647326d1 bitbake: prserv: Drop obsolete python version check
Bitbake no longer supports Python 2 so this version check is obsolete.

(Bitbake rev: 45eb6c6e124e507012df9c288f1fbde0e7899e5d)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Paul Barker
421e86e7ed bitbake: hashserv: Refactor to use asyncrpc
The asyncrpc module can now be used to provide the json & asyncio based
RPC system used by hashserv.

(Bitbake rev: 5afb9586b0a4a23a05efb0e8ff4a97262631ae4a)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Paul Barker
244b044fd6 bitbake: asyncrpc: Common implementation of RPC using json & asyncio
The hashserv module implements a flexible RPC mechanism based on sending
json formatted messages over unix or tcp sockets and uses Python's
asyncio features to build an efficient message loop on both the client
and server side. Much of this implementation is not specific to the
hash equivalency service and can be extracted into a new module for
easy re-use elsewhere in bitbake.

(Bitbake rev: 4105ffd967fa86154ad67366aaf0f898abf78d14)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Paul Barker
1023671823 bitbake: hashserv: Use generic ConnectionError
The Python built-in ConnectionError type can be used instead of a custom
HashConnectionError type. This will make code refactoring simpler.

(Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Diego Sueiro
cda69e0166 bitbake: layerindex: Add --fetchdir parameter to layerindex-fetch
Introduce --fetchdir parameter to layerindex-fetch enabling users to choose
the directory to fetch the layers different from BBLAYERS_FETCH_DIR.

[YOCTO #14347]

(Bitbake rev: 784a904faffac723ddf58ba765b9dd11ac068de5)

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-23 10:13:13 +01:00
Diego Sueiro
9fae310dd2 bitbake: layerindex: Fix bitbake-layers layerindex-show-depends command
Running 'bitbake-layers layerindex-show-depends meta-filesystems' fails with:
```
Traceback (most recent call last):
  File "<...>/poky/bitbake/bin/bitbake-layers", line 93, in <module>
    ret = main()
  File "<...>/poky/bitbake/bin/bitbake-layers", line 86, in main
    return args.func(args)
  File "<...>/poky/bitbake/lib/bblayers/layerindex.py", line 209, in do_layerindex_show_depends
    self.do_layerindex_fetch(args)
  File "<...>/poky/bitbake/lib/bblayers/layerindex.py", line 182, in do_layerindex_fetch
    args.shallow)
AttributeError: 'Namespace' object has no attribute 'shallow'
```

Initialize the shallow attribute to fix it.

(Bitbake rev: 71f095c147fe6aa7b5e6272002e0498cf9494256)

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-23 10:13:13 +01:00
Niels Avonds
d12a06cd49 bitbake: fetch/gitsm: Fix crash when using git LFS and submodules
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.

This issue was introduced in this
commit: 977b7268bf

[YOCTO #14283]

(Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)

Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Ross Burton
01066a584a bitbake: bitbake-server: ensure server timeout is a float
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer.  There's some type confusion here:

bitbake-server is called with a string representation of the timeout,
which may be None.  If the timeout is not set, pass 0 instead of None.

Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.

[ YOCTO #14350 ]

(Bitbake rev: c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Richard Purdie
d08d362cf9 bitbake: tinfoil/data_smart: Allow variable history emit() to function remotely
We can't access the emit() function of varhistory currently as the datastore parameter
isn't handled correctly, nor is the output stream. Add a custom wrapper for this
function which handles the two details correctly.

(Bitbake rev: ba0fa084ccd2b1ade96425d158fd31e49e42f286)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
46654e14a5 bitbake: tests/fetch: add tests for local and remote "noshared" git fetching
(Bitbake rev: e0267fe43bda208856af939b17e39beb9e5586c3)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
d274cf0cf8 bitbake: tests/fetch: deduplicate local git testing code
Purely cosmetic change that probably improves the code.

(Bitbake rev: 9c0733f0062f3cf19514c891cc06c9a8e0db429b)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
2cb6ce7788 bitbake: fetch/git: add support for disabling shared clones on unpack
By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour.

Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.

With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.

The switch is a global bitbake switch and will affect all git urls.
Build systems that need "noshared" most likely need it for everything
they do with git.

(Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Mikko Rapeli
33650ffdc7 bitbake: bitbake: tests/fetch: remove write protected files too
For some reason several git-annex files in Debian 10 buster
are read-only and removing them with "rm -rf" fails.

Fixes test failures like:

$ bitbake-selftest
...
rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied
rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied
EE..................................ssss.sssssssssssssss.sssss.......................................................................................................
======================================================================
ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex
    fetcher, ud = self.fetch_shallow(uri)
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow
    bb.utils.remove(ud.clonedir, recurse=True)
  File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove
    subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1.

Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod
the whole temporary directory which should cover any directory name differences between
different git-annex versions. Fixes tests failing after chmod call:

Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/

(Bitbake rev: 7729ef2983c72867e99fad82d671069ba5cb32b2)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Mikko Rapeli
490fb73e34 bitbake: bitbake: tests/fetch: fix test execution without .gitconfig
A CI user validating changes does not have any git push rights or
even a .gitconfig file so fix tests so that they run
by setting the user.name and user.email for the repo before
committing changes.

Fixes errors like:

ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist
    self.add_empty_file('a')
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file
    self.git(['commit', '-m', msg, path], cwd)
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git
    return bb.process.run(cmd, cwd=cwd)[0]
  File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run
    raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128:

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

(Bitbake rev: 57c0811f1ee19b6619f4840a39e01e3cb98c34c4)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Richard Purdie
7ea37b9291 bitbake: runqueue: Fix deferred task issues
In a multiconfig situation there are circumstances where firstly, tasks
are deferred when they shouldn't be, then later, tasks can end up as
both covered and not covered.

This patch fixes two related issues. Firstly, the stamp validity checking
is done up front in the build and not reevaulated. When rebuilding the
deferred task list after scenequeue hash change updates, we need therefore
need to check if a task was in notcovered *or* covered when deciding to
defer it. This avoids strange logs like:

NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene)
NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch

where tasks have run but are then deferred.

Since we're recalculating the whole list, we also need to clear it before
iterating to rebuild it. By ensuring covered tasks aren't added to the
deferred queue, the covered + notcovered issue should also be avoided.
in the task deadlock forcing code.

[YOCTO #14342]

(Bitbake rev: 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Richard Purdie
ca07d22405 bitbake: bitbake: Update version to 1.50.0 stable release series
(Bitbake rev: e70b925ba98fd4fedf3940d141a4210c953087ca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-09 14:20:01 +01:00
Richard Purdie
9982b2c0f2 bitbake: runqueue: Further fixes for confused setscene tasks
There is further evidence of tasks ending up being "covered" and "notcovered"
which shouldn't happen and is bad. The code that caused this problem last
time appears to have issues where stamps for tasks already exist.

Split out the setscene stamp checking code to a separate function and
use this when checking "hard dependencies" (like pseudo-native) so
that if the stamps exist and it will be "covered", it is not put on
the notcovered list.

(Bitbake rev: a1848a481e36b729c8e4130c394b1d462d4b488a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:31:24 +01:00
Michael Opdenacker
62880ffb17 bitbake: doc: Update links to documentation
(Bitbake rev: ed8e1fd4cf9d5ac8a8203638add99d686b4b3521)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Richard Purdie
86ec266d91 bitbake: runqueue/event: Add an event for notifying of stale setscene tasks
Use the new functionality in build.py to identify stale setscene tasks
and send an event to the metadata listing them. The metadata then
has the option of performing cleanup operations if it thinks that
appropriate.

(Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Richard Purdie
ab55f8bfec bitbake: build: Add find_stale_stamps function
Add a new function which compares the stamp filename we want (including
taskhash) with what is in the stamp directory (using the clean mask).
This tells us which stamp files are stale and are due to be rerun.

(Bitbake rev: b126a53882d202e4df0f9661303355c9fe9ec80e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Tomasz Dziendzielski
a9098e6423 bitbake: codeparser: Fix TypeError in bitbake debug mode
Commit 75f87db413 fixed the confusion between bitbake and python logger
but in codeparser still old method of setting debug level was used
causing TypeError, because debug level value was incorrectly returned
and assigned to event.msg.
| File "./bitbake/lib/bb/ui/knotty.py", line 660, in main
|   event.msg = event.fn + ': ' + event.msg
| TypeError: can only concatenate str (not "int") to str

[YOCTO #14298]

(Bitbake rev: bec9ea7ab28a8dfad1a6010ca5c6ec691754748d)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 21:47:42 +00:00
Richard Purdie
ddacd8c66f bitbake: bblayers/query: Update to handle REQUIRED_VERSION
Ii is unclear whether any changes are needed to bblayers are
needed to handle the extra data from REQUIRED_VERSION. Update
to at match the new API, at present it doesn't look necessary
to handle the required version data.

(Bitbake rev: 53c30efec4099035d19e6717059dfceff8ff88fd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
cb3cbd8bb2 bitbake: bitbake: cooker: Add REQUIRED_VERSION checks
Add logic to check that if REQUIRED_VERSION has been set that the cooker
class method findBestProviders properly handles the case where the
REQUIRED_VERSION has not been found.

Fixes [YOCTO #10096]

(Bitbake rev: 5df201d746f26154213e6d15d2721cd35b38b59e)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
889a7ce8de bitbake: bitbake: lib: add Required Version to Bitbake --show-versions command
This commit adds a new column to the Bitbake --show-versions command
called Required Version. This column will display any packages which
have a REQUIRED_VERSION successfully set.

Fixes [YOCTO #10096]

(Bitbake rev: 90c7d1815e41243323d32b9dbb865757a922578a)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
78fcf6831a bitbake: bitbake: providers: check for REQUIRED_VERSION in _filterProviders
Before the REQUIRED_VERSION variable was introduced the
PREFERRED_VERSION variable allowed for a fallback to the next most
suitable version.

Since REQUIRED_VERSION does not allow a fallback to a different version
implement a check in the _filterProviders function to make sure that
if a requested REQUIRED_VERSION is not found then the function returns
no eligible providers have been found.

Fixes [YOCTO #10096]

(Bitbake rev: c41386b78aa53e0bf081cd973c950b88126670a7)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
3752782cb7 bitbake: bitbake: providers: introduce logic for REQUIRED_VERSION variable
This commit adds checks during the findPreferredProvider function for a
new variable REQUIRED_VERSION. This can be set, in exactly the same
manner as PREFERRED_VERSION, on a per package basis to enforce the use
of a particular version of a package.

REQUIRED_VERSION is similar in behaviour to PREFERRED_VERSION except if
the version specified by REQUIRED_VERSION is not found an error occurs
and the execution of Bitbake stops.

Fixes [YOCTO #10096]

(Bitbake rev: 5cbf6d95fc1009e78e7d0745a49e0bf418b37abb)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
70f90dcfd7 bitbake: bitbake: providers: consistent single line formatting between functions
(Bitbake rev: e8c74ac8e589ee44d862b71fd63dd75ab303d3b5)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
04bb096170 bitbake: bitbake: providers: remove unneeded logging call
The logging line for when no eligible providers are found only adds
unnecessary noise and is not required. As the stack unwinds better
logging occurs which satisfactorily describes the no eligible providers
error condition.

(Bitbake rev: 933c9df919f7947f515d843199cce2621cf8195f)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Richard Purdie
cca5433baf bitbake: tests/color: Fix event register to pass the datastore
Adding the "if d is None" to the event handling code meant some of
these tests stopped working. The reason is that len(d) was zero but
not equal to None.

Passing the data object to the register() function in the test
correctly registers the event handler and avoids the problem, it
just happened to work previously, incorrectly.

(Bitbake rev: 426eb83c6668d82a2ebaca6c672db131e37c11da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12 15:28:57 +00:00
Richard Purdie
22d7989e8f bitbake: event: Fix another 'if d' test to test for None explictly
This should have been fixed in the previous patch, well spotted Chris!

(Bitbake rev: c3f6fee42bfa23f23f167cb29f0cfa05ac2fa197)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 21:38:32 +00:00
Richard Purdie
b1a5b43735 bitbake: event: Fix multiconfig event handler change performance regressions
There were two issues in this code, firstly the code could stack duplicates
in the variable, secondly, calling "if data" caused the datastore to compute
len(data) which is comparitively expensive. Checking "if data is not None"
is much much faster/cheaper.

The issue was clear from "bitbake -p -P" output where the time in register()
showed large amounts of time in the __len__ function of the datastore.

(Bitbake rev: 9298d08b7dcf9d79f54536f87686d65c4ad7deb9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Manuel Leonhardt
e6ec2b4dcc bitbake: tinfoil: Honor quiet when parsing recipes
When using parse_recipes, honor quiet so that scripts and custom plugins
for recipetool are able to mute progress bars from bitbake that would
otherwise print to STDOUT.

(Bitbake rev: ab9d26c4847a062cadaae5fb8caac0ead5f958db)

Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Tomasz Dziendzielski
5386b3db50 bitbake: runqueue: Print pseudo.log if fakeroot task failed
Currently if pseudo fails we can only see the path to pseudo.log. If we
have no access to server and can only rely on bitbake log then debugging
becomes impossible. This printing needs to be added in runqueue level,
not inside task execution, because in some cases task fails with pseudo
abort really early and we don't even see any log.

In this change I'm adding pseudo log printing in every fakeroot task
failure that logged `mismatch`, `error` or `fatal` to logfile, because
we have no other way to communicate with pseudo if it failed or not.
Only lines from last pseudo server execution will be printed.

(Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Adrian Walag
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com>
Signed-off-by: Wiktor Baura <wbaura@gmail.com>
Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00