Commit Graph

7699 Commits

Author SHA1 Message Date
Nicolas Dechesne
b3426e89f5 bitbake: doc: bitbake-user-manual: fix typo left over from Sphinx migration
Fixes d99760cc687c (sphinx: last manual round of fixes/improvements)

Reported-by: Michal Piechowski <m.z.piechowski@gmail.com>
(Bitbake rev: 00ce48919de720639eda2b6f7065a82b641e5167)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-26 15:26:14 +01:00
Michael Opdenacker
7e9305613d bitbake: doc: bitbake-user-manual: code insertion simplification over two lines
This simplifies paragraphs ending with a colon and followed
by code insertion.

Automatically substituted through the command:
sed -i -z "s/:\n\s*::/::/g" file.rst

This generates identical HTML output.

(Bitbake rev: 51c80fc3497eecc8e50194fe1ff8069b59f03eda)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-26 15:26:14 +01:00
Michael Opdenacker
91aacf4ed3 bitbake: doc: bitbake-user-manual: simplify colon usage
- This replaces instances of ": ::" by "::", which
  generates identical HTML output

(Bitbake rev: fd8ce4dcaff3aae395f9945fb0a3be54905e1727)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-26 15:26:14 +01:00
Diego Sueiro
cda69e0166 bitbake: layerindex: Add --fetchdir parameter to layerindex-fetch
Introduce --fetchdir parameter to layerindex-fetch enabling users to choose
the directory to fetch the layers different from BBLAYERS_FETCH_DIR.

[YOCTO #14347]

(Bitbake rev: 784a904faffac723ddf58ba765b9dd11ac068de5)

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-23 10:13:13 +01:00
Diego Sueiro
9fae310dd2 bitbake: layerindex: Fix bitbake-layers layerindex-show-depends command
Running 'bitbake-layers layerindex-show-depends meta-filesystems' fails with:
```
Traceback (most recent call last):
  File "<...>/poky/bitbake/bin/bitbake-layers", line 93, in <module>
    ret = main()
  File "<...>/poky/bitbake/bin/bitbake-layers", line 86, in main
    return args.func(args)
  File "<...>/poky/bitbake/lib/bblayers/layerindex.py", line 209, in do_layerindex_show_depends
    self.do_layerindex_fetch(args)
  File "<...>/poky/bitbake/lib/bblayers/layerindex.py", line 182, in do_layerindex_fetch
    args.shallow)
AttributeError: 'Namespace' object has no attribute 'shallow'
```

Initialize the shallow attribute to fix it.

(Bitbake rev: 71f095c147fe6aa7b5e6272002e0498cf9494256)

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-23 10:13:13 +01:00
Richard Purdie
8c5b67bec2 bitbake: doc/user-manual-fetching: Remove basepath unpack parameter docs
The code for this was removed in 2016 in commit
e659a3b0c2771679057ee3e13cd42e6c62383ff2. Nobody seems to have missed it
so remove the documentation so we match the code.

[YOCTO #13449]

(Bitbake rev: 76bf42ea41a28b19d0377c2e548b0a59119fdf67)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:51 +01:00
Niels Avonds
d12a06cd49 bitbake: fetch/gitsm: Fix crash when using git LFS and submodules
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.

This issue was introduced in this
commit: 977b7268bf

[YOCTO #14283]

(Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)

Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Richard Purdie
6584e59ff0 bitbake: bitbake-server: Remove now unneeded code
With the previous patch this code is now pointless as we'd have hit a TypeError
before now.

(Bitbake rev: 6301a99055c79d89b715f72182cd0ef1b781b89a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Ross Burton
01066a584a bitbake: bitbake-server: ensure server timeout is a float
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer.  There's some type confusion here:

bitbake-server is called with a string representation of the timeout,
which may be None.  If the timeout is not set, pass 0 instead of None.

Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.

[ YOCTO #14350 ]

(Bitbake rev: c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Richard Purdie
d08d362cf9 bitbake: tinfoil/data_smart: Allow variable history emit() to function remotely
We can't access the emit() function of varhistory currently as the datastore parameter
isn't handled correctly, nor is the output stream. Add a custom wrapper for this
function which handles the two details correctly.

(Bitbake rev: ba0fa084ccd2b1ade96425d158fd31e49e42f286)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
46654e14a5 bitbake: tests/fetch: add tests for local and remote "noshared" git fetching
(Bitbake rev: e0267fe43bda208856af939b17e39beb9e5586c3)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
d274cf0cf8 bitbake: tests/fetch: deduplicate local git testing code
Purely cosmetic change that probably improves the code.

(Bitbake rev: 9c0733f0062f3cf19514c891cc06c9a8e0db429b)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
2cb6ce7788 bitbake: fetch/git: add support for disabling shared clones on unpack
By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour.

Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.

With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.

The switch is a global bitbake switch and will affect all git urls.
Build systems that need "noshared" most likely need it for everything
they do with git.

(Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Mikko Rapeli
33650ffdc7 bitbake: bitbake: tests/fetch: remove write protected files too
For some reason several git-annex files in Debian 10 buster
are read-only and removing them with "rm -rf" fails.

Fixes test failures like:

$ bitbake-selftest
...
rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied
rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied
EE..................................ssss.sssssssssssssss.sssss.......................................................................................................
======================================================================
ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex
    fetcher, ud = self.fetch_shallow(uri)
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow
    bb.utils.remove(ud.clonedir, recurse=True)
  File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove
    subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1.

Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod
the whole temporary directory which should cover any directory name differences between
different git-annex versions. Fixes tests failing after chmod call:

Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/

(Bitbake rev: 7729ef2983c72867e99fad82d671069ba5cb32b2)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Mikko Rapeli
490fb73e34 bitbake: bitbake: tests/fetch: fix test execution without .gitconfig
A CI user validating changes does not have any git push rights or
even a .gitconfig file so fix tests so that they run
by setting the user.name and user.email for the repo before
committing changes.

Fixes errors like:

ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist
    self.add_empty_file('a')
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file
    self.git(['commit', '-m', msg, path], cwd)
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git
    return bb.process.run(cmd, cwd=cwd)[0]
  File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run
    raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128:

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

(Bitbake rev: 57c0811f1ee19b6619f4840a39e01e3cb98c34c4)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Richard Purdie
7ea37b9291 bitbake: runqueue: Fix deferred task issues
In a multiconfig situation there are circumstances where firstly, tasks
are deferred when they shouldn't be, then later, tasks can end up as
both covered and not covered.

This patch fixes two related issues. Firstly, the stamp validity checking
is done up front in the build and not reevaulated. When rebuilding the
deferred task list after scenequeue hash change updates, we need therefore
need to check if a task was in notcovered *or* covered when deciding to
defer it. This avoids strange logs like:

NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene)
NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch

where tasks have run but are then deferred.

Since we're recalculating the whole list, we also need to clear it before
iterating to rebuild it. By ensuring covered tasks aren't added to the
deferred queue, the covered + notcovered issue should also be avoided.
in the task deadlock forcing code.

[YOCTO #14342]

(Bitbake rev: 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Paul Eggleton
f2a20f7bd0 bitbake: bitbake-user-manual: add REQUIRED_VERSION and adjust PREFERRED_VERSION entry
Add REQUIRED_VERSION, add a reference to it in PREFERRED_VERSION and
adjust the opening statement to read slightly better.

(Bitbake rev: b32e6c8d4ea2f83fe77021207e9db883fec82d97)

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-15 20:18:57 +01:00
Paul Eggleton
1892aae539 bitbake: bitbake-user-manual: document no support for using passwords in git URLs
This is based on the comment added in revision
aded964eed4ce5a725ed1ab477efabc86b1aa481.

(Bitbake rev: 082683da089115d8b6f71f221cabb41ac401f733)

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-15 20:18:57 +01:00
Richard Purdie
ca07d22405 bitbake: bitbake: Update version to 1.50.0 stable release series
(Bitbake rev: e70b925ba98fd4fedf3940d141a4210c953087ca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-09 14:20:01 +01:00
Richard Purdie
9982b2c0f2 bitbake: runqueue: Further fixes for confused setscene tasks
There is further evidence of tasks ending up being "covered" and "notcovered"
which shouldn't happen and is bad. The code that caused this problem last
time appears to have issues where stamps for tasks already exist.

Split out the setscene stamp checking code to a separate function and
use this when checking "hard dependencies" (like pseudo-native) so
that if the stamps exist and it will be "covered", it is not put on
the notcovered list.

(Bitbake rev: a1848a481e36b729c8e4130c394b1d462d4b488a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:31:24 +01:00
Michael Opdenacker
62880ffb17 bitbake: doc: Update links to documentation
(Bitbake rev: ed8e1fd4cf9d5ac8a8203638add99d686b4b3521)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Richard Purdie
86ec266d91 bitbake: runqueue/event: Add an event for notifying of stale setscene tasks
Use the new functionality in build.py to identify stale setscene tasks
and send an event to the metadata listing them. The metadata then
has the option of performing cleanup operations if it thinks that
appropriate.

(Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Richard Purdie
ab55f8bfec bitbake: build: Add find_stale_stamps function
Add a new function which compares the stamp filename we want (including
taskhash) with what is in the stamp directory (using the clean mask).
This tells us which stamp files are stale and are due to be rerun.

(Bitbake rev: b126a53882d202e4df0f9661303355c9fe9ec80e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Robert P. J. Day
22931aec26 bitbake: doc: fix syntax error in layer.conf example
While this example really needs to be rewritten to not define multiple
patterns in the same layer.conf, as long as it's there, it might as
well be syntactically correct.

(Bitbake rev: 1c2d2875099a3ff9149710d42c679ab31b00f68b)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-22 18:10:06 +00:00
Robert P. J. Day
d4603d56c7 bitbake: doc: fix glossary link for BB_INVALIDCONF variable
(Bitbake rev: 4a4d55c41806815c32add32863a05351a6df30f7)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-22 18:10:06 +00:00
Robert P. J. Day
a29a37f0b0 bitbake: doc: mention that addtask handles multiple dependencies
Add a note explaining that "addtask" can accept multiple dependencies,
just in case someone runs across such an example and is confused.

(Bitbake rev: 5e4e9acd323b5ebc3a14c07384f6cf8f8a272066)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-22 18:10:06 +00:00
Robert P. J. Day
7a348cdc3a bitbake: doc: move BBFILES_DYNAMIC for alphabetical order
Since BBFILES_DYNAMIC does not have a "BB_" prefix, it belongs further
down in the variable glossary.

(Bitbake rev: 493f291601c7680f8b8146f358c545fcb5f53cc9)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 21:47:42 +00:00
Tomasz Dziendzielski
a9098e6423 bitbake: codeparser: Fix TypeError in bitbake debug mode
Commit 75f87db413 fixed the confusion between bitbake and python logger
but in codeparser still old method of setting debug level was used
causing TypeError, because debug level value was incorrectly returned
and assigned to event.msg.
| File "./bitbake/lib/bb/ui/knotty.py", line 660, in main
|   event.msg = event.fn + ': ' + event.msg
| TypeError: can only concatenate str (not "int") to str

[YOCTO #14298]

(Bitbake rev: bec9ea7ab28a8dfad1a6010ca5c6ec691754748d)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 21:47:42 +00:00
Richard Purdie
ddacd8c66f bitbake: bblayers/query: Update to handle REQUIRED_VERSION
Ii is unclear whether any changes are needed to bblayers are
needed to handle the extra data from REQUIRED_VERSION. Update
to at match the new API, at present it doesn't look necessary
to handle the required version data.

(Bitbake rev: 53c30efec4099035d19e6717059dfceff8ff88fd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
cb3cbd8bb2 bitbake: bitbake: cooker: Add REQUIRED_VERSION checks
Add logic to check that if REQUIRED_VERSION has been set that the cooker
class method findBestProviders properly handles the case where the
REQUIRED_VERSION has not been found.

Fixes [YOCTO #10096]

(Bitbake rev: 5df201d746f26154213e6d15d2721cd35b38b59e)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
889a7ce8de bitbake: bitbake: lib: add Required Version to Bitbake --show-versions command
This commit adds a new column to the Bitbake --show-versions command
called Required Version. This column will display any packages which
have a REQUIRED_VERSION successfully set.

Fixes [YOCTO #10096]

(Bitbake rev: 90c7d1815e41243323d32b9dbb865757a922578a)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
78fcf6831a bitbake: bitbake: providers: check for REQUIRED_VERSION in _filterProviders
Before the REQUIRED_VERSION variable was introduced the
PREFERRED_VERSION variable allowed for a fallback to the next most
suitable version.

Since REQUIRED_VERSION does not allow a fallback to a different version
implement a check in the _filterProviders function to make sure that
if a requested REQUIRED_VERSION is not found then the function returns
no eligible providers have been found.

Fixes [YOCTO #10096]

(Bitbake rev: c41386b78aa53e0bf081cd973c950b88126670a7)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
3752782cb7 bitbake: bitbake: providers: introduce logic for REQUIRED_VERSION variable
This commit adds checks during the findPreferredProvider function for a
new variable REQUIRED_VERSION. This can be set, in exactly the same
manner as PREFERRED_VERSION, on a per package basis to enforce the use
of a particular version of a package.

REQUIRED_VERSION is similar in behaviour to PREFERRED_VERSION except if
the version specified by REQUIRED_VERSION is not found an error occurs
and the execution of Bitbake stops.

Fixes [YOCTO #10096]

(Bitbake rev: 5cbf6d95fc1009e78e7d0745a49e0bf418b37abb)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
70f90dcfd7 bitbake: bitbake: providers: consistent single line formatting between functions
(Bitbake rev: e8c74ac8e589ee44d862b71fd63dd75ab303d3b5)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
04bb096170 bitbake: bitbake: providers: remove unneeded logging call
The logging line for when no eligible providers are found only adds
unnecessary noise and is not required. As the stack unwinds better
logging occurs which satisfactorily describes the no eligible providers
error condition.

(Bitbake rev: 933c9df919f7947f515d843199cce2621cf8195f)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Richard Purdie
cca5433baf bitbake: tests/color: Fix event register to pass the datastore
Adding the "if d is None" to the event handling code meant some of
these tests stopped working. The reason is that len(d) was zero but
not equal to None.

Passing the data object to the register() function in the test
correctly registers the event handler and avoids the problem, it
just happened to work previously, incorrectly.

(Bitbake rev: 426eb83c6668d82a2ebaca6c672db131e37c11da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12 15:28:57 +00:00
Richard Purdie
22d7989e8f bitbake: event: Fix another 'if d' test to test for None explictly
This should have been fixed in the previous patch, well spotted Chris!

(Bitbake rev: c3f6fee42bfa23f23f167cb29f0cfa05ac2fa197)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 21:38:32 +00:00
Richard Purdie
b1a5b43735 bitbake: event: Fix multiconfig event handler change performance regressions
There were two issues in this code, firstly the code could stack duplicates
in the variable, secondly, calling "if data" caused the datastore to compute
len(data) which is comparitively expensive. Checking "if data is not None"
is much much faster/cheaper.

The issue was clear from "bitbake -p -P" output where the time in register()
showed large amounts of time in the __len__ function of the datastore.

(Bitbake rev: 9298d08b7dcf9d79f54536f87686d65c4ad7deb9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Manuel Leonhardt
e6ec2b4dcc bitbake: tinfoil: Honor quiet when parsing recipes
When using parse_recipes, honor quiet so that scripts and custom plugins
for recipetool are able to mute progress bars from bitbake that would
otherwise print to STDOUT.

(Bitbake rev: ab9d26c4847a062cadaae5fb8caac0ead5f958db)

Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Tomasz Dziendzielski
5386b3db50 bitbake: runqueue: Print pseudo.log if fakeroot task failed
Currently if pseudo fails we can only see the path to pseudo.log. If we
have no access to server and can only rely on bitbake log then debugging
becomes impossible. This printing needs to be added in runqueue level,
not inside task execution, because in some cases task fails with pseudo
abort really early and we don't even see any log.

In this change I'm adding pseudo log printing in every fakeroot task
failure that logged `mismatch`, `error` or `fatal` to logfile, because
we have no other way to communicate with pseudo if it failed or not.
Only lines from last pseudo server execution will be printed.

(Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Adrian Walag
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com>
Signed-off-by: Wiktor Baura <wbaura@gmail.com>
Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Chen Qi
d3a0f074bc bitbake: event.py: fix regression about INVALIDCONF
A previous commit uses __BBHANDLERS_MC to record all handlers, but
it does not take into consideration of INVALIDCONF, thus causing regression.

We need to record the name before returned AlreadyRegistered, otherwise, when
reparsing due to INVALIDCONF, bascially all handlers are not called.

(Bitbake rev: f6fb6f22fa6ba1ffe121948d7f53ac724358491e)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Alejandro Hernandez Samaniego
1d2fc389f9 bitbake: docs: Add AZ_SAS definition to glossary
(Bitbake rev: 134df841b273d94e767426876459da348d96dd23)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Alejandro Hernandez Samaniego
3a60575cba bitbake: docs: Add Az fetcher documentation
(Bitbake rev: e3b480636a3c2716effd619b59cf55e11f9a6db0)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Alejandro Hernandez Samaniego
0f84d24df8 bitbake: bitbake: Add Azure Storage fetcher implementation
Allows bitbake to fetch from an Azure Storage account.

        The fetcher submodule is compatible with the az:// URI protocol, its
        functionality is based on bitbakes wget fetcher, superior in performance
        to using a propietary tool like azcopy which can handle cloud storage
        account operations with more functionality (that we dont need in a fetcher)
	but less compatibility.

        A sample URI uses can be defined in the following way:
	SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz"

        This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.:

        SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n"

        PREMIRRORS_prepend = "\
            git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
        "

        Can also be used with non-public access Azure Storage accounts/containers via a
        Shared Access Signature by declaring the AZ_SAS variable which will be
        automatically used by the fetcher:

        AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature"

(Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Richard Purdie
281eba70c2 bitbake: __init__.py: Fix bitbake debug log handling
For a while I've been puzzled as to why debug logging from runqueue wouldn't
appear on the console with -DD. The logic in the bbdebug handling is inverted
so fix it and now we see the expected messages from runqueue with -D and -DD.

This should then let us debug other issues using those log messages.

(Bitbake rev: 34145b950be03aff8f9b88207cf843abf002ab13)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09 00:02:41 +00:00
Richard Purdie
31c4eec40a bitbake: runqueue: Add setscene task overlap sanity check
We've seen hard to debug issues where a task ends up in both the
covered and notcovered list. Add a sanity check to ensure if this
happens in future, we see it in the logs.

(Bitbake rev: 6e001410854792f9bb66a0409a2ac176171b0507)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09 00:02:05 +00:00
Richard Purdie
341208aa87 bitbake: runqueue: Fix task execution corruption issue
We've seen occasional issues where linux-yocto:do_compile_kernelmodules would
run without do_shared_workdir running before it. do_shared_workdir is an
setscene task but never has an sstate object generated so it will always
rerun. This should not happen since compile_kernemodules should only
execute if a setscene that depends on it didn't run and that should trigger
do_shared_workdir not to be marked as covered.

The issue is that build-appliance-image:do_package is one of the tasks which
covers linux-yocto:do_compile_kernelmodules but it is also a noexec task
and has a dependecy on pseudo-native:do_populate_sysroot.

In the problem case, pseudo-native:do_populate_sysroot is unavailable but
marked as covered since it is noexec. The "harddeps" code then also marks it
as notcovered. No task should ever be both covered and notcovered and this
is where the problems come from.

The solution is for the harddeps code only to to fail tasks if they've not
already been handled in some way. The code is assuming code couldn't have
handled revdeps at this point but we now have clear evidence they can.

(Bitbake rev: f66556bbb38449789ceea2fd105e9f68df7fb660)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09 00:02:05 +00:00
Alejandro Hernandez Samaniego
255fdb9f74 bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatus
If an exception is raised when running host python code, the fetcher
    immediately crashes, this might be temporary depending on the servers
    reliability.

    Catch the exception when the connection was reset and try once again
    to fetch the data.

    File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto
         0665:        if self._timeout_occurred:
         0666:            raise OSError("cannot read from timed out object")
         0667:        while True:
         0668:            try:
     *** 0669:                return self._sock.recv_into(b)
         0670:            except timeout:
         0671:                self._timeout_occurred = True
         0672:                raise
         0673:            except error as e:
    Exception: ConnectionResetError: [Errno 104] Connection reset by peer

(Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 17:47:56 +00:00
Joshua Watt
40f5a289d5 bitbake: event: Fix broken builds when multiconfig has a hyphen in the name
5f7fdf7b2d ("bitbake: event: Prevent bitbake from executing event
handler for wrong multiconfig target") broke multiconfig builds contain
a hyphen, since it's attempt to use the multiconfig as part of a
function name and python functions are not allowed to contain a hyphen.

Rework the bitbake multiconfig test to test a multiconfig with a hyphen
and one with an underscore to validate this doesn't break in the future.

(Bitbake rev: c3168df330a4563cbd03ba74de55a22217d823ed)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21 22:02:01 +00:00
Jan Brzezanski
90e5cce0e3 bitbake: Force parser shutdown after catching an exception
Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing
parser shutdown as default in case of build abort.

In this case bitbake sometimes hangs after facing error during parsing,
waiting for child processes to finish. Killing it then will spawn zombie
processes.

Thus we force the shutdown after catching an exception.

(Bitbake rev: 915330e1dbae1ee8fd9a0358decf2c294f771961)

Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21 22:01:56 +00:00