Commit Graph

5201 Commits

Author SHA1 Message Date
Charlie Davies
70f90dcfd7 bitbake: bitbake: providers: consistent single line formatting between functions
(Bitbake rev: e8c74ac8e589ee44d862b71fd63dd75ab303d3b5)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
04bb096170 bitbake: bitbake: providers: remove unneeded logging call
The logging line for when no eligible providers are found only adds
unnecessary noise and is not required. As the stack unwinds better
logging occurs which satisfactorily describes the no eligible providers
error condition.

(Bitbake rev: 933c9df919f7947f515d843199cce2621cf8195f)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Richard Purdie
cca5433baf bitbake: tests/color: Fix event register to pass the datastore
Adding the "if d is None" to the event handling code meant some of
these tests stopped working. The reason is that len(d) was zero but
not equal to None.

Passing the data object to the register() function in the test
correctly registers the event handler and avoids the problem, it
just happened to work previously, incorrectly.

(Bitbake rev: 426eb83c6668d82a2ebaca6c672db131e37c11da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12 15:28:57 +00:00
Richard Purdie
22d7989e8f bitbake: event: Fix another 'if d' test to test for None explictly
This should have been fixed in the previous patch, well spotted Chris!

(Bitbake rev: c3f6fee42bfa23f23f167cb29f0cfa05ac2fa197)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 21:38:32 +00:00
Richard Purdie
b1a5b43735 bitbake: event: Fix multiconfig event handler change performance regressions
There were two issues in this code, firstly the code could stack duplicates
in the variable, secondly, calling "if data" caused the datastore to compute
len(data) which is comparitively expensive. Checking "if data is not None"
is much much faster/cheaper.

The issue was clear from "bitbake -p -P" output where the time in register()
showed large amounts of time in the __len__ function of the datastore.

(Bitbake rev: 9298d08b7dcf9d79f54536f87686d65c4ad7deb9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Manuel Leonhardt
e6ec2b4dcc bitbake: tinfoil: Honor quiet when parsing recipes
When using parse_recipes, honor quiet so that scripts and custom plugins
for recipetool are able to mute progress bars from bitbake that would
otherwise print to STDOUT.

(Bitbake rev: ab9d26c4847a062cadaae5fb8caac0ead5f958db)

Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Tomasz Dziendzielski
5386b3db50 bitbake: runqueue: Print pseudo.log if fakeroot task failed
Currently if pseudo fails we can only see the path to pseudo.log. If we
have no access to server and can only rely on bitbake log then debugging
becomes impossible. This printing needs to be added in runqueue level,
not inside task execution, because in some cases task fails with pseudo
abort really early and we don't even see any log.

In this change I'm adding pseudo log printing in every fakeroot task
failure that logged `mismatch`, `error` or `fatal` to logfile, because
we have no other way to communicate with pseudo if it failed or not.
Only lines from last pseudo server execution will be printed.

(Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Adrian Walag
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com>
Signed-off-by: Wiktor Baura <wbaura@gmail.com>
Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Chen Qi
d3a0f074bc bitbake: event.py: fix regression about INVALIDCONF
A previous commit uses __BBHANDLERS_MC to record all handlers, but
it does not take into consideration of INVALIDCONF, thus causing regression.

We need to record the name before returned AlreadyRegistered, otherwise, when
reparsing due to INVALIDCONF, bascially all handlers are not called.

(Bitbake rev: f6fb6f22fa6ba1ffe121948d7f53ac724358491e)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Alejandro Hernandez Samaniego
0f84d24df8 bitbake: bitbake: Add Azure Storage fetcher implementation
Allows bitbake to fetch from an Azure Storage account.

        The fetcher submodule is compatible with the az:// URI protocol, its
        functionality is based on bitbakes wget fetcher, superior in performance
        to using a propietary tool like azcopy which can handle cloud storage
        account operations with more functionality (that we dont need in a fetcher)
	but less compatibility.

        A sample URI uses can be defined in the following way:
	SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz"

        This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.:

        SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n"

        PREMIRRORS_prepend = "\
            git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
            https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \
        "

        Can also be used with non-public access Azure Storage accounts/containers via a
        Shared Access Signature by declaring the AZ_SAS variable which will be
        automatically used by the fetcher:

        AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature"

(Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Richard Purdie
281eba70c2 bitbake: __init__.py: Fix bitbake debug log handling
For a while I've been puzzled as to why debug logging from runqueue wouldn't
appear on the console with -DD. The logic in the bbdebug handling is inverted
so fix it and now we see the expected messages from runqueue with -D and -DD.

This should then let us debug other issues using those log messages.

(Bitbake rev: 34145b950be03aff8f9b88207cf843abf002ab13)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09 00:02:41 +00:00
Richard Purdie
31c4eec40a bitbake: runqueue: Add setscene task overlap sanity check
We've seen hard to debug issues where a task ends up in both the
covered and notcovered list. Add a sanity check to ensure if this
happens in future, we see it in the logs.

(Bitbake rev: 6e001410854792f9bb66a0409a2ac176171b0507)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09 00:02:05 +00:00
Richard Purdie
341208aa87 bitbake: runqueue: Fix task execution corruption issue
We've seen occasional issues where linux-yocto:do_compile_kernelmodules would
run without do_shared_workdir running before it. do_shared_workdir is an
setscene task but never has an sstate object generated so it will always
rerun. This should not happen since compile_kernemodules should only
execute if a setscene that depends on it didn't run and that should trigger
do_shared_workdir not to be marked as covered.

The issue is that build-appliance-image:do_package is one of the tasks which
covers linux-yocto:do_compile_kernelmodules but it is also a noexec task
and has a dependecy on pseudo-native:do_populate_sysroot.

In the problem case, pseudo-native:do_populate_sysroot is unavailable but
marked as covered since it is noexec. The "harddeps" code then also marks it
as notcovered. No task should ever be both covered and notcovered and this
is where the problems come from.

The solution is for the harddeps code only to to fail tasks if they've not
already been handled in some way. The code is assuming code couldn't have
handled revdeps at this point but we now have clear evidence they can.

(Bitbake rev: f66556bbb38449789ceea2fd105e9f68df7fb660)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09 00:02:05 +00:00
Alejandro Hernandez Samaniego
255fdb9f74 bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatus
If an exception is raised when running host python code, the fetcher
    immediately crashes, this might be temporary depending on the servers
    reliability.

    Catch the exception when the connection was reset and try once again
    to fetch the data.

    File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto
         0665:        if self._timeout_occurred:
         0666:            raise OSError("cannot read from timed out object")
         0667:        while True:
         0668:            try:
     *** 0669:                return self._sock.recv_into(b)
         0670:            except timeout:
         0671:                self._timeout_occurred = True
         0672:                raise
         0673:            except error as e:
    Exception: ConnectionResetError: [Errno 104] Connection reset by peer

(Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 17:47:56 +00:00
Joshua Watt
40f5a289d5 bitbake: event: Fix broken builds when multiconfig has a hyphen in the name
5f7fdf7b2d ("bitbake: event: Prevent bitbake from executing event
handler for wrong multiconfig target") broke multiconfig builds contain
a hyphen, since it's attempt to use the multiconfig as part of a
function name and python functions are not allowed to contain a hyphen.

Rework the bitbake multiconfig test to test a multiconfig with a hyphen
and one with an underscore to validate this doesn't break in the future.

(Bitbake rev: c3168df330a4563cbd03ba74de55a22217d823ed)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21 22:02:01 +00:00
Jan Brzezanski
90e5cce0e3 bitbake: Force parser shutdown after catching an exception
Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing
parser shutdown as default in case of build abort.

In this case bitbake sometimes hangs after facing error during parsing,
waiting for child processes to finish. Killing it then will spawn zombie
processes.

Thus we force the shutdown after catching an exception.

(Bitbake rev: 915330e1dbae1ee8fd9a0358decf2c294f771961)

Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21 22:01:56 +00:00
Charlie Davies
4952dcbd51 bitbake: bitbake: providers: use pythonic empty list check
(Bitbake rev: 591542734988856afa53b99e5eef6e5ca15630b8)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21 22:01:56 +00:00
Charlie Davies
d47f455389 bitbake: bitbake: providers: fix incorrect return type bug
If no eligible providers are found then an integer zero is returned.
This causes the following error, in two possible places in taskdata.py,
when the return value is used in a list comprehension:

[snip]
    eligible = [p for p in eligible if not p in self.failed_fns]
TypeError: 'int' object is not iterable
[\snip]

Fix by returning the variable eligible itself, of type list.

(Bitbake rev: 217c4b436b588a6a47aeaddf61531711ad3fca67)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21 22:01:56 +00:00
Richard Purdie
c8d4f2e401 bitbake: bitbake: Bump version to 1.49.2
This allows metadata to rely upon BB_DEFAULT_UMASK.

(Bitbake rev: 969ac64adab236ce2d5196bcc294005a497913ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-16 11:26:11 +00:00
Richard Purdie
f4fb744657 bitbake: bitbake-worker/runqueue: Add support for BB_DEFAULT_UMASK
Currently each task has to have a umask specified individually. This
is leading to determinism issues since it is easy to miss specifying
this for an extra task.

Add support for specifing the default task umask globally which
simplifies the problem.

(Bitbake rev: 3e664599fd54a8a37ce587022fcbce5ca26f2ed3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-16 11:26:11 +00:00
Tomasz Dziendzielski
8b792d4f75 bitbake: event: Prevent bitbake from executing event handler for wrong multiconfig target
When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.

This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.

See [YOCTO #13071] for detailed bug information.

Add mc target name as a prefix to event handler name so there won't be
two different handlers with the same name. Add internal __BBHANDLERS_MC
variable to have the handlers lists per machine.

(Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-16 11:26:11 +00:00
Richard Purdie
e1691ae855 bitbake: bitbake: Bump release to 1.49.1
(Bitbake rev: 9f23fa605c542a705d00c6c263491899d55bb0d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Joshua Watt
75f87db413 bitbake: logging: Make bitbake logger compatible with python logger
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.

Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.

[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Richard Purdie
df90345d33 bitbake: cooker: Ensure reparsing is handled correctly
From tinfoil, if you edit bblayers.conf and break it, then call
parseConfiguration (e.g. by adding a bad layer with bitbake-layers),
the system doens't show any parse error yet it should.

Add in a call to the updateCache function so that things really
are reparsed when requested.

Partially fixes [YOCTO #14054]

(Bitbake rev: e655f9361b9c3b77906b8e06b5cc76bc5180640e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Tomasz Dziendzielski
a30a06d9c1 bitbake: BBHandler: Don't classify shell functions that names start with "python*" as python function
If shell function name starts with 'python' or 'fakeroot' parser wrongly
assumes it's python/fakeroot function.

[YOCTO #14204]

Use regex lookahead assertions to check if 'python' expression is
followed by whitespace or '(' and if 'fakeroot' is followed by
whitespace.

(Bitbake rev: b07b226d5d1b3acd3f76d8365bc8002293365999)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Tomasz Dziendzielski
1d2fe91db5 bitbake: lib/bb: Don't treat mc recipe (Midnight Commander) as a multiconfig target
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.

| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot

(Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-06 09:12:00 +00:00
Matt Hoosier
45c02843d1 bitbake: fetch/git: download LFS content too during do_fetch
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.

[YOCTO #14191]

(Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c)

Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:10:46 +00:00
Ross Burton
8e92ec7832 bitbake: tests/fetch: add test for empty query parameters
To exercise the previous patch, add a test case.

(Bitbake rev: efdf6d5c4cd8155bc54dc667346f58dccb138e07)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Ross Burton
5d81a91861 bitbake: fetch2: handle empty elements in _param_str_split
_param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws
an exception if a lone semicolon is used as the value doesn't split into two
items.

Fix by checking that the result of the first split has content.

(Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Richard Purdie
5920b3f129 bitbake: fetch2/perforce: Fix localfile to include ud.module
As reported by Katu Txakur, the output depends on the module path
set so this needs to be accounted for in localfile.

(Bitbake rev: 9861ed37bb1c5d09c3b4852d2a252e3f3c86ab14)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Christophe Priouzeau
9f12323826 bitbake: fetch2/wget: Update user-agent
With the usage of enterprise proxy, the user-agent defined are
too old and refused by proxy configuration. Updating to something
more modern is desirable.

(Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400)

Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com>
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:09:26 +00:00
Joey Degges
10c69538c0 bitbake: fetch/git: Fix usehead for non-default names
The usehead url parameter for git repositories causes bitbake to use
whatever commit the repository HEAD is pointing to if the repository
happens to have the name 'default'. This is the default name so in many
cases it works just fine, but if a different name is specified with the
url parameter 'name=newName' then it will fail to parse the recipe with
an error along the lines of:

ERROR: ExpansionError during parsing /path/to/my/recipe.bb
Traceback (most recent call last):
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init:
    >        ud.setup_revisions(d)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions:
             for name in self.names:
    >            self.revisions[name] = srcrev_internal_helper(self, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'):
         if srcrev == "AUTOINC":
    >        srcrev = ud.method.latest_revision(ud, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d, name)
                 return rev
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'):
             raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
    >            (ud.unresolvedrev[name], ud.host+ud.path))
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo

Let's fix this by setting the unresolved rev of _all_ repository names
to 'HEAD' when the usehead url parameter is specified. Update the
currently failing test, test_local_gitfetch_usehead_withname, to now
expect success.

This change preserves existing behavior that allows usehead to be
overridden by a valid looking revision if one happens to be specified
instead of AUTOREV.

(Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c)

Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Joey Degges
41ed881fea bitbake: tests/fetch: Test usehead with a non-default name
Add tests for fetching a URL with the usehead parameter set and a
non-default name set. We currently expect the local version of this test
to fail since there is a bug in the usehead implementation that breaks
for non-default names.

(Bitbake rev: a2345110f217fac429f6ec15f699c87c39531e7c)

Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Joey Degges
d5c4cd6a0b bitbake: tests/fetch: Document behavior of test_gitfetch_usehead
The test `test_gitfetch_usehead' exercises a way to override the usehead
feature by setting SRCREV. It may not be obvious that this is what is
being exercised here so let's add some comments to document the expected
behavior.

(Bitbake rev: 1cd998c19101e3b093e81c126b3048c5d56058b0)

Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Joey Degges
54f4968f36 bitbake: tests/fetch: Organize usehead tests by net requirements
Move the local only usehead test to the FetcherLocalTest class so it
will be run when BB_SKIP_NETTESTS=yes since it does not require
network access.

Rename the usehead tests to better match the new organization.

(Bitbake rev: 137cfa13d5319bc91c3e5fe6c7062cb8c8484d64)

Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Milan Shah
0fa7258d27 bitbake: utils: add docstrings to functions
A list of functions that now has a docstring.
* vercmp_string
* explode_dep_versions
* prunedir
* prune_suffix
* to_boolean
* contains_any
* export_proxies

See [YOCTO #9725] for details.

(Bitbake rev: b61ba4a18693a9e553d2a93161feb0bcc1c82384)

Signed-off-by: Milan Shah <mshah@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Peter Kjellerstedt
b66ff69294 bitbake: cooker: Include all packages a recipe provides in SkippedPackage.rprovides
The provided packages by a skipped recipe are supposed to be listed in
SkippedPackage.rprovides, which is used when generating a meaningful
error message when a build fails because of a skipped package.
Previously this variable only contained the contents of ${RPROVIDES}.
However, most recipes don't define RPROVIDES, they define
RPROVIDES_<pkg> for each package they provide. Additionally, the recipe
provides the packages in PACKAGES without them being included in
${RPROVIDES}.

Before this change, having a runtime dependency on a skipped non-recipe
package would result in a build error stating that the build failed
because the package was skipped, but without providing any reason for
why it was skipped.

(Bitbake rev: efd026c26a377b826a49b945a8212bf7de8a480a)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-21 22:29:57 +00:00
Peter Kjellerstedt
8939086648 bitbake: cache: Make CoreRecipeInfo include rprovides_pkg for skipped recipes
This will be needed by SkippedPackage in the cooker.

(Bitbake rev: 93d01614565bd540d05fbc1791dd66e46723d683)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-21 22:29:57 +00:00
Richard Purdie
65e67f4028 bitbake: data_smart: Ensure hash reflects vardepvalue flags correctly
The get_hash() function is used to decide if the base configuration has changed
and hence whether a reparse is required. The vardepvalue flag's value was not
expanded but it is often used in contexts like:

METADATA_REVISION = "${@base_detect_revision(d)}"
METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"

which in it's unexpanded form means reparsing doesn't happen when it should
as the data appears unchanged. Update get_hash to expand the values of
vardepvalue so reparsing works as expected. This avoids basehash mismatch
errors such as the one recently caused by using METADATA_REVISION in poky.conf's
DISTRO_VERSION variable. The issue there could be exposed by a recipe using
DISTRO_VERSION with the sequence:

bitbake os-release
<change the revision of the metadata with a dummy commit>
bitbake os-release -C install

which was caused because METADATA_REVISION changed but the metadata didn't reparse.

(Bitbake rev: 26ccf1575aef2d6e2d7717d3bd10b1ed0d5a777d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-19 17:18:07 +00:00
Ross Burton
796be0593a bitbake: msg: use safe YAML loader
If using a YAML file in BB_LOGCONFIG, the unsafe loader is used and this
causes a runtime warning:

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated,
as the default Loader is unsafe. Please read https://msg.pyyaml.org/load
for full details.

As log configuration YAML is relatively simple we can just use
safe_load().

(Bitbake rev: 71aaac9efa69abbf6c27d174e0862644cbf674ef)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-10 13:39:15 +00:00
Richard Purdie
4db8652250 bitbake: Revert "fetch2: use relative symlinks for anything pulled from PREMIRRORS"
This reverts commit 481e66ea8fc2fc91903127d66b0f1b0fe86baedb.

Unfortunately this caused problems where incorrect paths were found in paths
and files were not being found. This was reported on the mailing list but no
invetigation or resolution was found. Revert until we can figure out
what was wrong.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-09 12:36:22 +00:00
Hongxu Jia
37b51ed076 bitbake: Revert "bb.ui: delete __init__.py to make bb.ui a namespace package"
The commit [991f92b4d bb.ui: delete __init__.py to make bb.ui a
namespace package] caused `bitbake -h' failed
...
$ bitbake -h
Traceback (most recent call last):
  File "/buildarea/raid5/hjia/community/poky/bitbake/bin/bitbake", line 35, in <module>
    sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
  File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/cookerdata.py", line 27, in __init__
    self.options, targets = self.parseCommandLine(argv or sys.argv)
  File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/main.py", line 297, in parseCommandLine
    options, targets = parser.parse_args(argv)
  File "/usr/lib64/python3.6/optparse.py", line 1387, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib64/python3.6/optparse.py", line 1431, in _process_args
    self._process_short_opts(rargs, values)
  File "/usr/lib64/python3.6/optparse.py", line 1536, in _process_short_opts
    option.process(opt, value, values, self)
  File "/usr/lib64/python3.6/optparse.py", line 785, in process
    self.action, self.dest, opt, value, values, parser)
  File "/usr/lib64/python3.6/optparse.py", line 807, in take_action
    parser.print_help()
  File "/usr/lib64/python3.6/optparse.py", line 1647, in print_help
    file.write(self.format_help())
  File "/usr/lib64/python3.6/optparse.py", line 1635, in format_help
    result.append(self.format_option_help(formatter))
  File "/usr/lib64/python3.6/optparse.py", line 1615, in format_option_help
    result.append(OptionContainer.format_option_help(self, formatter))
  File "/usr/lib64/python3.6/optparse.py", line 1061, in format_option_help
    result.append(formatter.format_option(option))
  File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/main.py", line 54, in format_option
    valid_uis = list_extension_modules(bb.ui, 'main')
  File "/buildarea/raid5/hjia/community/poky/bitbake/lib/bb/main.py", line 72, in list_extension_modules
    pkgdir = os.path.dirname(pkg.__file__)
AttributeError: module 'bb.ui' has no attribute '__file__'
...

This reverts commit 991f92b4d15b0571b6a540964e5216d1b9728539

(Bitbake rev: 22b8c53205f8915b33d1e0ad6a666dcacc01491d)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:26:12 +00:00
Ross Burton
101eec1978 bitbake: providers: selected version not available should be a warning
If the selected version if not available, bitbake will happily attempt
to build something else. This should be a loud warning not a small note.

(Bitbake rev: 78cd63285713fde59506eb2e71a7b7ee59a594ff)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:26:12 +00:00
Hombourger, Cedric
53217b1121 bitbake: fetch2: use relative symlinks for anything pulled from PREMIRRORS
try_mirror_url() creates a symlink named as the original file to make
everything look like files specified in SRC_URI were downloaded from
their original location. The link is however created as an absolute
reference, this makes DL_DIR non-relocatable. This also causes issues
with the Isar project since it bind mounts DL_DIR to /downloads to
perform some of its build tasks in a chrooted environment (rendering
all symbolic links from DL_DIR invalid). Modify ensure_symlink() to
take an optional "relative" argument and have that function use
os.path.relpath() to produce a relative symlink.

(Bitbake rev: 481e66ea8fc2fc91903127d66b0f1b0fe86baedb)

Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
Cc: Chris Larson <Chris_Larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:26:12 +00:00
Joshua Watt
7ae3a24079 bitbake: bitbake: cache: Remove bad keys() function
Removes the keys() function from the MulticonfigCache. This appears to
be a leftover from before the class inherited from collections.Mapping,
is now unnecessary, and was outright incorrect.

(Bitbake rev: 5f37b6d2829fcac1f16602d9697f8bfbcb65ff62)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:26:12 +00:00
Alexander Kanavin
0a2b76064c bitbake: lib/bb/fetch2/__init__.py: drop _PYTHON_SYSCONFIGDATA_NAME unsetting
With introduction of python3targetconfig class in core this is no longer
needed.

(Bitbake rev: 47b64cfacd7c498ef9ed5486d117f2d69a39f225)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 10:50:18 +00:00
Chris Laplante
6898b8e383 bitbake: cookerdata: tweak to avoid mutable default argument
(Bitbake rev: 36ee858415866a122784b05f06e2af0c92dfcafd)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-11 10:09:18 +00:00
Chris Laplante
d8babbe08a bitbake: bb.ui: delete __init__.py to make bb.ui a namespace package
This would permit creation of third-party UI modules without having to
incorporate them directly into BitBake, with BitBake able to
automatically discover them in the same way it finds its built-in UIs.

See https://packaging.python.org/guides/packaging-namespace-packages/

(Bitbake rev: 991f92b4d15b0571b6a540964e5216d1b9728539)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-11 10:09:18 +00:00
Chris Laplante
1eb5b30329 bitbake: main: extract creation of argument parser into function so it can be utilized externally, e.g. by unit tests
(Bitbake rev: 9599059672cc4ae88ba212e34336e0325d37bd75)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-11 10:09:18 +00:00
Lee Chee Yang
03f8c9d9f9 bitbake: BBHandler: prompt error when task name contain expression
Task name contain expresion (eg, do_foo_remove_bar) can cause fatal error.

Check for naming before addtask. Prompt with understandable error message
when expression found in task name.

[YOCTO #8805]

(Bitbake rev: 5b4a5bb0960386f9c524c220e43a16e60e38964d)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-08 14:04:42 +00:00
Richard Purdie
bd98ef572d bitbake: fetch2/git: Document that we won't support passwords in git urls
This keeps coming up, document why we don't do this and the alternatives.

(Bitbake rev: aded964eed4ce5a725ed1ab477efabc86b1aa481)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-03 08:21:10 +00:00