Commit Graph

106 Commits

Author SHA1 Message Date
Ross Burton
96f258ba12 bitbake: tests/fetch: handle network failures gracefully
If there is a network failure the return value from latest_versionstring() is
('','') which later causes an exception when comparing versions.

Improve this by checking the return value and failing the test early.

[ YOCTO #12053 ]

(Bitbake rev: 3f034d2172bf64ecc43577b43e0cf032a54b1358)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-11 17:19:37 +01:00
Christopher Larson
e75639f85a bitbake: tests/fetch: fix GitShallowTest.test_bitbake
`git fetch --tags` seems to interact badly with `mirror=fetch`, resulting in
the regular branches not being fetched, so drop the unnecessary `--tags`. This
fixes this unit test failure: `bb.fetch2.FetchError: Fetcher failure: Unable
to resolve 'master' in upstream git repository in git ls-remote output for
/tmp/tmp4ag_mgmn/gitsource`

[YOCTO #11698]

(Bitbake rev: 2d0203fae08c5ff8dc3e9afaa9a819abc4a1af6f)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Jair Gonzalez
df051ba9a2 bitbake: tests: create unit tests for event module
This change adds a new unit test module (bb.tests.event)
for bitbake event.
It includes the following items:

- Client and server stubs setup
- Testing the module's main functions including:
	- get_class_handlers
	- set_class_handlers
	- clean_class_handlers
	- enable_threadlock
	- disable_threadlock
	- get_handlers
	- set_handlers
	- execute_handler
	- fire_class_handlers
	- print_ui_queue
	- fire_ui_handlers
	- fire
	- fire_from_worker
	- register
	- remove
	- register_UIHhandler
	- unregister_UIHhandler
- Testing event handling using:
	- class Event(object)
	- class OperationStarted(Event)
	- class OperationCompleted(Event)
	- class OperationProgress(Event)
	- class ConfigParsed(Event)

[YOCTO #10368]

(Bitbake rev: 0be3ad391adc73cc0dff81bd0ed7874f2c6a00be)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 14:14:17 +01:00
Ola x Nilsson
ca68d71ddc bitbake: ConfHandler: Require whitespace between export and variable name
(Bitbake rev: 22bb7c9270f02ddae72e13d849375feee5f4a98b)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Christopher Larson
35ecff3cf0 bitbake: fetch/git: add support for removing arbitrary revs for shallow
In certain cases, it's valuable to be able to exert more control over what
history is removed, beyond srcrev+depth. As one example, you can remove most
of the upstream kernel history from a kernel repository, keeping predominently
the non-publically-accessible content. If the repository is private, the
history in that repo couldn't be restored via `git fetch --unshallow`, but
upstream history could be.

Example usage:

    # Remove only these revs, not at a particular depth
    BB_GIT_SHALLOW_DEPTH_pn-linux-foo = "0"
    BB_GIT_SHALLOW_REVS_pn-linux-foo = "v4.1"

(Bitbake rev: 97f856f0455d014ea34c28b1c25f09e13cdc851b)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
30485b2b1a bitbake: fetch/gitannex: add support for shallow mirror tarballs
When we're building from a shallow mirror tarball, we don't want to do
anything with ud.clonedir, as it's not being used when we unpack. As such,
disable updating annex in that case. Also include annex files in the shallow
tarball.

(Bitbake rev: ca0dd3c95502b22c369fbf37f915f45e02c06887)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
f5308b8cc1 bitbake: fetch/gitsm: add support for shallow mirror tarballs
When we're building from a shallow mirror tarball, we don't want to do
anything with ud.clonedir, as it's not being used when we unpack. As such,
disable updating the submodules in that case. Also include the repositories in
.git/modules in the shallow tarball. It does not actually make the submodule
repositories shallow at this time.

(Bitbake rev: 6c0613f1f2f9d4f009545f82a9173e80396f9d34)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
8144f6f408 bitbake: fetch/git: add support for keeping extra refs for shallow
By default, all unused refs (branches & tags) are removed from the repository,
as shallow processing scales with the number of refs it has to process. Add
the ability to explicitly specify additional refs to keep. This is
particularly useful for recipes with custom checkout processes, or whose
git-based versioning requires a tag be available (i.e. for `git describe
--tags`). The new `BB_GIT_SHALLOW_EXTRA_REFS` variable is a space-separated
list of refs, fully specified, and support wildcards.

Example usages:

    BB_GIT_SHALLOW_EXTRA_REFS = "refs/tags/v1.0"
    BB_GIT_SHALLOW_EXTRA_REFS += "refs/heads/*"

(Bitbake rev: 1771934cd9f8b5847c6fcae0a906fb99d6b0db16)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
bf87c5cd19 bitbake: fetch/git: support per-branch/per-url depths for shallow
Allow the user to explicitly adjust the depth for named urls/branches. The
un-suffixed BB_GIT_SHALLOW_DEPTH is used as the default.

Example usage:

    BB_GIT_SHALLOW_DEPTH = "1"
    BB_GIT_SHALLOW_DEPTH_doc = "0"
    BB_GIT_SHALLOW_DEPTH_meta = "0"

(Bitbake rev: 9dfc517e5bcc6dd203a0ad685cc884676d2984c4)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
27d56982c7 bitbake: fetch/git: add support for shallow mirror tarballs
This adds support to the git fetcher for fetching, using, and generating
mirror tarballs of shallow git repositories. The external git-make-shallow
script is used for shallow mirror tarball creation.

This implements support for shallow mirror tarballs, not shallow clones.
Supporting shallow clones directly is not really doable for us, as we'd need
to hardcode the depth between branch HEAD and the SRCREV, and that depth would
change as the branch is updated.

When BB_GIT_SHALLOW is enabled, we will always attempt to fetch a shallow
mirror tarball. If the shallow mirror tarball cannot be fetched, it will try
to fetch the full mirror tarball and use that. If a shallow tarball is to be
used, it will be unpacked directly at `do_unpack` time, rather than extracting
it to DL_DIR at `do_fetch` time and cloning from there, to keep things simple.
There's no value in keeping a shallow repository in DL_DIR, and dealing with
the state for when to convert the clonedir to/from shallow is not worthwhile.

To clarify when shallow is used vs a real repository, a current clone is
preferred to either tarball, a shallow tarball is preferred to an out of date
clone, and a missing clone will use either tarball (attempting the shallow one
first).

All referenced branches are truncated to SRCREV (that is, commits *after*
SRCREV but before HEAD are removed) to further shrink the repository. By
default, the shallow construction process removes all unused refs
(branches/tags) from the repository, other than those referenced by the URL.

Example usage:

    BB_GIT_SHALLOW ?= "1"

    # Keep only the top commit
    BB_GIT_SHALLOW_DEPTH ?= "1"

    # This defaults to enabled if both BB_GIT_SHALLOW and
    # BB_GENERATE_MIRROR_TARBALLS are enabled
    BB_GENERATE_SHALLOW_TARBALLS ?= "1"

(Bitbake rev: 5ed7d85fda7c671be10ec24d7981b87a7d0d3366)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
2a60c40637 bitbake: git-make-shallow: add script to make a git repo shallow
This script will be used by the git fetcher to create shallow mirror tarballs.

    usage: git-make-shallow [-h] [--ref REF] [--shrink] REVISION [REVISION ...]

    Remove the history of the specified revisions, then optionally filter the
    available refs to those specified.

    positional arguments:
      REVISION           a git revision/commit

    optional arguments:
      -h, --help         show this help message and exit
      --ref REF, -r REF  remove all but the specified refs (cumulative)
      --shrink, -s       shrink the git repository by repacking and pruning

While git does provide the ability to clone at a specific depth, and fetch all
remote refs at a particular depth, the depth is across all branches/tags, and
doesn't provide the flexibility we need, hence this script.

Refs (branches+tags) can be filtered, as the process of history removal scales
up rapidly with the number of refs. Even the existing `git fetch --depth=` is
extremely slow on an upstream kernel repository with all the branches and tags
kept.

This uses the same underlying mechanism to implement the history removal which
git itself uses (.git/shallow), and the results, when configured similarly, are
in line with the results git itself produces with `fetch --depth`.

(Bitbake rev: 0254020f0e1911c0eaf99111b91828d2a74a4ee1)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Richard Purdie
000b7d3d1c bitbake: data_smart: Ensure _remove operations on newly set variables are cleared
We clear append/prepend on newly set variables, we should also clear
remove operations. If we don't do this, there is no way we can actually
delete a remove operation. Bitbake internally uses parsing=True to avoid
these side effects when making its own internal calls.

Also add a testcase to bitbake-selftest to ensure we remain consistent going
forward from here.

(Bitbake rev: 3a319f079d699c870d8531e051ab65e6278d1fa5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:58 +01:00
Paul Eggleton
1fe95cfc12 bitbake: bitbake-selftest: add contains tests
Add some tests to verify that we are extracting "contains" information
from python expressions in the code in the bb.data and bb.codeparser
modules.

(Bitbake rev: 88fda492df875dd79b7aecf1f34b38517fc1eb33)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:19:17 +01:00
Joshua Lock
ad1d65ca32 bitbake: lib/bb/tests/codeparser: update expand syntax in comment
bb.data.expand(x, d) is deprecated API

(Bitbake rev: 71185c19205a77d0511fc00baf95a5433e7106d6)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:23 +00:00
Paul Eggleton
5b1807021e bitbake: data_smart: implement missing remote datastore operations
Enable the following operations from a remote datastore to affect the
other end:

* setVarFlag()
* delVar()
* delVarFlag()
* renameVar()

In practice I don't expect these to be used much, but they should be
present so that the implementation is at least reasonably filled out
and that the tests pass.

Also add tests for the interface, mostly by subclassing the existing
local test classes so that they are using a remote datastore. (These
don't actually test remote usage via tinfoil, just that the
datastore's interface can be used.)

(Bitbake rev: 282dc0719d22a39df746eea762ebe05c66aa8f8a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:22 +00:00
Paul Eggleton
99414bdb1c bitbake: tinfoil: fix override handling in remote datastores
There was a huge gap in the remote datastore code introduced in the
tinfoil2 rework - we weren't handling overrides at all, since these are
stored separately from the actual data in the DataSmart object. Thus,
when a datastore actually represents a remote datastore we need to go
back to that remote datastore to get the override data as well, so
introduce code to do that.

To avoid a second round-trip I had to modify the _findVar() function to
return the override data as well. This will increase the overhead a
little when that data is superfluous, but without making the function
even uglier I don't think there's a way to avoid that.

(Bitbake rev: 4f9d6f060ed247fb6fa2f45668a892a1788d3f91)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:22 +00:00
Richard Purdie
da577c7d37 bitbake: tests/data: Add inactive remove override test
In 2bc4d35fb32defc59cd6ed1fc87e35924c201a5c we fixed cases where an inactive
remove override was influecing the value of a variable. This adds a test
case for this issue.

(Bitbake rev: 1ff9b3c669fa187f152de7b8b57d14c2468d926c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-16 22:12:07 +00:00
Richard Purdie
f0d5eb39c3 bitbake: lib: Drop now unneeded update_data calls
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.

(Bitbake rev: 2300beb50333bb620013b058a7309e7f2042101d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
Joshua Lock
97f83e911e bitbake: bitbake: remove True option to getVar calls (take 2)
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(a follow on patch to fix up a few recent introductions)

(Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:24 +00:00
Paul Eggleton
7229250411 bitbake: data_smart: support serialisation
The COW object used within VariableHistory can't be serialised itself,
but we can convert it to a dict when serializing and then back when
deserialising. This finally allows DataSmart objects to be serialized.
NOTE: "serialisation" here means pickling, not over XMLRPC or any other
transport.

(Bitbake rev: bbbb2a53d5decf3b613a92c4ff77c84bfc5d4903)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
8f8a9ef669 bitbake: tinfoil: pass datastore to server when expanding python references
If you're expanding a value that refers to the value of a variable in
python code, we need to ensure that the datastore that gets used to get
the value of that variable is the client-side datastore and not just the
part of it that's on the server side. For example, suppose you are in
client code doing the following:

d.setVar('HELLO', 'there')
result = d.expand('${@d.getVar("HELLO", True)}')

result should be "there" but if the client part wasn't taken into
account, it would be whatever value HELLO had in the server portion of
the datastore (if any).

(Bitbake rev: cbc22a0a9aadc8606b927dbac0f1407ec2736b35)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
d325d94f3f bitbake: data_smart: implement remote datastore functionality
This allows you to maintain a local reference to a remote datastore. The
actual implementation of the remote connection is delegated to a
connector object that the caller must define and supply. There is
support for getting variable values and expanding python references
(i.e. ${@...} remotely, however setting variables remotely is not
supported - any variable setting is done locally as if the datastore
were a copy (which it kind of is).

Loosely based on an earlier prototype implementation by Qing He.

(Bitbake rev: a3edc3eefa2d03c4ad5d12187b32fa4dc495082a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Joshua Lock
ddaac5e4e3 bitbake: bitbake: remove True option to getVarFlag calls
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.

Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)

(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Joshua Lock
1fce7ecbbb bitbake: bitbake: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Richard Purdie
8de811ae76 bitbake: lib/bb: Don't use deprecated bb.data.getVar/setVar API
The old style bb.data.getVar/setVar API is obsolete. Most of bitbake
doesn't use it but there were some pieces that escaped conversion. This
patch fixes the remaining users mostly in the fetchers.

(Bitbake rev: ff7892fa808116acc1ac50effa023a4cb031a5fc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Ross Burton
37d5307db0 bitbake: fetch2: handle absolute paths in subdir
Currently if you use the subdir parameter in a SRC_URI and pass an absolute path
then it gets appended to the unpack directory instead of being used directly.
This is inconvenient as it may be useful to use ${S} when you want to unpack a
file into the source tree.

Change this behaviour so that absolute paths are used directly instead of being
appended to the root directory.  To ensure that recipes cannot write files to an
arbitrary location enforce that the subdir starts with the unpack root.

(Bitbake rev: c3873346c6fa1021a1d63bddd9b898a77c618432)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-22 11:18:11 +01:00
Markus Lehtonen
0b409117c9 bitbake: tests: add unit tests for the usehead url parameter
[YOCTO #9351]

(Bitbake rev: 63031c0236ace10a9d52b9db9bbb892c1b4bf7db)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-20 16:08:59 +01:00
Markus Lehtonen
9cce855f47 bitbake: bitbake-selftest: introduce BB_TMPDIR_NOCLEAN
Set this env variable to 'yes' to preserve temporary directories used by
the fetcher tests. Useful for debugging tests.

(Bitbake rev: 04132b261df9def3a0cff14c93c29b26ff906e8b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-20 16:08:59 +01:00
Jérémy Rosen
0eb6d709b6 bitbake: ast/ConfHandler: Add a syntax to clear variable
unset VAR
will clear variable VAR
unset VAR[flag]
will clear flag "flag" from var VAR

(Bitbake rev: bedbd46ece8d1285b5cd2ea07dc64b4875b479aa)

Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Ross Burton
407ba77fe2 bitbake: lib/bb/tests/fetch: remove URL that doesn't exist anymore
The CUPS ipptool URL we were checking now redirects to github where the tarball
isn't present, so remove it from the test suite.

(Bitbake rev: 4b50895fb3462b21e3874a2e99c363c8d05e89e6)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Richard Purdie
0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie
ca824a976a bitbake: fetch: Use OrderedDict for url parameters
Without this, the dict can reorder causing sanity test failures.

(Bitbake rev: ca8c91acc9396385834b266d4e8b84d917e5e298)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Richard Purdie
fce8da957f bitbake: daemonize/prserv/tests/fetch: Convert file() -> open()
Use python3 compatible functions.

(Bitbake rev: e6a0296ba29c3fbc8417d1df7a01d50562668a41)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Richard Purdie
a9d90f7405 bitbake: tests: assertEquals -> assertEqual
The preferred form is assertEqual, assertEquals is deprecated and
not present in python v3.

This is v2.7 safe.

(Bitbake rev: b60261bf8ade14aca31238b50c243c01adcabc59)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:34:30 +01:00
Randy Witt
7e739acfa7 bitbake: tests/utils.py: test origvalue in a callback matches what is expected
There were no tests that verified the value of origvalue in the callback
routines used by edit_metadata(). This patch adds one for a simple
multiline variable.

(Bitbake rev: ece3a4d02d8162dee78c2062c10291b5fd625c36)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-09 23:00:46 +01:00
Randy Witt
e1e459e480 bitbake: lib/bb/utils.py: Fix a bug in edit_metadata() that could corrupt vars
edit_metadata() would corrupt a variable that was multiline, but
had the ending quotes on the same line as the last value. For example:

    TEST_VAR = " foo \
    bar"

would become " foo ba" because the code would always delete the last
character on the line and then do it again if the line ended in the
quote. This however doesn't show up if you have:

    TEST_VAR = " foo \
    bar \
    "

which is how all the test cases were written.

This patch fixes that bug and adds and fixes a test that matched the bugs
behavior rather than the expected behavior.

(Bitbake rev: 14f05cbdc2ad8d59a94af1c8816567d93c39c88c)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-09 23:00:46 +01:00
Olof Johansson
2c722e227f bitbake: tests/fetch.py: Improve unit tests for trusted network check
The tests were skipped when running without network even though they
didn't require network. This commit also adds a test case for URLs with
ports in them (the ports should not be considered when doing trusted
network checks).

(Bitbake rev: 77747de6b20538063eef3b188489a35bef225359)

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 15:53:44 +01:00
Richard Purdie
8037ba4d86 bitbake: bb/tests/fetch: Update cups url
Update the upstream url used for testing cups versions after upstream website
changes.

(Bitbake rev: 5f06041d4936fc22297945bbbad7020bfa9083c6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-21 12:43:12 +00:00
Alexander Shashkevich
865d2feff6 bitbake: fetch2: fixes copying of file://dir; subdir=foo, bug 6128 and bug 6129
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.

These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files

(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)

Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Richard Purdie
ab7b7bf94d bitbake: fetch2/__init__: Fix decodeurl to better handle urls without paths
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.

This tweaks the code so that unless its a file url, the host is returned
correctly.

This patch also adds test cases for these urls to the exist set of test
urls.

We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.

(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie
4628fe12e7 bitbake: lib/bb: Add expansion parameter to getVarFlag
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.

On the most part this is an automatic translation with:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g'  -i `grep -ril getVar *`

There should be no functional change from this patch.

(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Ross Burton
9f22898e0f bitbake: fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives).  This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.

Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.

(Bitbake rev: 6ec70d5d2e330b41b932b0a655b838a5f37df01e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-26 22:34:45 +00:00
Richard Purdie
cf9cb65eec bitbake: data_smart: Don't show exceptions for EOL literals
If variables are unset, the code simply doesn't expand them, there
aren't errors. If the code is a python expression, this can get a bit
messy, see the attached test case. The python expansion code sees the }
of the unexpanded value rather than the close of the python expression
and then raises a SyntaxError exception.

Ideally, we'd update the code to match pairs of brackets. I don't know
how to do that with the current regex and this is unfortunately a
performance sensitive piece of code. We also run the risk of breaking
existing code in OE-Core where there are "{" characters but not "}"
to close them (PKGE and PE).

Rather than raising the exception, matching the existing "just return
the expression" behaviour seems more consistent with the standard
variable behaviour.

This addresses an issue found in the recent image.bbclass code where
there are some variables we choose not to expand (TMPDIR/DATETIME).

This patch also adds a test case for this behaviour. It wouldn't preclude
improved bracket matching code in the future either.

(Bitbake rev: d80d39e73223a50fda0090784303d2c57167bb4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:01:24 +00:00
Olof Johansson
c886a786cf bitbake: tests/codeparser.py: Add filename/lineno flags to test variable
A recent change in bitbake added filename/lineno information to the
parameters of bb.data.build_dependencies(). The codeparser tests
required a little adaption to the changes, adding the flags to the FOO
variable used in the tests.

The error seen when running the tests is a TypeError exception raised
in bb.codeparser:

  TypeError: int() argument must be a string or a number, not 'NoneType'

(Bitbake rev: f1fe674397ac5cd355696d5b4cc90b7cfa6c867f)

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:38:51 +00:00
Alexander Kanavin
3745479596 bitbake: bitbake: rename REGEX, REGEX_URI, and GITTAGREGEX.
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.

(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-08 10:20:31 +00:00
Paul Eggleton
4de214f752 bitbake: lib/bb/utils: improve edit_bblayers_conf() handling of bblayers.conf formatting
Make the following improvements to edit_bblayers_conf():

* Support ~ in BBLAYERS entries
* Handle where BBLAYERS items are added over multiple lines with +=
  instead of one single long item

Also add some comments documenting the function arguments and return
values as well as a set of bitbake-selftest tests.

(This function is used by the bitbake-layers add, remove and
layerindex-fetch subcommands, as well as devtool when adding the
workspace layer).

(Bitbake rev: e9a0858023c7671e30cc8ebb08496304b7f26b31)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:21 +00:00
Mark Hatle
2bb600a579 bitbake: tests/fetch.py: Fix recursion failure in url mapping
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list.  This permits us to re-evaluate the list
while excluding all matches that have previousily occured.

Without this fix, adding this test results in a failure:

RuntimeError: maximum recursion depth exceeded in cmp

(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:34 +01:00
Mark Hatle
4c3d4eceb1 bitbake: fetch2/__init__.py: uri_replace regex handling
We should only substitute one time.  If we do it without a max count, we can
end up matching over and over.

Before this change:

https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz

with a mirror of

https://.*/[^/]*    http://AAAA/A/A/A/

would end up either recursing indefinitely or result in:

http://AAAA/A/A/A/A/A/A/A/A/A/bitbake-1.0.tar.gz

(Bitbake rev: 4d254e02e2867dd9a6663508c8ca9f2733af71a8)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:34 +01:00
Richard Purdie
aaea533e70 bitbake: tests/parse: Add BBCLASSEXTEND multiple data store corruption reproducer
One data store changing a variable poked through into a different data
store. This test case replicates that issue where the value 'B' would
become unset/disappear.

We also enhance parsehelper to generate files with an optional suffix
such as bbclass.

(Bitbake rev: 5c4179f58a4e04f1c354df5f17d1860eb403f0ac)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-23 08:48:41 +01:00
Aníbal Limón
205c39bf78 bitbake: tests/fetch.py: Updated test name FetchMethodTest -> FetchLatestVersionTest
Change the test name to be more specific on what is tested.

(Bitbake rev: 3e39156bc330c4c726058a5b9c13d33e2daad89f)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-22 08:23:50 +01:00