Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.
Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.
Also add some unittests for this.
(Bitbake rev: e82a4ab48991035866da9914c8b75a9bfbc9a7fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently INVALID and None are checked as incorrect values under different
circumstances. This code standardises those checks to be consistent. We
should phase out the use of "INVALID".
(Bitbake rev: 86ef4e65ce18b71dc69643586bd2aa8f48703171)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher will try:
1) PREMIRROR
2) Upstream
3) MIRROR
If it fails to download from the Upstream, but succeeds from the MIRROR,
and ud.localpath != origud.localpath (for example, the git tarball),
then we will get the error (e.g.: xf86-video-omapfb):
ERROR: Function failed: Fetcher failure for URL: 'xxx'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /path/to/log.do_fetch.28024
It should not show the error and let the build go on since it succeeds.
(e.g.: xf86-video-omapfb)
[YOCTO #5686]
(Bitbake rev: c08ca1e4eeb04f78e1354780cf5a4c3855e49572)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake -S throws exception when 'noop' signature handler is used
[YOCTO #5738]
(Bitbake rev: 53352e8d388b7fc4da73f95b93dcc087e76d0426)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows FETCHCMD_git to override the fetcher command as the git fetcher does.
[YOCTO #5717]
(Bitbake rev: 23ab943be3a33077d6ad8be68bba53cd1e2270b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* currently decode_url regexp parses branch=@foo as username so it ends like this:
- ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
+ ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
that there is at least one '/' as separator between netloc and path,
params, so it looks reasonable to prevent including '/' in username
(Bitbake rev: 2c82742114091cb55055328b54223686816582f2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fsync was added for belt and braces protection for things like sstate
on NFS. To be honest, it probably doesn't buy much, if the rename isn't
atomic, all bets are off anyway and there are bigger issues to worry about.
The issue is that at the end of every task, the dump_sig() code is triggered
to save out information about the task and this was triggering an fsync(fd).
Whilst it may select the file descriptor, on file systems like ext4, it will
require large parts of the journal to be written out so it can have
significant impact. latencytop showed an average fsync() call overhead of about
2s and if that happens for 5000 tasks, the time mounts up. This blocks the next
task execution by that time.
We therefore drop the fsync since in reality its causing problems and is unlikely
to buy much.
(Bitbake rev: 46fd841319479f6079d850b3813e64bd8c2680a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code clearly uses an earlier fetcher API. Update it to match master.
(Bitbake rev: e13acb4113ce75226664c3006a9776cc885e860d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This add a Git Annex backend which reuses the Git fetcher code; it
allows managing files with git, without checking the file contents
into git, being useful when dealing with files larger than git can
currently easily handle, whether due to limitations in memory, time,
or disk space.
(Bitbake rev: a61fc4db598e9d13c966712a6a0e4783e19448be)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For rebased git tree, some commits doesn't exist in any branch, and such commits are
valid in tag, the change is useful for such case.
(Bitbake rev: f594cb9f5a18dd0ab2342f96ffc6dba697b35f65)
Signed-off-by: Zhenhua Luo <zhenhua.luo@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user manual said:
'cleandirs' - directories which should created before the task runs but should be empty
But it only removes the dir, doesn't create it
[YOCTO #5703]
(Bitbake rev: 0636797d75874ce4577f29011d69c56a4c6b9e89)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
Everything should be removed, but the
0006-omapfb-port-to-new-xserver-video-API.patch.done still exists in the
DL_DIR, this is because the clean() in the fetch2/__init__.py skips
removing the local file, so that it will skip removing the .done.
The local file (file://) isn't needed to be removed since it is not
downloaded into DL_DIR, but the .done should be removed, this patch will
remove the .done, and it doesn't remove anything else since the clean()
in local.py does nothing.
[YOCTO #5687]
(Bitbake rev: 2bc99b9dfa532430a13c39fca4e5ef3a2206b3b8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
The git2_git.pingu.fi.xf86-video-omapfb.tar.gz has been removed from the
DL_DIR, but the git2_git.pingu.fi.xf86-video-omapfb.tar.gz.done still exists,
this is because the "open(ud.donestamp, 'w').close()" in try_mirror_url() will
create the git2_git.xxx.tar.gz.done, but no one removes it (the clean() in
fetch2/__init__.py removes the DL_DIR/git2/pkg.done)
This only happens on the git fetcher AFAIK.
[YOCTO #5688]
(Bitbake rev: fb2dc84875eb477661f421b21bc404d4805ce379)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A part of fetch2/__init__.py uses 3 spaces as the indent, I
think that they should be typos.
(Bitbake rev: abafd85e2fcf23cee872e0e9e468898101430f1f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Because of the way we were handling this error, it was printed twice -
once via logger.error() (to avoid the log being printed) and a second
time when the exception gets wrapped in a FuncFailed at a higher level.
Call logger.error() earlier and change the text we send in the
exception to be more brief, so it more closely resembles the behaviour
when there is an invalid checksum.
(Bitbake rev: 46765369d7f76ec7f67b90430131a79eb6a66235)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were checking SRC_URI md5sum/sha256sum values against None here, so
if they were set to "" then no error was produced. Since the value is
still effectively unset in this case, this is not the right behaviour;
just check if the value doesn't evaluate to False instead.
(Bitbake rev: 040943a718795c64dc4e604abfcf08b26b7d00e6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output when comparing siginfo files for dict_diff is reversed and shows
additions when things were removed and vice versa. This patch reverses the operation
so the changes are shown correctly and makes the output less confusing.
(Bitbake rev: 9b4142df36619099670740a5d3bc94e404ab2b56)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Based upon the list of difference starting points, we can use the siggen.find_siginfo()
function call and the difference printing code to provide a list of differences
between the current build target and whatever can be obtained from the sstate cache.
(Bitbake rev: 7a77861feb62750ef166d2d1e89ed1f444ca8dc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code prints all differences. If the task dependencies have changed hash,
we recurse into those and print those differences as well. This leads to a lot
of output. The reality is if the parents changed signature, we might as well just
say that and recurse with no other output since we're much more interested in how
the parents changed in nearly all cases. The changes in the parent are probably
the same ones we'd have printed at each level anyway.
By doing this we focus the output more carefully on the thing the user wants/needs
to see.
(Bitbake rev: 7a17fd6e51a76d3582c357b79f5ef86e1969650c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way hash_deps was being generated was different to the way siggen generated
the data internally which lead to seemingly different sigdata/siginfo files
for the same checksum. The -S output correct but the files written during
builds contained superflous data which would look like a difference.
This patch removes the badly duplicated data and uses it from the source
which ensures its consistent.
(Bitbake rev: e6d5e925c402cd2cc7ee034e9de4cc6df8944a34)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to understand where the delta starts against an existing sstate cache
for a given target. Adding this to the output of the -S option seems like a
natural fit.
We use the hashvalidate function to figure this out and assume it can find siginfo
files for more than just the setscene tasks.
(Bitbake rev: c18b8450640ebfd55a2b35b112959f9ea3e0a700)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The basecmd is initialized in urldata_init; there's no need redoing that
work.
(Bitbake rev: f8df6f746fb2e27f029a5449cee6c891b1f36f4f)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the fetcher doesn't distinguish between names that the fetcher
needs to resolve verses branch names that the user specified.
This meant that if you specify a tag and a branch, the fetcher broke. This
separates the two so that the branch name is preserved and can be used in
appropriate places.
(Bitbake rev: e85f39fe9d1b224414b5da0780da514f75c5df92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its near impossible to control task addition/deletion from
metadata context. This adds stong add/deltask API to bb.build
which is traditionally where it resided. The rather broken
remove_tasks function was removed, it didn't appear to do anything
useful or have any users.
This allows us to clean up hacks currently in use in metadata and use
standard API for it instead.
(Bitbake rev: bf7138dd38fc1f8efca80891198e3422fef64093)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Back in the depths of time we did support task removal. In the pre
AST days it was nearly impossible to continue supporting it, it wasn't
used so it was dropped. With the modern codebase we can easily now support
deltask and it would be very useful within the metadata since it can
massively simplify dependency trees.
As an example, a core-image-sato had 47703 inter task dependencies before
this patch and a patch to native.bbclass, afterwards with the noexec tasks
deleted, we had 29883. Such a significant simplification is worthwhile
and justifies adding a deltask operation to the system.
(Bitbake rev: acecbde6fb70ff3c96deab3cdf819d8442e87ed4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were accidentally using references to sets in the contains functionality
instead of creating a copy. This could cause data corruption and corruption
of the resulting sstate checksums.
This patch fixes this to make a copy of the set and resolved the corruption
issue.
(Bitbake rev: 8f4733257ad665aa7c7e7061c543379d5e4e3af2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On some machines, a hang has been noticed where the system sits in
the select call despite the task having completed.
The exact reasons for this as unknown but adding a timeout unblocked
the builds and resolved the hangs in question.
(Bitbake rev: 5223ffb5b6a46d8b3f6ac3362bd2672e2edf2691)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved the parseConfiguration method before obtaining the machines and
other configurations from bitbake. If not, Hob doesn't see the new machines
added by the new layer.
[YOCTO #5632]
(Bitbake rev: 8de14b2a481d61424eb32fd0234f7a38a961a75b)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PE is an optional field in recipes specifiying the epoch
for the recipe. The canonical form for the
full recipe version string is: PE:PV-PR
If no PE is specified, we shouldn't store the initial ":"
character, as it leads to inconsistency with how the
version string is used elsewhere. This patch drops the leading ":"
[YOCTO #5459]
(Bitbake rev: f6031bd753917c459ab232d88d7dcfc3f10e8184)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Up until this patch, package information lived in two
places - one table for build packages and one table for
target installed packaged. This situation leads to
two problems: there is no direct link between a build
package and a installed package, and a lot of data is duplicated.
This change unifies all package types in a single table.
The SimpleUI remains the same for continuity sake,
but the REST API will be changed in a future patch.
The package dependencies and package files are now
kept in a single table.
Since we collect target installed package information at all times,
we need to expand it to supplement missing information if a
package is not actually built in the current build.
Small changes to the Simple UI reflect the updated database schema.
[YOCTO #5565]
[YOCTO #5269]
(Bitbake rev: f5d655bfaeb349c8680d74530617e34aa389d1f0)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the processes of removing local system accesses
from toaster UI, we remove the build stats
code that was moved to toaster.bbclass, and
adapt the database writing code to read the data
from BuildStatsList event sent by the toaster.bbclass
[YOCTO #5604]
(Bitbake rev: 4930ff5b471761c2a8d16c1935cdab9cf141d2d8)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the processes of removing local system accesses
from toaster UI, we remove the layer data reading
code that was moved to toaster.bbclass, and
adapt the database writing code to read the data
from event sent by the toaster.bbclass
[YOCTO #5604]
(Bitbake rev: 33b60a940f58e8374a8c7baa9bf28a98f54cdf13)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we don't sort the data, the values can reorder changing the
signatures meaning we get confused builds and significant cache
misses.
(Bitbake rev: 8f453bb11d72afc90a986ac604b3477d97eaf9a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
That has no impact on the builds themselves, so we should just
remove that line from the variable history.
[YOCTO #5561]
(Bitbake rev: ae0ed55e80b7bd30c775b128b4114b306a50ff69)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting INHERIT var means to remove other operations made on INHERIT.
This is too intrusive, so we decided to use append for this case.
[YOCTO #5448]
(Bitbake rev: a2d0122c198ee50325e6f0e2f5d1c2284475fc7b)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher made the rather bold assumption that if it fetched from the upstream,
the revisions were present and correct. These checks are fast and ensure that
really is the case. The avoids accidental network accessed and missing
branch configuration problems.
(Bitbake rev: a9112a102a89049cda597dad449e922c9e957a5d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If warnings come from recipes parsing
and not from package build, 'parent' object
will be None; so don't update the color for it.
[YOCTO #5621]
(Bitbake rev: f9d24f55a5ffa9257e7ba3257be1210687513733)
Signed-off-by: Valentin Popa <valentin.popa@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The FILES_INFO structure is now much simpler, so remove all of the
horrible mangling we had to do here in order to read it.
(Bitbake rev: 11a664292064dbf76850bf21ba386f78a43a56b6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This treeview code was obviously copy-pasted from one of the config
dialogs and the variables were never renamed. Rename them now to improve
readability.
(Bitbake rev: 93b3ddad4396d757d1d0c199ac168f8b1a205fe3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PKGSIZE is now in bytes in pkgdata, so we need to treat it as such in
the UI code for Hob / Toaster.
(Bitbake rev: 3b5ff814cd4a3efa4b17c6b343ec39c9acca5c9e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
initConfigurationData and loadConfigurationData are similar functions, the only
reason for them appears to be to be able to reset the pre/post configuration
files. The current code is confusing and unmaintainable.
Instead this patch creates a new Sync command which allows these to be explicitly
set. The init and load functions can then be merged into one. There is then no
need for a parseConfiguration command, we can simply reset the server to have the
settings take effect.
The reset fuction is not an instant value return and triggers an event so it should
be an Async command, not a sync one.
The number of calls for the set pre/post command is probably higher than it
need be but someone with more familiarity with the hob code base can probably
figure out the right places its needed (maybe just init_cooker?).
(Bitbake rev: bae5210d7e048022f083361964ebec7daf1608f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Might as well use this functionality now we have it available.
(Bitbake rev: cd7f4d85e3f187140d1bb0aecf82f657a8f8701a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous "contains" changes caused a ~3% parsing speed impact.
Looking at the cause of those changes was interesting:
* Use of defaultdict was slower than just checking for missing entries
and setting them when needed.
* Even the "import collections" adversely affects parsing speed
* There was a missing intern function for the contains cache data
* Setting up a log object for each variable has noticeable overhead
due to the changes in the code paths uses, we can avoid this.
* We can call getVarFlag on "_content" directly within VariableParse
for a noticeable speed gain since its a seriously hot code path.
This patch therefore tweaks the code based on the above observations to
get some of the speed back.
(Bitbake rev: fca802187a2a30686a8a07d2b6b16a3e5716e293)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've always found it somewhat annoying that this icon's rotation
animation wasn't quite correct; this was because it was looping around
to the second position instead of the first, which made it appear to
stutter.
(Bitbake rev: 78ad15b669b9c7cde41f7bd1ab884c1d2e0db91b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you had more than 15 layers the system would crash since one more
value is added to one array than the other. This fixes the code
so equal numbers of values are added to the arrays and hence
doesn't crash when many layers are enabled.
(Bitbake rev: 4e65463886a2ef245b2f8974e82e9cb942af224b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
