pre/postfuncs were not being added to checksums. This meant that when reconfiguration
occurred, tasks were not always being rerun when they should. This include
sstate functions as well as systemd's do_install function in the OE metadata.
With the addition of postfuncs, its possible a shell task can have a python
pre/postfunc so we have to guard against this when generating shell output
in emit_func.
(Bitbake rev: b84d010144de687667cf855ddcb41c9b863c236e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Shared work directories work by assuming bitbake will not run
more than one task with a specific stamp name. Recent runqueue optimisations
accidentally broke this meaning there could be races. This fixes the code.
(Bitbake rev: b1628b1a260ddf43fc9985535b1ddcfcebbb1e5b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we set:
BB_SIGNATURE_HANDLER = "noop"
Then we would get the following errors:
[snip]
File "runqueue.py", line 876, in RunQueue._start_worker(fakeroot=False, rqexec=None):
"fakerootnoenv" : self.rqdata.dataCache.fakerootnoenv,
> "hashes" : bb.parse.siggen.taskhash,
"hash_deps" : bb.parse.siggen.runtaskdeps,
AttributeError: 'SignatureGenerator' object has no attribute 'taskhash'
[snip]
This patch fixes the problem.
[YOCTO #5741]
(Bitbake rev: 2bfcb751891cf3b4050e996b3c8e28678c3a8bf4)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hob doesn't read these variable from file and bitbake doesn't use them,
so they shouldn't be set in conf file.
(Bitbake rev: a8c9df86b96e27dc49028c2da42034d13988960c)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If enable_proxy is not activated, Hob should not save the proxy values
in the conf file when user hits save button.
[ HOB #5308 ]
(Bitbake rev: fbe0851221ecfcefea5bdd4b629a05ed4f5ac189)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When Hob receives a bb.command.CommandFailed event, it should check if
it's a log kind of information.
"Forced shutdown" and "Stopped build" are messages that show when a build
is not complete, but Hob considered them error.
[HOB #5609]
(Bitbake rev: ea1939f7ec8d8a71ce16a60c251c2413d7d91eb3)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The busy cursor would never change after pressing Stop button.
It should be set after the possible return inside machine_combo_changed_cb()
method.
(Bitbake rev: d440d3ad4b2d99bc20e06d2d5f5e76d07864dff3)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It must be checked first if parser has the attribute 'shutdown' when
user hits Stop button and the forceshutdown state is given.
[HOB #5579]
(Bitbake rev: 46943b442ea4fa778f70590b6dcce483595efaf8)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a bb.command.CommandFailed event is received by Hob, the error
message is stored inside event.error.
This information tells exactly why bitbake failed, so Hob should display
it instead of the current composed message.
(Bitbake rev: 24543ff6b45771712d624541ae35738d7d98f33c)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its near enough impossible to tell bitbake to run target X,
task Y and target A, task B. We could hack various parts of the API
around but it would mean incompatibilities.
An alternative is to accept the syntax "<target>:do_<task>" as a target
name. The default task would be used where the task is unspecified.
This has the advantage that its neat/clean code and works from all
current APIs including the commandline.
(Bitbake rev: 55f6bee3114e582333a1784caeddb197b9163d02)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under OE-Core, the name under which a package would
be installed in a target may have been different than the
name under it has been built or recorded in the dependencies
listings.
This patch addresses the way that Toaster records package
names, and adds the field of "installed_name" to save the
name under which a package have been installed in an image.
(Bitbake rev: 24e0367429b248108b104ab5a2af05efcf7a8c39)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
THis is a large set of fixes for the generic table, Build and
Dashboard pages.
Among the fixes:
* the table remembers which columns to show across refreshes,
based on saving the settings in a cookie
* added column timespent for a build which is a denormalization
of the completed_on - started_on information due to limits in
computing datetime differences in the SQL engine
* fixed formatting of the time differences
* various sorting header links fixed
* correct error and warning CSS classes applied to the
respective rows
* fixes multiple divide-by-zero error in displaying duration
estimations
(Bitbake rev: 61e3dee55ac577fce1c0ae0fe7e0d3cf644e8ae6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch updates the task classification fields (outcome, etc)
as to
* Changes outcome names from SSTATE to CACHED and
from EXISTING to PREBUILT
* NoExec tasks now recorded as Not Executed / script type NA instead
of Executed / script type NOEXEC. Script type NOEXEC is deleted.
* SetScene tasks do not get order numbers
* New task method that returns a QuerySet for setscene tasks related
to this task: Task.get_related_setscene()
* New custom TaskManager that allows searching for setscene tasks
related to a certain task: Task.objects.related_setscene(task)
(Bitbake rev: a4164821a142f8b625a5fdc209adc6dc80874241)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I can't think of a reason we'd download zero sized files however there are
reasons zero length files can accidently make it onto source mirrors.
This check allows us to ignore the broken files and switch to another
mirror rather than fail with odd checksum failures.
(Bitbake rev: 300cba2e1a720dba4b83b0c76208ea93c608c1de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to deference tags when trying to map them to commit IDs with
ls-remote. If we don't do this, a given commit might not show up
later in a specific branch. There appears to be no good reason not
to do this.
(Bitbake rev: 8ef24f4c834298348172b96ec0b855bf09552b09)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When specifying tags, they're searched for unanchored so foo/bar could
match:
refs/heads/abc/foo/bar
refs/heads/xyz/foo/bar
refs/heads/foo/bar
This change anchors the expressions so they are based against heads
or tags (or any other base level tree that has been created).
(Bitbake rev: df2e0972cd1db7abd5ec8b7cb295fb0c42e284a4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current message can be ambiguous, improve it (and also rename a
variable to clean up the rest of the function).
(Bitbake rev: 0c1bb7c0fce7b0f334311a2893ccb00385fa8d55)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.
Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.
Also add some unittests for this.
(Bitbake rev: e82a4ab48991035866da9914c8b75a9bfbc9a7fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently INVALID and None are checked as incorrect values under different
circumstances. This code standardises those checks to be consistent. We
should phase out the use of "INVALID".
(Bitbake rev: 86ef4e65ce18b71dc69643586bd2aa8f48703171)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher will try:
1) PREMIRROR
2) Upstream
3) MIRROR
If it fails to download from the Upstream, but succeeds from the MIRROR,
and ud.localpath != origud.localpath (for example, the git tarball),
then we will get the error (e.g.: xf86-video-omapfb):
ERROR: Function failed: Fetcher failure for URL: 'xxx'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /path/to/log.do_fetch.28024
It should not show the error and let the build go on since it succeeds.
(e.g.: xf86-video-omapfb)
[YOCTO #5686]
(Bitbake rev: c08ca1e4eeb04f78e1354780cf5a4c3855e49572)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake -S throws exception when 'noop' signature handler is used
[YOCTO #5738]
(Bitbake rev: 53352e8d388b7fc4da73f95b93dcc087e76d0426)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows FETCHCMD_git to override the fetcher command as the git fetcher does.
[YOCTO #5717]
(Bitbake rev: 23ab943be3a33077d6ad8be68bba53cd1e2270b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* currently decode_url regexp parses branch=@foo as username so it ends like this:
- ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
+ ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
that there is at least one '/' as separator between netloc and path,
params, so it looks reasonable to prevent including '/' in username
(Bitbake rev: 2c82742114091cb55055328b54223686816582f2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fsync was added for belt and braces protection for things like sstate
on NFS. To be honest, it probably doesn't buy much, if the rename isn't
atomic, all bets are off anyway and there are bigger issues to worry about.
The issue is that at the end of every task, the dump_sig() code is triggered
to save out information about the task and this was triggering an fsync(fd).
Whilst it may select the file descriptor, on file systems like ext4, it will
require large parts of the journal to be written out so it can have
significant impact. latencytop showed an average fsync() call overhead of about
2s and if that happens for 5000 tasks, the time mounts up. This blocks the next
task execution by that time.
We therefore drop the fsync since in reality its causing problems and is unlikely
to buy much.
(Bitbake rev: 46fd841319479f6079d850b3813e64bd8c2680a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code clearly uses an earlier fetcher API. Update it to match master.
(Bitbake rev: e13acb4113ce75226664c3006a9776cc885e860d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This add a Git Annex backend which reuses the Git fetcher code; it
allows managing files with git, without checking the file contents
into git, being useful when dealing with files larger than git can
currently easily handle, whether due to limitations in memory, time,
or disk space.
(Bitbake rev: a61fc4db598e9d13c966712a6a0e4783e19448be)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For rebased git tree, some commits doesn't exist in any branch, and such commits are
valid in tag, the change is useful for such case.
(Bitbake rev: f594cb9f5a18dd0ab2342f96ffc6dba697b35f65)
Signed-off-by: Zhenhua Luo <zhenhua.luo@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user manual said:
'cleandirs' - directories which should created before the task runs but should be empty
But it only removes the dir, doesn't create it
[YOCTO #5703]
(Bitbake rev: 0636797d75874ce4577f29011d69c56a4c6b9e89)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
Everything should be removed, but the
0006-omapfb-port-to-new-xserver-video-API.patch.done still exists in the
DL_DIR, this is because the clean() in the fetch2/__init__.py skips
removing the local file, so that it will skip removing the .done.
The local file (file://) isn't needed to be removed since it is not
downloaded into DL_DIR, but the .done should be removed, this patch will
remove the .done, and it doesn't remove anything else since the clean()
in local.py does nothing.
[YOCTO #5687]
(Bitbake rev: 2bc99b9dfa532430a13c39fca4e5ef3a2206b3b8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
The git2_git.pingu.fi.xf86-video-omapfb.tar.gz has been removed from the
DL_DIR, but the git2_git.pingu.fi.xf86-video-omapfb.tar.gz.done still exists,
this is because the "open(ud.donestamp, 'w').close()" in try_mirror_url() will
create the git2_git.xxx.tar.gz.done, but no one removes it (the clean() in
fetch2/__init__.py removes the DL_DIR/git2/pkg.done)
This only happens on the git fetcher AFAIK.
[YOCTO #5688]
(Bitbake rev: fb2dc84875eb477661f421b21bc404d4805ce379)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A part of fetch2/__init__.py uses 3 spaces as the indent, I
think that they should be typos.
(Bitbake rev: abafd85e2fcf23cee872e0e9e468898101430f1f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Because of the way we were handling this error, it was printed twice -
once via logger.error() (to avoid the log being printed) and a second
time when the exception gets wrapped in a FuncFailed at a higher level.
Call logger.error() earlier and change the text we send in the
exception to be more brief, so it more closely resembles the behaviour
when there is an invalid checksum.
(Bitbake rev: 46765369d7f76ec7f67b90430131a79eb6a66235)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were checking SRC_URI md5sum/sha256sum values against None here, so
if they were set to "" then no error was produced. Since the value is
still effectively unset in this case, this is not the right behaviour;
just check if the value doesn't evaluate to False instead.
(Bitbake rev: 040943a718795c64dc4e604abfcf08b26b7d00e6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output when comparing siginfo files for dict_diff is reversed and shows
additions when things were removed and vice versa. This patch reverses the operation
so the changes are shown correctly and makes the output less confusing.
(Bitbake rev: 9b4142df36619099670740a5d3bc94e404ab2b56)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Based upon the list of difference starting points, we can use the siggen.find_siginfo()
function call and the difference printing code to provide a list of differences
between the current build target and whatever can be obtained from the sstate cache.
(Bitbake rev: 7a77861feb62750ef166d2d1e89ed1f444ca8dc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code prints all differences. If the task dependencies have changed hash,
we recurse into those and print those differences as well. This leads to a lot
of output. The reality is if the parents changed signature, we might as well just
say that and recurse with no other output since we're much more interested in how
the parents changed in nearly all cases. The changes in the parent are probably
the same ones we'd have printed at each level anyway.
By doing this we focus the output more carefully on the thing the user wants/needs
to see.
(Bitbake rev: 7a17fd6e51a76d3582c357b79f5ef86e1969650c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way hash_deps was being generated was different to the way siggen generated
the data internally which lead to seemingly different sigdata/siginfo files
for the same checksum. The -S output correct but the files written during
builds contained superflous data which would look like a difference.
This patch removes the badly duplicated data and uses it from the source
which ensures its consistent.
(Bitbake rev: e6d5e925c402cd2cc7ee034e9de4cc6df8944a34)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to understand where the delta starts against an existing sstate cache
for a given target. Adding this to the output of the -S option seems like a
natural fit.
We use the hashvalidate function to figure this out and assume it can find siginfo
files for more than just the setscene tasks.
(Bitbake rev: c18b8450640ebfd55a2b35b112959f9ea3e0a700)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The basecmd is initialized in urldata_init; there's no need redoing that
work.
(Bitbake rev: f8df6f746fb2e27f029a5449cee6c891b1f36f4f)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the fetcher doesn't distinguish between names that the fetcher
needs to resolve verses branch names that the user specified.
This meant that if you specify a tag and a branch, the fetcher broke. This
separates the two so that the branch name is preserved and can be used in
appropriate places.
(Bitbake rev: e85f39fe9d1b224414b5da0780da514f75c5df92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its near impossible to control task addition/deletion from
metadata context. This adds stong add/deltask API to bb.build
which is traditionally where it resided. The rather broken
remove_tasks function was removed, it didn't appear to do anything
useful or have any users.
This allows us to clean up hacks currently in use in metadata and use
standard API for it instead.
(Bitbake rev: bf7138dd38fc1f8efca80891198e3422fef64093)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Back in the depths of time we did support task removal. In the pre
AST days it was nearly impossible to continue supporting it, it wasn't
used so it was dropped. With the modern codebase we can easily now support
deltask and it would be very useful within the metadata since it can
massively simplify dependency trees.
As an example, a core-image-sato had 47703 inter task dependencies before
this patch and a patch to native.bbclass, afterwards with the noexec tasks
deleted, we had 29883. Such a significant simplification is worthwhile
and justifies adding a deltask operation to the system.
(Bitbake rev: acecbde6fb70ff3c96deab3cdf819d8442e87ed4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were accidentally using references to sets in the contains functionality
instead of creating a copy. This could cause data corruption and corruption
of the resulting sstate checksums.
This patch fixes this to make a copy of the set and resolved the corruption
issue.
(Bitbake rev: 8f4733257ad665aa7c7e7061c543379d5e4e3af2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On some machines, a hang has been noticed where the system sits in
the select call despite the task having completed.
The exact reasons for this as unknown but adding a timeout unblocked
the builds and resolved the hangs in question.
(Bitbake rev: 5223ffb5b6a46d8b3f6ac3362bd2672e2edf2691)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
