If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.
[YOCTO #6934]
(Bitbake rev: 923fc5ee0ace02cc29110bff502a2c65e6bdebf0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bug has a long discussion of this. Basically, in some environments,
the exact details of which aren't understood, a Ctrl+C signal to the
UI is being transmitted to all the process children. Looking at the output
of "ps ax -O tpgid", its clear the main process is still the terminal
owner of these processes.
stty -a on a problematic system shows: "-ignbrk brkint"
and on a working system shows: "-ignbrk -brkint"
The description of brkint would suggest this is the problem, setting up
that terminal environment wasn't able to reproduce the problem though.
It was confirmed that using setsid() caused the problem to be resolved
and is probably the right thing to be doing anyway, so lets do it.
[YOCTO #6949]
(Bitbake rev: 81d90389edd4d4778d3aec86e0775ab98dd1496e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 67ebf368aab8fbe372374190f013bdf2c83c59de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases for cache purpoes we not only need to know which file
is going to be used but also which paths were considered. Add a
localpaths method which includes the history.
The core which() funciton already supports this, this just extends
the function to preserve the extra data we need. localpath becomes
just a special case of the case with history.
(Bitbake rev: d71407dbbf82659f245e002ecaad02b26838f455)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.
We can't change sh, we can ensure the user is warned about the problem
straight away though.
[YOCTO #7006]
(Bitbake rev: 879fe20f47ba75f4afb3484d4398d5fd60431e12)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get/set_taskdata functions are now part of the API of the class,
ensure they exist in the base class definition so the noop handler
works.
[YOCTO #7233]
(Bitbake rev: d571149cd82028c5e05cca33a3007ce1b779a654)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possble to fetch Gerrit review references which are
normally stored under refs/changes.
Please disregard previous patch with the same topic.
(Bitbake rev: ab8cbf2a71750f5ea36e218036b050857301607b)
Signed-off-by: Fredrik Svensson <fredrik.svensson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To support yocto on systems with python3 as default version, scripts
should use /usr/bin/env python in the shebang, as this allows the use of
a fake env to mimic python2 as default version.
This patch simply replaces occurrences of #!/usr/bin/python with
#!/usr/bin/env python and was done with this oneliner:
git grep -lE '^#!/usr/bin/python' | xargs \
sed -i 's|/usr/bin/python|/usr/bin/env python|'
(Bitbake rev: 0f9823adb7832c4ca3b2985391473aa6e8c22148)
Signed-off-by: Martin Hundebøll <martin@hundeboll.net>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.
(Bitbake rev: 72536d4e0cc3379001b730950afa012f7a96a79b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.
(Bitbake rev: f12c738d3dc1f0fd105d457385511440024bffab)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shellshock patches changed the way bash functions are exported.
Unfortunately different distros used slightly different formats,
Fedora went with BASH_FUNC_XXX()=() { echo foo; } and Ubuntu went with
BASH_FUNC_foo%%=() { echo foo; }.
The former causes errors in dealing with out output from emit_env,
the functions are not exported in either case any more.
This patch handles things so the functions work as expected in either
case.
[YOCTO #6880]
(Bitbake rev: 4d4baf20487271aa83bd9f1a778e4ea9af6f6681)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
(Bitbake rev: dca5d82830ef2838439e5272da9dac1f28954cf1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to consistently use LogMessage.INFO/WARNING/ERROR to make sure toaster knows
how to categories these rather than passing in the "raw" loglevel value
which in best case comes from python logging but worst case any value.
[YOCTO 6885]
(Bitbake rev: 926235aad806232bc73e33d6dd8955dd26562e6b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ideally, we want the PR service to have minimal influence from
queued disk IO. sqlite tends to be paranoid about data loss and
locks/fsync calls. There is a "WAL mode" which changes the journalling
mechanism and would appear much better suited to our use case.
This patch therefore switches the database to use WAL mode. With this
change, write overhead appears significantly reduced.
(Bitbake rev: 90b05e79764b684b20ce8454e89f05763b02ac97)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sync/commit calls are happening in the submission thread which can
race against the handler. The handler may start new transactions which
then causes the submission thread to error with "cannot start a
transaction within a transaction".
The fix is to move the calls to the correct thread.
(Bitbake rev: 08cf468ab751f4c6e4ffdab2d8e5d748f7698593)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the commit 'prserv: Ensure data is committed', the PR server moved to
only committing transactions to the database when the PR server is
stopped. This improves performance, but it means that if the machine
running the PR server loses power unexpectedly or if the PR server
process gets SIGKILL, the uncommitted package revision data is lost.
To fix this issue, sync the database periodically, once per 30 seconds
by default, if it has been marked as dirty. To be safe, continue to
sync the database at exit regardless of its status.
(Bitbake rev: 973ac2cc63323ca9c3e916effa4765747db3564c)
Signed-off-by: Ben Shelton <ben.shelton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given that the "do_package_write" task doesn't exist in OE anymore,
steal another, existing example to demonstrate the "rdeptask" flag.
(Bitbake rev: d412d3680f78eebe0517e4f933d853b8973df711)
Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similar to the last shared work task signature bug, we've found another
one. Looking at the improved output of diffsigs in this case:
runtaskdeps changed from [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native',
'libgcc-initial_4.9.bb.do_patch:virtual:nativesdk'
] to [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gcc-crosssdk-initial_4.9.bb.do_patch',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native'
]
so we can get a different task hash since libgcc sorts before gnu-config
and gcc sorts after it. We could do with a way of fixing this, the best
I can come up with is to include a single parent directory. Since
recipes are never at the top of any metadata trees I've seen, this
should suffice for now.
I'm planning to burn the concept of shared work within bitbake
and do something at the metadata level in the 1.8 timeframe as its just
too fragile as things stand and hard to fix well.
(Bitbake rev: fc7ebf3835a206a5daafd4e1b73bac2549714ad3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we split variables only at whitespaces, a slipped in tab will render
a value unremovable.
(Bitbake rev: 0da22ba3e930fbb060b31fc423fd3333ca8843a0)
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We fix the rendering of the size field in dirinfo-related pages
by directly calling filtered_filesizeformat and not rendering
it through the template engine.
Additionally, we enable error dumping into logs if an
Exception happens.
[YOCTO #6669]
(Bitbake rev: afa2431c21b8271b05dc4cca4265f98d9f338007)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If localhost resolves to a remote address (due to a misconfigured network),
starting the pr server will fail without useful information.
To reproduce, add '<bogus ip> localhost' to /etc/hosts and run
'bitbake -p'. The error message will be:
ERROR: Timeout while attempting to communicate with bitbake server
ERROR: Could not connect to server False:
Running 'bitbake-prserv --host=localhost --port=0 --start' will fail with:
error: [Errno 99] Cannot assign requested address
Since these errors does not show the IP address of the attempted socket
binding, this results in a lot of wasted time looking at firewall rules, etc.
This patch results in the following error message if the socket binding fails:
PR Server unable to bind to <bogus ip>:0
(Bitbake rev: fae5914030bcf4c061c22fc61034c40c87b7121a)
Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob was adding the redundant characters "\1" in SSTATE_MIRRORS variable. If
needed it is expected the user will add this instead so remove the code
that was doing this.
[YOCTO #6600]
(Bitbake rev: 73bf120062fc00c7e26dc4e77a7d140658d89daf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check if the 'subdir' parameter exists and assign it to 'destdir' so that
files are copied in ${WORKDIR}/destdir. This fixes urls that are of the form
file://a;subdir=b.
(Bitbake rev: 836a986b365eb9798563ec08d90b346596de7791)
Signed-off-by: Roxana Ciobanu <roxana.ciobanu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Writing a log that the filesystem isn't being monitored for inode usage just
confuses users who are not aware about the nature of inodes in their filesystem,
so don't say anything, just silently disable the monitor. In general this only
happens on filesystems which don't have a limit on inodes.
(Bitbake rev: ca93bc84ee5fb94a50c11c47e4d212d7da649e24)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I could apply the label "release_1.0" to a super project that contains
many sub projects. If my recipes have SRC_URI's that use that label but
grab different sub-folders, than there's a bug where the cached localfile
(tar.gz) will not be unique and reused at the wrong times.
SRC_URI = "p4://perforce::1234@//depot/SuperProject/MiniProjectAAA/...;label=release_1.0 \
p4://perforce::1234@//depot/SuperProject/MiniProjectBBB/...;label=release_1.0"
(Bitbake rev: 3b5b1703b77490116dda895b29737cea51a3d6a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using options like -k, -f, -v and so on with the memory resident
server, they'd currently only be set on the initial values passed to
the original command. This ensures they now match those specified
on the commandline for the options where this makes sense.
To make this work, a command to update the options on the server side
is required so this is added.
[YOCTO #5292]
(Bitbake rev: 1c75cc4d0c8b606c1fe76e6bf60bf6a32298b105)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this you see:
File "bitbake/bin/bitbake-worker", line 201, in fork_off_task
os._exit(child())
TypeError: an integer is required
(Bitbake rev: cd477b5e77ab0373248b8a8fa30e1c7b8ea984fd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have:
FOO = "${bindir}/X Y"
FOO_remove = "${bindir}/X"
the expected result is "Y". Currently this doesn't work since the removed
expressions are not expanded first. This patch adjusts things so the
expressions are expanded before being processed for removal.
Also add a test to ensure this case continues to work.
[YOCTO #6624]
(Bitbake rev: 72a1ca4a104ccab73d6abcbd44db9c2636a58572)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
(Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately it seems like the external use of the cooker
enableDataTracking() function broke at some point since the code that
reads it now runs within BBCooker's constructor. Since this now has to
be done early, add a parameter to Tinfoil's constructor to allow
enabling variable history tracking.
Fixes [YOCTO #6676].
(Bitbake rev: a9439b136f55f3f0e80ff053cd3b159da69ba362)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Code is related to the basetable templates files.
It had mixed tabs and spaces and was miss aligned in various places,
making it hard to read.
(Bitbake rev: cdaea8951df6b707afd1fefbf22295088256dd6f)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Until now cookies were used to save which columns were shown and which
were hidden in toaster tables. The tables from the templates also have
functionalities like sorting the entries on a certain column and
limiting the number of entries displayed on a page. The later however
were not saved using cookies. This patch brings this new feature.
The cookies are not saved only in the front-end. They are saved both
in the frontend in case the user uses the inputs/buttons to change
a parameter and also in the backend in case the user specifies manually
using GET variables the value of the parameters.
When no GET parameters are given the views will redirect the url to one
containg the parameters saved as cookies. When no cookies exist, default
values will be used.
[YOCTO #6126]
(Bitbake rev: 880b58c845e3a501fa90d24e1bd89c87ca84b709)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to avoid problems when using JSON serializer for
saving sessions, we move from storing the objects themselves
in the session to storing the object id and reloading the
object when retrieved.
This allows, for example, to use cookie-storage sessions if the
infrastructure owner so desires.
(Bitbake rev: 39d0f0c2e87d4b161f1eeaa2657e61b5a6bc9ee2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We enable support for starting builds on remote machines
through SSH. The support is limited to poky-based distributions.
We refactor localhost build support and we update
bldcontrol application tests to uniformely test the APIs
of localhost and SSH build controllers.
[YOCTO #6240]
(Bitbake rev: c2ad9c9bb83f61c171434324df8c4d5ee655a556)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We change the toaster starting script to make the
bitbake server listen on all interfaces on the local machine.
This is needed to be able to receive a controlling client
running on a remote machine.
(Bitbake rev: 137179eafca8d1a5a69b6302f8cc8961be3b45c4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We rename the projecttags in bldviewer.templatetags to
simple_projecttags in order to avoid conflict with the
similarly named tagset in toastergui.
The conflict leads to an intermittent bug where proper
tags are not read correctly since Django uses only the
module name as global tag library identificator.
(Bitbake rev: a37f2c194d7e59611177cb8755524b7ad702fe91)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the improved exception handling added in an earlier commit bitbake
now stops when recieving a SIGWINCH. This happens frequently when
disconnecting and reconnecting tmux sessions and bitbake didn't survive.
Restore old behaviour of ignoring interrupted system calls but keep
proper exception handling for other errors.
(Bitbake rev: 418358a595c75f45b8d15160ec42bbe569562d91)
Signed-off-by: Dan McGregor <dan.mcgregor@usask.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add new pages for the layer importing, layer details,
showing project builds and project configuration.
The pages are in read-only mode, but they're needed as
to be able to verify the quality of data in the system.
Write capabilities will be added in a subsequent patch.
[YOCTO #6595]
[YOCTO #6590]
[YOCTO #6591]
[YOCTO #6588]
[YOCTO #6589]
(Bitbake rev: eed9ae5c2a2bd7567e12ae9a4f02a5a966a1e1a3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add new pages for the all-machines and all-targets
project-related views.
We update the existing template structure to create
a base project view, similar to a base build view, that includes
a breadcrumb.
Updating existing all layers view to use the new structure.
We update methods in the models to provide corrent
information display.
[YOCTO #6592]
[YOCTO #6593]
(Bitbake rev: 973f582a19441c1ec67061160e4c50ce03ed7b68)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We move the code to calculate build progress as percent
and the ETA of the build to the model, so that they can be
reused across different pages.
(Bitbake rev: c2ced09e7ea4a1762d2788bb12a761734d20fd8e)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
