Updates for YOCTO #12891 allowed a user to have a directory
structure different to that of yocto (bitbake isn't inside
oe-core) whereas the definition of OE_ROOT in the main toaster
binary still assumes the same while checking for .templateconf
and hence we see an error on the cmdline in such cases:
bash: <repo-path>/bitbake/bin/../../.templateconf: No such file or directory
The change here now allows the user to provide OE_ROOT through
the environment in such cases and otherwise defaults to the older
mechanism to fix this issue.
(Bitbake rev: 9cc6fe4be797e106899d2448797a4b3af8eace4e)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case of a sublayer of an existing layer, where the sublayer and
main layer share a path, the system may not match the paths properly resulting
in:
No bb files matched BBFILE_PATTERN_sublayer '^/path/main/sublayer'
because it has already matched the main layer.
Fix this issue by sorting the collection items based on the pattern, using
longest to shortest. Obviously regex wildcards could still be an issue
but these are typically not used, so this simply fix should work in the
existing cases.
(Bitbake rev: 1787cef7221b88f6920ea70fadaffc117d84c7aa)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TOASTER_DIR is used for higher level toaster artifacts
such the SQL DB and creating toaster internal build
directories for projects. Prior to this change it was
evaluated as `dirname $BUILDDIR` and user had no control
over it. This change allows to override this variable
from the command line for more flexibility. The variable
defaults to its original setting if the optional argument
is not passed.
[YOCTO #12891]
(Bitbake rev: e073775d3b6980fc8004ae28a3ccc3c5bbf50fb2)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The layer index module is expected to be used by various parts of the system
in order to access a layerindex-web (such as layers.openembedded.org) and
perform basic processing on the information, such as dependency scanning.
Along with the layerindex implementation are associated tests. The tests
properly honor BB_SKIP_NETTESTS='yes' to prevent test failures.
Tests Implemented:
- Branch, LayerItem, LayerBranch, LayerDependency, Recipe, Machine and
Distro objects
- LayerIndex setup using the layers.openembedded.org restapi
- LayerIndex storing and retrieving from a file
- LayerIndex verify dependency resolution ordering
- LayerIndex setup using simulated cooker data
(Bitbake rev: fd0ee6c10dbb5592731e56f4c592fe687682a3e6)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix typo in shutdown code to kill threads when "kill -0" is not enough.
Use the '--noreload' flag for 'runserver' so that there are no extra
and unaccounted threads.
[YOCTO #12555]
(Bitbake rev: 256990943075e89cb9aee2bc6488344b6783e07b)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a 'nobuild' option for starting Toaster without the project
and hosted builds support. This allows a Toaster host to provide
local build statistics without opening the host to external users
building projects.
[YOCTO #12315]
(Bitbake rev: 2d14d6004b6add5ce07295fff1144ade2e54e1c9)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs the ability to allow custom extensions to execute when
Toaster is started and stopped. Toaster will look for a custom
extension script in the fixtures directory and execute any applicable
hooks.
[YOCTO #11938]
(Bitbake rev: 12a73f6914488029f2b9dd680e004fba7dde41af)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.
(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)
Fixes [YOCTO #11844].
(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python style recommends underscore based naming rather than camelCase,
and thus the former has been used for most of tinfoil's functions. Add
an underscored version of parseRecipes() for consistency and change the
one place we call it to use the new version.
(Bitbake rev: 821f6c41d850752d2bcc2ccd4f8e75b2897a0a3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are four main API deprecations in Django-1.10:
(a) String view arguments to url() must be replaced by
the explicit class reference
(b) New TEMPLATES stucture in settings.py consolidates
TEMPLATE_DIRS, TEMPLATE_CONTEXT_PROCESSORS,
TEMPLATE_LOADERS, TEMPLATE_STRING_IF_INVALID, and
TEMPLATE_DEBUG
(c) patterns() wrapper in url() is removed, with
urlpatterns now a simple list
(d) NoArgsCommand in commands() must be replace by
BaseCommand, and handle_noargs() changed to
handle()
Also, the Django version checker must be updated to accept
two digit sub-version numbers (e.g. "1.8" < "1.10")
[YOCTO #11684]
(Bitbake rev: e4c7a94fac7a53fc146387a57e5a09b9ec3caca0)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tests for bitbake event module were created on bb.tests.event.
This change is to include them on the default test list in
bitbake-selftest script.
[YOCTO #10368]
(Bitbake rev: 22cc318be1aff16f1b653ff0650fe71d0e9f60bb)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the 'noweb' option for Toaster is used, perform the database
check/create if the Toaster database does not yet exist.
This will allow Toaster to not fail if the first use is with 'noweb'.
This avoids potentially clashing database updates if there are
multiple overlaping 'noweb' sessions (for example with a CI system).
If the user wished to update the database, they can either use the
explicit "lsupdate" command or (re)start a web hosted Toaster session
(which is gated by the webserver's PID).
[YOCTO #11378]
(Bitbake rev: 910b96b9894c712aa32b5d4dadda88b766d86e35)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the original implementation, "bitbake-layers add-layers <layer>"
succeeded without error checking. This will further introduce
failures in recipe parsing only when "bitbake" command is executed.
Adding a meta layer without its dependency layer(s) should failed
and exit the process gracefully.
Added extra argument "-F" to force add a layer without checking
layer dependency.
[YOCTO #10913]
(Bitbake rev: 705ab252e631903e6d2e46202b419a9e8adcd861)
Signed-off-by: Phoong Stanley Cheong Kwan <stanley.cheong.kwan.phoong@intel.com>
Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script will be used by the git fetcher to create shallow mirror tarballs.
usage: git-make-shallow [-h] [--ref REF] [--shrink] REVISION [REVISION ...]
Remove the history of the specified revisions, then optionally filter the
available refs to those specified.
positional arguments:
REVISION a git revision/commit
optional arguments:
-h, --help show this help message and exit
--ref REF, -r REF remove all but the specified refs (cumulative)
--shrink, -s shrink the git repository by repacking and pruning
While git does provide the ability to clone at a specific depth, and fetch all
remote refs at a particular depth, the depth is across all branches/tags, and
doesn't provide the flexibility we need, hence this script.
Refs (branches+tags) can be filtered, as the process of history removal scales
up rapidly with the number of refs. Even the existing `git fetch --depth=` is
extremely slow on an upstream kernel repository with all the branches and tags
kept.
This uses the same underlying mechanism to implement the history removal which
git itself uses (.git/shallow), and the results, when configured similarly, are
in line with the results git itself produces with `fetch --depth`.
(Bitbake rev: 0254020f0e1911c0eaf99111b91828d2a74a4ee1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
bitbake-layers show-recipes | less
press "q" to exit
There will be a Broken pipe error output as follows:
"BrokenPipeError: [Errno 32] Broken pipe"
(Bitbake rev: 4fca9a07f2d6b0544977112672b786982d7bb8f2)
Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the move over to argparse we've made the two signature file arguments
optional and thus if -t is not in use we need to explicitly check if at
least one signature file has been specified - and if not, show an error
and the command-line help.
(Bitbake rev: 9011366689b26305281fcb2d412dcacece917e18)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the output is a TTY, add colour to the output in order to make it
easier to read. At the moment this is fairly basic, just add colour to
the "titles" of each change and to the diff output.
I tried to introduce this without changing the code too much - rather
than moving everything over to the new python formatting style, I've
introduced a color_format() function which takes care of the colour
formatting, either accepting additional format arguments or
alternatively leaving the caller to use the old-style formatting (%) to
insert values.
(Bitbake rev: 04a023c8fdea1e1812fcdcaf00345aab59f9abe1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the -t option which recurses to find the ultimate cause of a
signature change, it was hardcoded to take the last two executions of
the specified task. On the other hand, if you have two specific task
hashes (say from bitbake output, or some other tool) then you'll want to
pick those, so provide an option to specify those as well. (Note, the
new -s option needs to be specified alongside -t rather than instead of
it.)
(Bitbake rev: d9813b1a4223cf8dc80cab90e467ddf4bf8d1078)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Argparse is a bit easier to deal with than optparse, and since we're
about to add some options, migrate this script over.
(Bitbake rev: 7f130e0b5ce6cfc6b35176465f260092cd3b3d64)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When dumping changes to signatures e.g. output of bitbake -s printdiff,
if for example a function has changed, it's much more readable to see a
unified diff of the changes rather than just printing the old function
followed by the new function, so use difflib to do that.
Note: I elected to keep to one item in the returned list per change,
rather than one line per line of output, so that the caller can still
look at changes individually if needed. Thus I've added some handling to
bitbake-diffsigs to split the change into lines so that each line is
displayed indented.
(Bitbake rev: 4d254ae63a35231c98e3f73f669b040ed1144042)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This logic doesn't work in practice, certainly not with current versions
where sigdata files are preserved in the stamps directory and therefore
there will often be multiple sigdata files - you can now easily get
files for the same signature from sstate and the stamps directory with the
result that bitbake-diffsigs reports nothing has changed. Instead, let's
change the find_siginfo function in OE-Core to simply not return
duplicates so we don't have to filter them out here.
(Bitbake rev: f0d7ab259d8ef95643e7229474b7850608aa4426)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If just one of the two signatures we want to compare aren't available,
report that one rather than misleadingly claiming both are missing.
(Bitbake rev: c87764b9147792a10efad3ed5378f36f0a055bc6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We weren't picking the right files to compare here - according to the
order in which the list is sorted (by mtime), we need to be taking the
last two items and not the first two.
(Bitbake rev: 99f49b56115b1f8d1a0a0b911da62ffd1f997b5f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We use this code to set up a logger with colour in a number of different
places, so create one function that does this and make some of bitbake's
utility scripts use it.
(Bitbake rev: b1ba7d1cc8ec33e2d081230287abd07f52136097)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently shows the name of the signature files that were found when
--task is used.
(Bitbake rev: 751b06c25d22eea8240f9429cb49874082245e52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently shows the name of the signature file that was found when
--task is used.
(Bitbake rev: 84533c6dbd175a51f4dd59735161adfd10056888)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use --task <recipe> <task> to dump the signature info for a given
recipe and task. This is similar to the --task option of
bitbake-diffsigs.
(Bitbake rev: bdc4356c7afc542b67b78e4e5225b813d7668ecd)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tinfoil.prepare method can raise Exceptions when is parsing initial
data so add this call inside try, finally to avoid get blocked for
don't call tinfoil.shutdown().
The tinfoil_init function was remove because isn't make sense now since
tinfoil.prepare() needs to be inside try, finally closures.
Example of raised exception and gets blocked:
$ bitbake-layers add-layer ~/repos/meta-openembedded/meta-python/
Traceback (most recent call last):
File "/home/alimon/repos/poky/bitbake/bin/bitbake-layers", line 83, in
main
tinfoil.prepare(True)
...
File "/home/alimon/repos/poky/bitbake/lib/bb/tinfoil.py", line 268, in
run_command
raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
File "/home/alimon/repos/poky/bitbake/lib/bb/command.py", line 81, in
runCommand
result = command_method(self, commandline)
...
File "/home/alimon/repos/poky/bitbake/lib/bb/cooker.py", line 1314, in
handleCollections
raise CollectionError("Errors during parsing layer configuration")
bb.cooker.CollectionError: Errors during parsing layer configuration
(Bitbake rev: 61ebe9a026652e32f9482f66fc3fe7a8672fc093)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a superfluous section in toaster start/stop script that was
setting the TOASTER_DIR and had a set of comments around that setting.
This was done in two places and only the last one was effective. This
patch removes the spurious section to make it clearer what was
happening and what TOASTER_DIR was actually getting set to.
(Bitbake rev: e551e73a094f5a1937bce78de9e41908bf3e07c3)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a superfluous section in toaster start/stop script that was
setting the TOASTER_DIR and had a set of comments around that setting.
This was done in two places and only the last one was effective. This
patch removes the spurious section to make it clearer what was
happening and what TOASTER_DIR was actually getting set to.
(Bitbake rev: 1f0eac0a172a4fbe1799675f2c3ce989743bd862)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited.
We really need a way to know this from the metadata and this new variable
provides this in worker context. This means existing code can stop having
to guess.
(Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the tinfoil2 changes resulted in TOASTER_DIR being cleared by
the memory resident bitbake server toaster starts up. This prevented
toaster from being able to connect to its sqlite database. Adding
TOASTER_DIR to the BB_ENV_EXTRAWHITE list stops the cooker from clearing
out the TOASTER_DIR variable.
[YOCTO #9252]
(Bitbake rev: 2420953b3d03551d8254609300ca572717aecdcd)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used
by the installation process for the extensible SDK), we don't actually
need the whitelisted real tasks to execute - we just need to have them
in the dependency tree so that we get all of the setscene tasks they
depend on to run. Therefore we can actually dry-run those real tasks
i.e. they won't be run (and thus we won't waste a significant amount of
time doing so) and won't be stamped as having run either. We do already
have a dry-run mode in BitBake (activated by the -n or --dry-run command
line option), but it dry-runs the setscene tasks as well which we don't
want here.
Note that this has no effect on the checking we are doing with
BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are
scheduled to run - that's handled separately.
Fixes [YOCTO #10369].
(Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
(Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we use python3, we should refer to pip3, not pip.
[YOCTO #10774]
(Bitbake rev: 99136f5f591deef0c96d9aea2dbea1c216f38121)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking further at the CPU loads on systems running large numbers of tasks,
the following things helps performance:
* Loop on waitpid until there are no processes still waiting
* Using select to wait for the cooker pipe to be writable before writing
avoiding pointless 100% cpu usage
* Only reading from worker pipes that select highlights are readable
(Bitbake rev: 9375349e27b08b4d1cfe4825c042d4c82120e00b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Write the pid file out in the start up of this management command. This
ensures this has happened instead of relying on the shell command having
been run which may or may not be the case. This also makes it simpler for
testing.
Couple of clean ups of runbuilds as identified by pyflake
(Bitbake rev: 999e980ee1a58d16f33ef6c0e41aecdcd0206f39)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
