Currently the code ignores lightweight tags which has caused some user
complaints. We can't put the right search list in place easily since
the results don't come back in a good order, head happens to sort
before tags.
In the end I refactored the function so we get the complete list of
remotes and then we can filter it ourselves in the order we chose,
including checking for light weight tags, preferring the proper ones.
Hopefully this resolves the issues people have been seeing.
[YOCTO #6881]
(Bitbake rev: 07ad307065bb15a48f0015b9e4a643201abdc283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.
We can't change sh, we can ensure the user is warned about the problem
straight away though.
[YOCTO #7006]
(Bitbake rev: 86704281b79e524dccccc88cbf996b299b33bae2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases for cache purpoes we not only need to know which file
is going to be used but also which paths were considered. Add a
localpaths method which includes the history.
The core which() funciton already supports this, this just extends
the function to preserve the extra data we need. localpath becomes
just a special case of the case with history.
(Bitbake rev: ea5efeac5c1f7986666c979f789786f29fc1619a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.
[YOCTO #6934]
(Bitbake rev: 294bb9cad294423d4f8998405ceff58655f12660)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a little crude as the usage basically ignores that you're doing
an append operation, but for a lot of cases it will be sufficient.
(Bitbake rev: 24a28205ab680b6cc645d97b76c9855920608229)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix add-layer error message when a layer is already in BBLAYERS
* Ensure we show an error message if we can't find BBLAYERS at all
(Bitbake rev: 1c743fd2103730e27699dd55efc6914d3b0c3702)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hardcoding a temp directory is bad practice and leads to races between
the tests. There is no longer any good reason for doing this, drop it
and ensure the files get cleaned up correctly.
(Bitbake rev: 10a47b1ec7470c9e8c4ffe0bb35cdf6d1bb2ee2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a key, it should not exist in
d.keys(). This adds a test to cover the recently found data store bug.
(Bitbake rev: 16d5f40ad20fd08bf7a4d0e36200c739b5a9f59e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The versionstring tests hit the network so should only run when
network tests are enabled.
Also remove the print statement which confuses the test output and
add it to the test failure message instead.
(Bitbake rev: 6c046660cfc4fb3792a42aeafff91a13f68a2e89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a variable, it still shows up
in d.keys() when it should not. This patch addresses the issue.
(Bitbake rev: f28ee1bb03cb32d3757fbef67c9fbe143e3dadfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to fix multithread usage of latest_versionstring moves package_custom_regex_comp
from class to method level because need to be defined by package.
Remove code for build url's with /download suffix because it's deprecated since you can
specify the download directory using package_regex.inc file.
(Bitbake rev: 231cae9f9b552ec6737795c098d1de426b5adcbc)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BBFILE_PATTERN_IGNORE_EMPTY variable to allow ignoring the fact
that a regex specified in BBFILE_PATTERN for a particular collection
doesn't match any recipes. This will be used in OpenEmbedded in the
workspace layers created by "devtool" which may not always contain any
recipes (which is not cause for warning the user).
(Bitbake rev: 19c74aaa2836a88d666f0032452fac521689ab6b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add a generic edit_metadata_file() function to modify variable
assignments in any metadata file (conf, bb, bbappend) using a callback
for flexibility
* Add a specific edit_bblayers_conf() function to modify
conf/bblayers.conf and add and/or remove layers from the BBLAYERS
value within it.
(Bitbake rev: aa03a28b442549dd8ffe92ae4d6390f62202a76a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a nostamp task is depended on by a non-nostamp task, then we want the
signature of that task to change such that it re-executes afterwards.
This is an unusual situation, but we want this to work in OE in
externalsrc.bbclass so that compilation happens every time it is
requested.
(Bitbake rev: 73498afc3d45beede5b8f24a9acd523a1663b793)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running bitbake inside make results in the exported environment variable
MAKEOVERRIDES="${-*-command-variables-*-}", which the shell chokes on
when trying to expand it. But of course, it probably shouldn't have been
trying to expand it in the first place -- so just escape the dollar
sign.
(Bitbake rev: 18cd0ce6a55c9065c3f1bf223b47d817b5efcd8f)
Signed-off-by: Richard Tollerton <rich.tollerton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.
The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.
The "-w" option can also be supplied as the BBEVENTLOG
environment variable.
We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.
We modify the toasterui to fix minor issues with reading
events from an event log file.
Performance impact is undetectable under no-task executed builds.
(Bitbake rev: 1befb4a783bb7b7b387d4b5ee08830d9516f1ac2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update and add logs throughout the code in order to help
with development. The extra logging is turned off by default,
but it can be enabled by using environment variables.
All logging happens through the Python logging facilities.
The toaster UI will save a log of all incoming events if the
TOASTER_EVENTLOG variable is set.
If TOASTER_SQLDEBUG is set all DB queries will be logged.
If TOASTER_DEVEL is set and the django-fresh module is available,
the module is enabled to allow auto-reload of pages when the
source is changed.
(Bitbake rev: 10c27450601b4d24bbb273bd0e053498807d1060)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a BuildArtifacts class to store data about files
discovered during the build process and not stored anywhere
else.
Small cosmetic changes in the toasterui.
Add model methods to return file path display data relative
to the build environment instead of absolute file paths.
[YOCTO #6834]
(Bitbake rev: bbe24d912869312d561be199b2c029b0c898e049)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
(Bitbake rev: 497404e8484b7ca7c11e459bf0845642156eb677)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shellshock patches changed the way bash functions are exported.
Unfortunately different distros used slightly different formats,
Fedora went with BASH_FUNC_XXX()=() { echo foo; } and Ubuntu went with
BASH_FUNC_foo%%=() { echo foo; }.
The former causes errors in dealing with out output from emit_env,
the functions are not exported in either case any more.
This patch handles things so the functions work as expected in either
case.
[YOCTO #6880]
(Bitbake rev: f28f37220e7787721a31b659521a1c44ebea92bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.
(Bitbake rev: ac45ea848901b0f6cd23087b662dde8ce9cd807e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using external tinfoil-based utilities, it is useful to be able to
turn off most of the event handlers; for example sstate_eventhandler
doesn't like being sent events for any recipe which has been skipped.
(Bitbake rev: 41236c28985a3b66b3737382a94e39dbf6480160)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.
(Bitbake rev: 49dbcfbc56a206964acc5de761bba31be0283ba1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function that allows executing a flat python function (defined
with def funcname(args): ...).
(Bitbake rev: 20e6939ebcb62e08a9a7ad586a915dfe368136a0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The defaultval field is intended to be internal and the only use of that field
outside of data.py is to skip over it when iterating over a value's flags.
For clarity and convenience, rename the field to _defaultval so that it is
considered internal and not exposed through the data API.
(Bitbake rev: 2800958dadaa5c055ba21d52c98d842d360f0785)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if an invalid PR service is selected the server will error
with a traceback. This is because its set into the error state and the
setFeature code will then fail since its not in the initial state.
Modifying the featureset in the error state is acceptable, we just need
to ensure we don't trigger a reset, that would happen from whichever
code handles the error.
[YOCTO #6934]
(Bitbake rev: c52841445d8db8f84c4da34203b195fea5874247)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test case for cups is needed because match only 2.0.0
versions see VERSION=2\.0\.0 in the previous string.
(Bitbake rev: 148aba30155f4de17f6c6fb9b4c37f08a2db202b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some upstream sites put the name of the package in the body of href tags,
i.e. <a href="#43">somepackage-v1.4.10.tar.gz </a>.
(Bitbake rev: 4fa8542a8880d5749fbb3382706e011b012024d0)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Validate if package contain version string if not return the current
version cases for spectrum-fw and corpus recipes.
_check_latest_version return the latest version available don't
take into account the current version previous this only return
the upstream version if it greater than the current version.
(Bitbake rev: 91a7ac8c7f87f98e366585cf9720ec35b0790bae)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package_custom_regex_comp is built with the current package name and
then used to search upstream version this reduces custom regex'es in
sites that have different packages in the same directory.
(Bitbake rev: a69d5d48c4bf9a3df3121cc4e69ba5d7e947ad8d)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Latest version string only try to find latest directory when REGEX_URI
isn't specified to avoid unnecessary processing and makes code easier
(Bitbake rev: afc33ec7cdb7d8ee3602a23fa973551ca5510ac4)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in changes that allow a toasterUI coming in
from 'master' branch to record data from a 'daisy' or 'dizzy'
bitbake server.
This is needed to allow Toaster to record builds running
on older branch releases.
(Bitbake rev: 8d75e28e0688a6520311afce36543175f36910b3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we have a managed build, we match the layers used for build
with the layers configured for project, as we know where the layers
are coming from
[YOCTO #6962]
(Bitbake rev: e02ec052a62cbc476bdac65cb7cea1167ce04781)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing ToasterUI to log toaster exceptions on a different level than
build errors.
Updating the build dashboard to show Toaster exceptions.
We add extra logging to console for exceptions.
Fixed a problem where packages database entries were created instead of
being looked up in the database, conficting with entries created to
satisfy dependency information.
Toaster now checks for invalid states at startup and performs needed
cleanups.
Removed loading reference to jquery-ui.min.css as we do not have this
file.
(Bitbake rev: 2378812bc24d433125fb940f110154f0ce638448)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.
(Bitbake rev: 40d2ae0723de2bf5fee343faafb4afda40546839)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The oe.utils.contains function has been removed from OE-Core metadata
as the references for it has been replaced to use the
bb.utils.contains.
(Bitbake rev: 5cfdebe7a67dccc7552ff80c1ccc970e36d562df)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This state was renamed in bitbake 12e9d3 but this use of it wasn't changed.
[ YOCTO #5445 ]
(Bitbake rev: 196106100e2dedfacce95be0ee6def94c5a80f27)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
loginfo "op" was being set to invalid values in saveConfigurationVar it was
working because set is a python global but append is not. This replaces both
the "op" with strings and removes the crash when calling appendConfigurationVar
(Bitbake rev: 2a8e847de85546d43600a561f9c63aa36bd69222)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improve the performance of data logging in toasterui.
We modify the data queries used to:
* cache searching in memory
* insert in bulk (i.e. multiple values per insert, where possible)
On development test rig (networked mysql), on no-op build,
time for data recording is reduced from 4:10 to 1:30 (minutes).
We also improve the logging, so it is easier to detect
toasterui errors.
(Bitbake rev: d42784432f927f58730caf80546c66772e0fec89)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: a8272e22b7819e0e8afd8e291d276f5f28fc0007)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: f71c8c0354e87fed80bc845db6728e6e18ce9c4d)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
