This reverts commit 99305c25fafa61064a1bb0849616e4627e3453d7.
After discussion we found various versions of selenium don't work
with this change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a user is authenticated with the server, report them as the owner of
a report
(Bitbake rev: a9fd4a45bb6e5ac9832835897f594f3bbf67e1aa)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allows users to self-service deletion of their own user accounts
(meaning, they can delete their own accounts without special
permissions).
(Bitbake rev: 2d4439948a5328a9768bca9eaec221eb82af3cb2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_TEST_HASHSERV_USERNAME and BB_TEST_HASHSERV_PASSWORD are provided
for a server admin user, the authentication tests for the external
hashserver will run. In addition, any users that get created will now be
deleted when the test finishes.
(Bitbake rev: 0e945d3dec02479df1157f48fd44223c2bfb34a3)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Outputting the stats in JSON format makes more sense as it's easier for
a downstream tool to parse if desired.
(Bitbake rev: 3a18066e479ab06bdb08e258fc4aacad5e73222e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake-hashclient command-line tool now has a lot more features
which should be tested, so add some tests for them.
(Bitbake rev: 178cf99673d7ddf8e0bb63a5a43331a18f3286d5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an API to retrieve the columns that can be queried on from the
database backend. This prevents front end applications from needing to
hardcode the query columns
(Bitbake rev: abfce2b68bdab02ea2e9a63fbb3b9e270428a0a6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an API to query the server for the usage of the database (e.g. how
many rows are present in each table)
(Bitbake rev: c9c1224447e147e0de92953bc85cea75670b898c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds API that allows a user admin to impersonate another user in the
system. This makes it easier to write external services that have
external authentication, since they can use a common user account to
access the server, then impersonate the logged in user.
(Bitbake rev: 71e2f5b52b686f34df364ae1f2fc058f45cd5e18)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for the hashserver to have per-user permissions. User
management is done via a new "auth" RPC API where a client can
authenticate itself with the server using a randomly generated token.
The user can then be given permissions to read, report, manage the
database, or manage other users.
In addition to explicit user logins, the server supports anonymous users
which is what all users start as before they make the "auth" RPC call.
Anonymous users can be assigned a set of permissions by the server,
making it unnecessary for users to authenticate to use the server. The
set of Anonymous permissions defines the default behavior of the server,
for example if set to "@read", Anonymous users are unable to report
equivalent hashes with authenticating. Similarly, setting the Anonymous
permissions to "@none" would require authentication for users to perform
any action.
User creation and management is entirely manual (although
bitbake-hashclient is very useful as a front end). There are many
different mechanisms that could be implemented to allow user
self-registration (e.g. OAuth, LDAP, etc.), and implementing these is
outside the scope of the server. Instead, it is recommended to
implement a registration service that validates users against the
necessary service, then adds them as a user in the hash equivalence
server.
(Bitbake rev: 69e5417413ee2414fffaa7dd38057573bac56e35)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an API to explicitly disconnect a client. This can be useful for
testing the auto-reconnect behavior of clients
(Bitbake rev: cb2fec7eaca20608faf4b8ff11ec8590ac7f2229)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Invalidate the loop in the client close() call so that it is not closed
twice (which is an error in the asyncio code)
(Bitbake rev: ef22f8ee82c242383625f078baafb629e45dad7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for Invocation Errors (that is, errors raised by the actual
RPC call instead of at the protocol level) to propagate across the
connection. If a server RPC call raises an InvokeError, it will be sent
across the connection and then raised on the client side also. The
connection is still terminated on this error.
(Bitbake rev: 50ee68175e7cf20a32bfbb176db2c47d7859da04)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the hash equivalence server is in read-only mode, it should still
return a unihash for a given "report" call if there is one.
(Bitbake rev: d0bbb98553f5f3451606bd5f089b36cfe4219dc2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an SQLAlchemy backend to the server. While this database backend is
slower than the more direct sqlite backend, it easily supports just
about any SQL server, which is useful for large scale deployments.
(Bitbake rev: e0b73466dd7478c77c82f46879246c1b68b228c0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Abstracts the way the database backend is accessed by the hash
equivalence server to make it possible to use other backends
(Bitbake rev: 04b53deacf857488408bc82b9890b1e19874b5f1)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a logging adaptor to the asyncrpc clients that prefixes log
messages with the client remote address to aid in debugging
(Bitbake rev: f4d64ce73c2449c008ff5d9b32376a2893ef7195)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for running the hash equivalence test suite against an
external hash equivalence implementation.
(Bitbake rev: c1fbc3f68b94905d19ffcf4a6da5b27f0bf14599)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
(Bitbake rev: d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support to the hash equivalence client and server to communicate
over websockets. Since websockets are message orientated instead of
stream orientated, and new connection class is needed to handle them.
Note that websocket support does require the 3rd party websockets python
module be installed on the host, but it should not be required unless
websockets are actually being used.
(Bitbake rev: 56dd2fdbfb6350a9eef43a12aa529c8637887a7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrites the asyncrpc client and server code to make it possible to have
other transport backends that are not stream based (e.g. websockets
which are message based). The connection handling classes are now shared
between both the client and server to make it easier to implement new
transport mechanisms
(Bitbake rev: 2aaeae53696e4c2f13a169830c3b7089cbad6eca)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test that the followings link are visible and clickable:
- OpenEmbedded
- BitBake
- Yocto Project
- Read the Toaster manual
- Contribute to Toaster
- Fixed typo: using instead of usign line:97
- Remove whitespace
(Bitbake rev: d4c35eaff3c152ee864a886aac8ad5476f7f29f7)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- check that the documentation link is visible
- check browser open new tab toaster manual when clicking on the documentation link
(Bitbake rev: befe970fcc7742cf266e03849be36c733bb09de6)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check the information icon is visible and clickable
Also if popup appearing when clicked and contain "Toaster version information"
(Bitbake rev: 0508fd74ebd43359473a9718edf0d146002041ad)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The main reason for using pytest is to be able to generate a positive test report
using the pytest-html plugin.
Integrating Pytest with Tox is a straightforward process, this can be
done using tox.ini instead of pytest.ini used to configure pytest, that
is another reason for using pytest. Tox is a tool that automates testing
across different virtual environments, it can help ensure application
will be tested against multiple Python versions and environments.
https://github.com/pytest-dev/pytest/blob/main/tox.ini
Generated reports create a historical record of test results over time.
This can help track the progress of the application's stability and quality
Documentation and Transparency: Test reports provide us a clear and detailed
documentation of the test results. They show what tests were executed,
which ones passed, and which ones failed. This transparency is critical
for understanding the current state of the application and its test coverage.
Communication: Test reports are an effective means of communication among community
to understand the testing progress and results.
Debugging, Troubleshooting Historical Tracking and Regression Testing:
In case of test failures, a detailed test report can be invaluable for debugging.
It provides information about the specific test case that failed,
the input data used, and any error messages.
(Bitbake rev: 6ba046b8d0d821e304c14b78ef6b00945e0ab453)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As mentionned in earlier commit, pytest-django takes a conservative approach
to enabling database access. By default our tests will fail if they try to access
the database, Only if we explicitly request database access will this be allowed,
using pytest marks to tell pytest-django our test needs database access.
A side effect of pytest mark, is test_case method marked is execute out of
scope of its module class, which create an inconsistance sequence and make fails
followings tests.
The scope of the ordering is global per default, e.g. tests with lower ordinal
numbers are always executed before tests with higher numbers in the same test session,
regardless of the module and class they reside in. This can be changed by using
the --order-scope option from module pytest-order.
To fix that i added execution order to tests suite using pytest-order.
(Bitbake rev: 8bf32459ab6d8ce7632fca0a4ce0f3aecc7495c2)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toastergui-unittest-data.xml fixture contains a release objects with
name="master" or an release with same name seem existing
- Change release name and fix bitbakeversion instead of bitbake_version
(Bitbake rev: 36b48905ea26e5173b5ab2369b90952bc13e9c0d)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove load and create build environment from tests/functional/functional_helpers
- Testcases in the FunctionalTestCase do not require a build process,
- Also marked setUpClass or class with pytest django_db, db ... not
working, as declared above of file functional_helpers.py, The database access
process runs as an external process, separate from the test case process
and outside the context of pytest.
(Bitbake rev: 35c02c26541c05a8d147f41d0855e09d8fba1a83)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The pytest framework makes it easy to write small, readable tests, and can scale
to support complex functional testing for applications and libraries.
- Pytest-html is a plugin for pytest that generates a HTML report for test results.
- Pytest-django allows us to test Django project/applications with the pytest testing tool.
- Pytest-env is a pytest plugin that enables us to set environment variables in a pytest.ini or pyproject.toml file
- Pytest-metadata is a plugin for pytest that provides access to test session metadata, required by pytest-html plugin
to provides metadata information in the HTML report like below:
Python 3.10.12
Platform Linux-6.2.0-35-generic-x86_64-with-glibc2.35
Packages:
- pytest: 7.4.2
- pluggy: 1.3.0
Plugins :
- django: 4.5.2
- metadata: 3.0.0
- order: 1.1.0
- html: 4.0.2
- env: 1.1.0
To make difference between bitbake and toaster python requirements,
a dedicated requirements file is created for toaster in root of toaster dir.
(Bitbake rev: 804d17df22391b2d18a68d8cb05a04841d4e551b)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running printdiff, messages like:
"Can't find a task we're supposed to have written out?"
might be shown. This happens when hashequivalence is enabled and is due
to the use of "hash" when "unihash" should be used (which defaults back
to hash). Changing this fixed various errors oe-selftest highlighted.
Also print the task ID when erroring about a missing task as it aids
debugging.
(Bitbake rev: e7783bbe0559abb91f5038cf83575c0f25138af2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If --runall is used with setscene tasks, it will still skip them if it
can. By marking the tasks as targets, this avoids that skipping
and means --runall deploy_source_date_epoch works as expected for
example.
(Bitbake rev: 8c6cd1e8dd224bc1cc92736cbb7e113fd1fb5aae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The IPC mechanism between runqueue.py and bitbake-worker is currently
not scalable:
The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker
has no information about the size of the message. Therefore, the bitbake-worker
is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")"
for each chunk received.
This does not scale, because queue.find has a linear complexity relative to the size of the queue,
and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI.
The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each
iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup
35 seconds before this fix, and 1.5 seconds after this fix).
This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be
received, and does not need to constantly search the whole queue for </tag>.
(Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updated tox file to fix enviroments variables.
to run tox use the following command:
tox -e env_name -c /path/to/file/tox.ini
See tox cli https://tox.wiki/en/latest/cli_interface.html
The variable env_name is linked to the python versions (py38, py39, py310).
If env_name option is not set, the test suite will run for all indicated versions.
It was also modified the webdriver options for chrome to run selenium test in console because of the error detected in the previous test run
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
(session not created: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
(Bitbake rev: 11dfd7c05642269b7a8bff2918667348d041ec79)
Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com>
Changes in v2
- Updated variable path BUILDDIR and EVENTREPLAY_DIR to use enviroment value
- Updated toaster-requirements.txt file path
- Removed flake8
- Added webdriver options to chrome
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes "2efb14648 toaster: Monitoring - implement Django logging system" when
running in a container.
When running in a container, the previous approach of using BASE_DIR
is not a writable path. Also, we really do not want to be writing logs into
the source tree, as the BASE_DIR was resolving to bitbake/lib/toaster/logs
Since Toaster is only ever running in an environment where oe-init-buildenv
or similar has been sourced, we should instead write the logs to BUILDDIR.
Using BUILDDIR to logs make path writable but django-log-viewer does'nt manage
to write logs using an absolute path as BUILDDIR, where the existing toaster_ui.log
was already being written.
Also drop the /logs/ directory, as it has not been created which also breaks
in a container environment
To handle the constraints linked to django-log-viewer and /logs/, we've updated
bitbake/bin/toaster to create a toaster_logs/ directory in BUILDDIR if it doesn't exist,
when toaster starts up.
Also manage to set BUILDDIR/toaster_logs/ as default location for toaster logs.
(Bitbake rev: efbd9d54f57be7a7a10f0b56e7e62c25974e99e6)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Tested-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pre-fetching of LFS blobs now also works if a destsuffix parameter is
supplied.
This also fixes issues with Yocto Kirkstone (and newer) builds, where
fetching the LFS blobs is retried unsuccessfully during do_unpack, as
network access is blocked for any task other than do_fetch.
(Bitbake rev: e411dc07d032be6811d0393c50a06fc28e669b24)
Signed-off-by: Ragesh Nair <ragesh.nair@duagon.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
