mirrors/poky
1
0
Fork 0
mirror of https://git.yoctoproject.org/poky synced 2026-04-13 14:02:21 +02:00
Code Issues Packages Projects Releases Wiki Activity

valgrind: Improve non-deterministic ptest reliability

Browse Source 
Several of the valgrind tests (particulaly helgrind) are unreliable and
can fail with a different output.

Particularly, there is a higher chance of failure on QEMU instances with
SMP enabled and on systems with more interrupts such as laptops on powersave.

The tests have been reported upstream as being unreliable dating back
over 5 years, due in part to the ordering of threads during
an "unwinding" process in helgrind.
https://bugs.kde.org/show_bug.cgi?id=345121
https://bugs.kde.org/show_bug.cgi?id=430321

A workaround to improve the reliability of such tests is to force them
to run on a single CPU core using taskset. This greatly reduces the
chance of a failure.

>From my testing, I have found it can help reduce the rate of failures
on both a laptop and QEMU by over 5x. Stress-testing in QEMU for several
hours did not result in a failure while running the test normally did.

The flaky or undeterministic thread-based tests are defined in the
taskset_nondeterministic_tests file. These test cases will be run with
taskset 0x00000001 to run on a single CPU core rather then the regular
test.

The edited run-ptest executes the flaky tests first, then ignores them
to not duplicate the results from the main tests. Everything modified is
restored when testing is complete.

The drawback is that this isn't a foolproof solution. It helps the tests
fail much less frequently, and considering how this issue has been documented
for a long time, a workaround such as this is needed.

(From OE-Core rev: b318944dd72ca7b0408e955f3599381ab3ac3ba8)

Signed-off-by: Tony Tascioglu <tony.tascioglu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Tony Tascioglu
2021-06-04 10:07:56 -04:00
committed by Richard Purdie
parent 48ecf617b5
commit be491f7343
2 changed files with 16 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
meta/recipes-devtools/valgrind/valgrind/run-ptest
View File

@@ -32,6 +32,13 @@ if [ "$arch" = "aarch64" ]; then
done
fi
echo "Run flaky tests using taskset to limit them to a single core."
for i in `cat taskset_nondeterministic_tests`; do
taskset 0x00000001 perl tests/vg_regtest --valgrind=${VALGRIND_BIN} --valgrind-lib=${VALGRIND_LIBEXECDIR} --yocto-ptest $i 2>&1|tee ${LOG}
mv $i.vgtest $i.IGNORE
done
cd ${VALGRIND_LIB}/ptest && ./tests/vg_regtest \
--valgrind=${VALGRIND_BIN} \
--valgrind-lib=${VALGRIND_LIBEXECDIR} \
@@ -56,6 +63,11 @@ for i in `cat remove-for-all`; do
mv $i.IGNORE $i.vgtest;
done
echo "Restore flaky and other non-deterministic tests"
for i in `cat taskset_nondeterministic_tests`; do
mv $i.IGNORE $i.vgtest;
done
echo "Failed test details..."
failed_tests=`grep FAIL: ${LOG} | awk '{print $2}'`
for test in $failed_tests; do

4
meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests Normal file
View File

@@ -0,0 +1,4 @@
helgrind/tests/hg05_race2
helgrind/tests/tc09_bad_unlock
drd/tests/bar_bad
drd/tests/bar_bad_xml