should tests pass release builds?
On my x86-64 Parabola Linux machine, if I run on latest git master 6412fda the tests for RELEASE build:
make release && sudo make install && cd build.release/mtest && make && ctest
Then I get some failures:
Test project /home/e/MuseScore/build.release/mtest
Start 1: tst_album
1/46 Test #1: tst_album ........................ Passed 1.22 sec
Start 2: tst_barline
2/46 Test #2: tst_barline ...................... Passed 0.75 sec
Start 3: tst_beam
3/46 Test #3: tst_beam .........................***Failed 0.93 sec
Start 4: tst_breath
4/46 Test #4: tst_breath ....................... Passed 0.75 sec
Start 5: tst_chordsymbol
5/46 Test #5: tst_chordsymbol .................. Passed 0.80 sec
Start 6: tst_clef
6/46 Test #6: tst_clef ......................... Passed 0.75 sec
Start 7: tst_clef_courtesy
7/46 Test #7: tst_clef_courtesy ................ Passed 0.76 sec
Start 8: tst_compat
8/46 Test #8: tst_compat .......................***Failed 2.25 sec
Start 9: tst_concertpitchbenchmark
9/46 Test #9: tst_concertpitchbenchmark ........ Passed 1.66 sec
Start 10: tst_copypaste
10/46 Test #10: tst_copypaste ....................***Failed 5.18 sec
Start 11: tst_copypastesymbollist
11/46 Test #11: tst_copypastesymbollist ..........***Failed 2.33 sec
Start 12: tst_dynamic
12/46 Test #12: tst_dynamic ...................... Passed 0.52 sec
Start 13: tst_earlymusic
13/46 Test #13: tst_earlymusic ................... Passed 0.75 sec
Start 14: tst_element
14/46 Test #14: tst_element ...................... Passed 0.52 sec
Start 15: tst_hairpin
15/46 Test #15: tst_hairpin ...................... Passed 0.51 sec
Start 16: tst_instrumentchange
16/46 Test #16: tst_instrumentchange ............. Passed 0.79 sec
Start 17: tst_join
17/46 Test #17: tst_join ......................... Passed 0.79 sec
Start 18: tst_keysig
18/46 Test #18: tst_keysig ....................... Passed 0.82 sec
Start 19: tst_benchmark
19/46 Test #19: tst_benchmark .................... Passed 8.05 sec
Start 20: tst_links
20/46 Test #20: tst_links ........................ Passed 0.76 sec
Start 21: tst_parts
21/46 Test #21: tst_parts ........................ Passed 2.54 sec
Start 22: tst_measure
22/46 Test #22: tst_measure ...................... Passed 1.01 sec
Start 23: tst_midi
23/46 Test #23: tst_midi ......................... Passed 1.38 sec
Start 24: tst_midimapping
24/46 Test #24: tst_midimapping .................. Passed 3.70 sec
Start 25: tst_note
25/46 Test #25: tst_note ......................... Passed 0.80 sec
Start 26: tst_plugins
26/46 Test #26: tst_plugins ...................... Passed 0.56 sec
Start 27: tst_repeat
27/46 Test #27: tst_repeat ....................... Passed 0.98 sec
Start 28: tst_selectionfilter
28/46 Test #28: tst_selectionfilter .............. Passed 1.01 sec
Start 29: tst_selectionrangedelete
29/46 Test #29: tst_selectionrangedelete ......... Passed 0.84 sec
Start 30: tst_spanners
30/46 Test #30: tst_spanners .....................***Failed 0.86 sec
Start 31: tst_split
31/46 Test #31: tst_split ........................ Passed 0.80 sec
Start 32: tst_splitstaff
32/46 Test #32: tst_splitstaff ................... Passed 0.75 sec
Start 33: tst_timesig
33/46 Test #33: tst_timesig ...................... Passed 0.72 sec
Start 34: tst_tools
34/46 Test #34: tst_tools ........................***Failed 0.98 sec
Start 35: tst_transpose
35/46 Test #35: tst_transpose .................... Passed 0.43 sec
Start 36: tst_tuplet
36/46 Test #36: tst_tuplet ....................... Passed 0.40 sec
Start 37: tst_text
37/46 Test #37: tst_text ......................... Passed 1.40 sec
Start 38: tst_importmidi
38/46 Test #38: tst_importmidi ................... Passed 1.16 sec
Start 39: tst_capella_io
39/46 Test #39: tst_capella_io ...................***Failed 0.92 sec
Start 40: tst_biab
40/46 Test #40: tst_biab ......................... Passed 0.66 sec
Start 41: tst_mxml_io
41/46 Test #41: tst_mxml_io ...................... Passed 19.10 sec
Start 42: tst_guitarpro
42/46 Test #42: tst_guitarpro ....................***Failed 3.09 sec
Start 43: tst_scripting
43/46 Test #43: tst_scripting .................... Passed 0.65 sec
Start 44: tst_ove_bdat
44/46 Test #44: tst_ove_bdat .....................***Failed 0.96 sec
Start 45: tst_ove_ove3
45/46 Test #45: tst_ove_ove3 ..................... Passed 0.69 sec
Start 46: tst_ove_structure
46/46 Test #46: tst_ove_structure ................ Passed 0.71 sec
80% tests passed, 9 tests failed out of 46
Total Test time (real) = 78.01 sec
The following tests FAILED:
3 - tst_beam (Failed)
8 - tst_compat (Failed)
10 - tst_copypaste (Failed)
11 - tst_copypastesymbollist (Failed)
30 - tst_spanners (Failed)
34 - tst_tools (Failed)
39 - tst_capella_io (Failed)
42 - tst_guitarpro (Failed)
44 - tst_ove_bdat (Failed)
But normally when run debug, I don't get failures. I'm wondering if anyone knows why RELEASE would fail when DEBUG doesn't. I'd like the tests to be able to pass RELEASE if DEBUG passes. Just asking before I look more closely at these test scripts.
Comments
I'd think they should pass in any case. Doesn't help the users if something works as per the tests in debug build but they got a release build and the same stuff doesn't work there
I agree. I might try looking into this to see why fail...will be difficult without being able to step through the code, but I think there is a way to set qDebug statments to print, or I can always just replace those with regular console out. I did the exact same tests on debug:
and they all pass:
Test project /home/e/MuseScore/build.debug/mtest Start 1: tst_album 1/45 Test #1: tst_album ........................ Passed 2.46 sec Start 2: tst_barline 2/45 Test #2: tst_barline ...................... Passed 1.20 sec Start 3: tst_beam 3/45 Test #3: tst_beam ......................... Passed 1.51 sec Start 4: tst_breath 4/45 Test #4: tst_breath ....................... Passed 1.20 sec Start 5: tst_chordsymbol 5/45 Test #5: tst_chordsymbol .................. Passed 1.34 sec Start 6: tst_clef 6/45 Test #6: tst_clef ......................... Passed 1.23 sec Start 7: tst_clef_courtesy 7/45 Test #7: tst_clef_courtesy ................ Passed 1.25 sec Start 8: tst_compat 8/45 Test #8: tst_compat ....................... Passed 2.17 sec Start 9: tst_concertpitchbenchmark 9/45 Test #9: tst_concertpitchbenchmark ........ Passed 4.58 sec Start 10: tst_copypaste 10/45 Test #10: tst_copypaste .................... Passed 5.65 sec Start 11: tst_copypastesymbollist 11/45 Test #11: tst_copypastesymbollist .......... Passed 2.83 sec Start 12: tst_dynamic 12/45 Test #12: tst_dynamic ...................... Passed 0.79 sec Start 13: tst_earlymusic 13/45 Test #13: tst_earlymusic ................... Passed 1.98 sec Start 14: tst_element 14/45 Test #14: tst_element ...................... Passed 0.79 sec Start 15: tst_hairpin 15/45 Test #15: tst_hairpin ...................... Passed 0.79 sec Start 16: tst_instrumentchange 16/45 Test #16: tst_instrumentchange ............. Passed 1.25 sec Start 17: tst_join 17/45 Test #17: tst_join ......................... Passed 1.24 sec Start 18: tst_keysig 18/45 Test #18: tst_keysig ....................... Passed 1.22 sec Start 19: tst_benchmark 19/45 Test #19: tst_benchmark .................... Passed 20.81 sec Start 20: tst_parts 20/45 Test #20: tst_parts ........................ Passed 3.04 sec Start 21: tst_measure 21/45 Test #21: tst_measure ...................... Passed 1.64 sec Start 22: tst_midi 22/45 Test #22: tst_midi ......................... Passed 2.45 sec Start 23: tst_midimapping 23/45 Test #23: tst_midimapping .................. Passed 4.46 sec Start 24: tst_note 24/45 Test #24: tst_note ......................... Passed 1.29 sec Start 25: tst_plugins 25/45 Test #25: tst_plugins ...................... Passed 0.83 sec Start 26: tst_repeat 26/45 Test #26: tst_repeat ....................... Passed 1.70 sec Start 27: tst_selectionfilter 27/45 Test #27: tst_selectionfilter .............. Passed 1.56 sec Start 28: tst_selectionrangedelete 28/45 Test #28: tst_selectionrangedelete ......... Passed 1.34 sec Start 29: tst_spanners 29/45 Test #29: tst_spanners ..................... Passed 1.31 sec Start 30: tst_split 30/45 Test #30: tst_split ........................ Passed 1.24 sec Start 31: tst_splitstaff 31/45 Test #31: tst_splitstaff ................... Passed 1.08 sec Start 32: tst_timesig 32/45 Test #32: tst_timesig ...................... Passed 1.21 sec Start 33: tst_tools 33/45 Test #33: tst_tools ........................ Passed 1.79 sec Start 34: tst_transpose 34/45 Test #34: tst_transpose .................... Passed 0.75 sec Start 35: tst_tuplet 35/45 Test #35: tst_tuplet ....................... Passed 0.70 sec Start 36: tst_text 36/45 Test #36: tst_text ......................... Passed 1.90 sec Start 37: tst_importmidi 37/45 Test #37: tst_importmidi ................... Passed 2.06 sec Start 38: tst_capella_io 38/45 Test #38: tst_capella_io ................... Passed 1.53 sec Start 39: tst_biab 39/45 Test #39: tst_biab ......................... Passed 1.15 sec Start 40: tst_mxml_io 40/45 Test #40: tst_mxml_io ...................... Passed 21.03 sec Start 41: tst_guitarpro 41/45 Test #41: tst_guitarpro .................... Passed 5.42 sec Start 42: tst_scripting 42/45 Test #42: tst_scripting .................... Passed 1.08 sec Start 43: tst_ove_bdat 43/45 Test #43: tst_ove_bdat ..................... Passed 1.51 sec Start 44: tst_ove_ove3 44/45 Test #44: tst_ove_ove3 ..................... Passed 1.21 sec Start 45: tst_ove_structure 45/45 Test #45: tst_ove_structure ................ Passed 1.19 sec 100% tests passed, 0 tests failed out of 45 Total Test time (real) = 118.80 secI reran RELEASE tests, and got the same exact errors. So that means are not some random unpredictable exceptions.
In reply to I reran RELEASE tests, and by ericfontainejazz
Tests are not supposed to be run in release mode currently. Some code is instrumented only in debug mode. For example, the first failed test is tst_beam, and the beam class only write some tags when compiled in debug mode: https://github.com/musescore/MuseScore/blob/d349682845f93b5e082cd726fe5…
In reply to Tests are not supposed to be by [DELETED] 5
aha, that would explain it.
I'd like to be able to run tests in RELEASE and at least have it report no errors if nothing is wrong. Can I maybe add some if statement to exclude those tests that depend on specific Debug instrumentation if compiling as RELEASE. Something like: