| Testing with QMTest |
| =================== |
| |
| You can use QMTest to test G++. (In the future, it may be possible to |
| test other parts of GCC with QMTest as well, but it is not possible |
| yet.) |
| |
| The use of QMTest to run the G++ tests has not been approved as an |
| officially supported testing procedure. Therefore, you must run the |
| tests using DejaGNU (with "make check-g++") before committing changes |
| that affect G++. |
| |
| QMTest emulates DejaGNU behavior very closely when running the tests. |
| |
| QMTest has two output modes: a DejaGNU emulation mode and a native |
| QMTest mode. |
| |
| In the DejaGNU mode, you should receive output that is almost exactly |
| the same as the DejaGNU output; in particular, you should see the same |
| number of passes, failures, etc. When using the DejaGNU-style output, |
| QMTest uses the "xfail" indications in the test cases to determine |
| which tests are expected to pass and which are expected to fail, and |
| presents that information in the same way as DejaGNU. |
| |
| In the QMTest mode, the number of passes and failures will be |
| different from that obtained when using DejaGNU. The reason is that a |
| single source file may contain multiple DejaGNU tests. In DejaGNU, |
| each line where a diagnostic is expected is considered a separate |
| test. Testing for successful compilation and testing for successful |
| execution of the generated program are considered separate tests. So, |
| a single source file "test.C" could contain, say, seven tests; some of |
| which might pass and some of which might fail. |
| |
| In the QMTest mode, each source file is considered a single test. If |
| any of the seven sub-tests fail, the entire test is considered to |
| fail. However, QMTest does present information about *why* the test |
| failed, so the same information is effectively available. |
| |
| In the QMTest mode, whether or not a test is expected to fail is |
| determined not by an indication in the test, but rather by comparing |
| the new results to the results of a previous run. Testing for whether |
| a change caused a regression is very simple: run the tests before |
| making the change, run them again after making the change, and let |
| QMTest compare the results. |
| |
| The mode chosen only affects the output from QMTest, not how it runs |
| the tests or how it stores the data. Therefore, if you choose to run |
| in the QMTest mode and later want to get the DejaGNU style output, or |
| vice versa, you can do that as described below. |
| |
| Setting Up |
| ========== |
| |
| You must download and install the following software: |
| |
| - Python 2.2 (or greater) |
| |
| See http://www.python.org. |
| |
| You may already have Python on your system; in particular, many |
| GNU/Linux systems ship with Python installed. |
| |
| Installation instructions are available on the web-site. |
| |
| - A current version of QMTest. No released version provides all of |
| the functionality required, so you must obtain QMTest from CVS. |
| |
| To do that, follow the instructions at: |
| |
| http://www.codesourcery.com/qmtest |
| |
| Installation instructions are available in the file called README |
| after you check out QMTest. |
| |
| - The "qmtc" and "qmtest_gcc" QMTest support packages. These are |
| available from the same CVS repository as QMTest. For example, to |
| check out "qmtc", do: |
| |
| cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \ |
| co qmtc |
| |
| You do not have to install these packages; you need only check them |
| out. |
| |
| Running the Tests |
| ================= |
| |
| First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and |
| qmtest_gcc support packages: |
| |
| export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc |
| |
| The, run "make qmtest-g++" in the gcc directory of your build tree. |
| |
| Here are some more advanced usage instructions: |
| |
| 1. To run a particular set of tests (rather than all of the tests), |
| use the make variable "QMTEST_GPP_TESTS". For example, |
| |
| make QMTEST_GPP_TESTS="g++.dg" qmtest-g++ |
| |
| will run only the tests in the g++.dg subdirectory, and: |
| |
| make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \ |
| g++.old-deja/g++.other/access2.C" |
| qmtest-g++ |
| |
| will run only the two tests indicated. |
| |
| 2. To run qmtest with particular flags, use the make variables |
| "QMTESTFLAGS" and "QMTESTRUNFLAGS". For example: |
| |
| make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++ |
| |
| will run qmtest like this: |
| |
| qmtest -v run -f full ... |
| |
| (The "-f full" mode will provide detailed information about each |
| test as it runs.) |
| |
| 3. To run the compiler with particular flags, use QMTESTRUNFLAGS to |
| set the QMTest context variable "CompilerTable.cplusplus_options", |
| like this: |
| |
| make \ |
| QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \ |
| qmtest-g++ |
| |
| The compiler will then use the "-funroll-loops" switch when |
| compiling. |
| |
| 4. If qmtest is not in your path, you can indicate the full path to |
| QMTest by using the make variable "QMTEST_PATH", like this: |
| |
| make QMTEST_PATH=/path/to/qmtest qmtest-g++ |
| |
| 5. To start the QMTest GUI, use: |
| |
| make qmtest-gui |
| |
| (Note that this will run the program called "mozilla" in your path. |
| If you want to use another browser, you must configure qmtest as |
| described in its manual.) |
| |
| Bear in mind that the QMTest GUI is insecure; malicious users with |
| access to your machine may be able to run commands as if they were |
| you. The QMTest GUI only binds to the loopback IP addresss, which |
| provides a measure of security, but not enough for use in untrusted |
| environments. |
| |
| 6. If you have a multiprocessor, you can run the tests in parallel by |
| passing the "-j" option to qmtest: |
| |
| make QMTESTRUNFLAGS="-j 4" qmtest-g++ |
| |
| will run tests in four threads. (It is also possible to run tests |
| across multiple machines; for more information see the QMTest |
| manual.) |
| |
| 7. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get |
| more detailed information, you can do: |
| |
| cd qmtestsuite |
| qmtest summarize g++.qmr g++.dg/abi/bitfield1.C |
| |
| to get more information about the commands that were run and the |
| output produced. |