| The Insight Testsuite |
| --------------------- |
| Keith Seitz (keiths@cygnus.com) |
| May 1, 2001 |
| |
| RUNNING THE TESTSUITE |
| |
| The Insight testsuite is run in much the same way that gdb's testsuites |
| are run. The one big difference is the environment variable GDB_DISPLAY, |
| which governs what display should be used for the tests. |
| |
| When GDB_DISPLAY is not set in the user's environment, the Insight testsuite |
| will attempt to run Xvfb, an X server with a virtual frame buffer. Using |
| Xvfb, the testsuite can run without interuppting the user. |
| |
| When Xvfb is not available, the testsuite will mark the Insight tests |
| "untested" and print out some appropriate warning to the testsuite log |
| file. |
| |
| If GDB_DISPLAY is set in the user's environment, the testsuite will attempt |
| to use this display for the tests. If this display is a desktop display, |
| it is very likely that any interaction between the user and his desktop |
| will interfere with the tests. Some tests warp the cursor, i.e., they |
| force the mouse to move on the screen. If you don't want this to happen to |
| you, put Xvfb in your path. |
| |
| On Cygwin systems, Xvfb is not supported. Only two choices are available in |
| this environment: run the testsuites using the desktop or do not run the |
| testsuite. To run the testsuite on Cygwin, just define the environment |
| variable GDB_DISPLAY to anything. |
| |
| The examples below summarize the usage of the environment variable GDB_DISPLAY |
| on unix/X-Windows hosts and Cygwin hosts. In all examples, assume that DISPLAY |
| set to the local workstation's main display (:0). |
| |
| To run the testsuite using Xvfb -- unix only (Xvfb must be in PATH): |
| $ make check |
| |
| To run the testsuite using a given display (either the desktop or a peviously |
| started Xvfb): |
| $ GDB_DISPLAY=$DISPLAY make check |
| |
| To run the testsuite on Cygwin: |
| $ GDB_DISPLAY=foo make check |
| |
| |
| TESTSUITE INFRASTRUCTURE |
| |
| The rest of this document deals with writing tests for Insight. This reading |
| is only noteworthy for developers and contributors. |
| |
| The Insight testsuite consists of two large portions of code: code which is |
| run in dejagnu and code which runs in Insight's built-in Tcl interpreter. Files |
| containing dejagnu code (those files ending in ".exp" in the testsuite directory) |
| are "glue code" between gdb's dejagnu testsuite and Insight's Tcl testsuite. |
| |
| Dejagnu testsuite files are considered "drivers" for any particular set of |
| tests, since they allow dejagnu to control Insight's Tcl testsuite. |
| |
| |
| Dejagnu Testsuite Infrastructure |
| |
| The dejagnu code is responsible for doing several things. Some of the more |
| important responsibilities include: |
| |
| o Initializing the display |
| o Determining if tests should be run |
| o Results accounting |
| o Compiling testcases in various languages |
| o Repoting results to gdb's testsuite |
| |
| There are various functions defined to facilitate the writing of tests. These |
| functions currently reside in gdb's gdb.exp (src/gdb/testsuite/lib/gdb.exp) and |
| include: |
| |
| Pulic functions: |
| proc gdbtk_initialize_display {} |
| |
| gdbtk_initialize_display should be the first function called from the |
| (dejagnu) test file. It initializes the DISPLAY variable on unix systems |
| and determines (for all host systems) whether or not the testsuite should |
| run. It returns 1 if the test should run. If tests should not run, it |
| marks the test as "untested" and leaves a suitable message about why |
| the test should not run. If gdbtk_initialize_display returns zero, a test |
| should simply exit. |
| |
| proc gdbtk_start {test} |
| |
| This function marks the start of a test and will execute Insight for |
| testing. The TEST parameter should be the file name of the Tcl test |
| file to load into Insight's Tcl interpreter. It returns a list of |
| test results suitable for passing to gdbtk_done or gdbtk_analyze_results. |
| See gdbtk_analyze_results for more information on the format of results. |
| |
| gdbtk_start is responsible for communicating target startup information |
| to Insight, so that Insight's testsuite may be run on any target supported |
| by gdb. It does this by setting several environment variables just before |
| it starts Insight. These environment variables are: |
| |
| OBJDIR |
| The object directory of the dejagnu testsuite (i.e., |
| objdir/gdb/testsuite). |
| SRCDIR |
| The dejagnu source directory in which the tests are located (i.e, |
| src/gdb/testsuite) |
| SUBDIR |
| The dejagnu testsuite subdirectory for the test (i.e., gdb.gdbtk) |
| DEFS |
| The location of the testsuite definitions file (i.e., |
| src/gdb/testsuite/gdb.gdbtk/defs) |
| |
| Note that DEFS is converted to abs$lute tcl-style paths. On unix, |
| this means that DEFS would point to, for example, |
| /home/keiths/insight/src/gdb/testsuite/gdb.gdbtk/defs. On Cygwin it |
| would point to C:/cygwin/home/keiths/insight/src/gdb/testsuite/gdb.gdbtk/defs. |
| This is because of a descrepency between Cygwin's posix paths and Tcl's |
| not-quite-posix paths. |
| |
| proc gdbtk_analyze_results {results} |
| This function translates the list of results in RESULTS into dejagnu |
| results, reporting the number of failures, errors, passes, and expected |
| failures and passes. It currently does not deal with "untested" and other |
| test statuses from dejagnu since Insight's tcl testsuite does not |
| issue such results. |
| |
| The format of the results expected by gdbtk_analyze_results is simple: |
| it is a list of {status name description msg}. "status" is the execution |
| status of one of the tcl tests run. This can be "PASS", "FAIL", "ERROR", |
| "XFAIL", or "XPASS". |
| |
| "name" is the name of the test, and it is reported in all testsuite |
| results in dejagnu. This speeds location of the failing test. This |
| "name" may also be given to Insight's testsuite, telling it to |
| only run this test. See "do_tests" in Tcl Testsuite Infrastructure |
| for more information. |
| |
| "description" is a textual description of the test given by "name". |
| |
| "msg" is currently not used. |
| |
| proc gdbtk_done {{results {}}} |
| gdbtk_done takes any RESULTS and passes it gdbtk_analyze_results for |
| outputting to the dejagnu part of the testsuite. It may be called |
| without an argument, in which case it will only terminate Xvfb if it |
| was started. Tests must call gdbtk_done _once_ at the end of their |
| test drivers. |
| |
| Private functions: |
| proc _gdbtk_export_target_info |
| This functin exports information about the target into the environment |
| so that Insight's testsuite may run programs on any supported gdb |
| target. This target information is passed into the Tcl testsuite |
| via an environment variable, TARGET_INFO, which is really a Tcl array, |
| i.e., the array is constructed in tcl and exported into the environment |
| with Tcl's "array get" command). |
| |
| There are four elements to the array: |
| TARGET_INFO(init) - (optional) A list of commands to execute in gdb |
| to initialize the session. This usually includes |
| setting baud rates and remote protocol options. |
| TARGET_INFO(target) - (required) The complete "target" command to connect |
| to the given target. |
| TARGET_INFO(load) - (optional) The complete "load" command to load an |
| executable into a target. |
| TARGET_INFO(run) - (required) The complete "run" command, sans arguments, |
| to start a process on the target. For remote targets, |
| this is usually just "continue". |
| |
| proc _gdbtk_xvfb_init |
| This procedure actually determines whether the an Insight test should |
| run and what DISPLAY it should use for that test. It is called by |
| gdbtk_initialize_display to do most of the dirty work. |
| |
| It has a simple heuristic: If GDB_DISPLAY is not set and Xvfb is available |
| (on unix), it starts Xvfb, using the current process id as the screen number. |
| If Xvfb is not available and GDB_DISPLAY was not set, it skips the tests. |
| |
| proc _gdbtk_xvfb_exit |
| _gdbtk_xvfb_exit will kill any previously started Xvfb. |
| |
| Private globals: |
| global _xvfb_spawn_id |
| This variable holds the spawn_id of any Xvfb process started |
| by the testsuite (or it is left undefined). |
| |
| global _using_windows |
| A global variable which indicates whether the testsuite is running |
| on cygwin. Unfortunately, as of writing, the global variable |
| tcl_platform(platform) is "unix" on Cygwin, so it is not possible |
| to rely on this for platform-dependent operations. |
| |
| Instead, this variable is set by gdbtk_initialize_display. The test |
| it uses to determine if Cygwin is being used: it looks for the program |
| cygpath in the PATH. Therefore, cygpath is REQUIRED to run the testsuite |
| on Cygwin. (gdbtk_start also uses cygpath to determine Windows |
| pathnames for Cygwin.) |
| |
| |
| Testsuite Driver Basics |
| |
| Given the above interfaces for connecting Insight's Tcl testsuite and |
| gdb's dejagnu testsuite, the basic testsuite driver file should look |
| (minimally) like this: |
| |
| File: mytest.exp |
| 1 load_lib "insight-support.exp" |
| 2 if {[gdbtk_initialize_display]} { |
| 3 # We found a display to use |
| 4 gdb_exit; # Make sure any previous gdb is gone |
| 5 set results [gdbtk_start mytest.test] |
| 6 |
| 7 # Done! |
| 8 gdbtk_done [split $results \n] |
| 9 } |
| |
| Line 1 loads the insight testsuite support library which contains definitions |
| for all the procedures used in dejagnu to initialize and run the Insight testsuite. |
| Line 2 calls gdbtk_initialize_display to ascertain whether there is a display |
| to use for the test. This could use an existing display (if GDB_DISPLAY is |
| set in the environment) or gdbk_initialize_display could startup an Xvfb |
| for use by the testsuite. |
| |
| Line 4 forces any previously executing gdb to terminate. |
| |
| Line 5 signals the start of the test run. "mytest.test" is the name of the |
| Tcl test file to execute in Insight's built-in Tcl interpreter. The output |
| of gdbtk_start_test is all of the results of the Tcl test from Insight, which |
| is subsequently passed to gdbk_analyze_results via gdbtk_done on Line 8. |
| |
| Note how nothing happens if gdbtk_initialize_display returns false. |
| |
| |
| Tcl Testsuite Infrastructure |
| |
| The heart of Insight's testsuite is its Tcl testsuite. It is these tests |
| which run directly in Insight's Tcl interpreter and allow test writers |
| access to Insight's internals. Tcl testsuite files have the filename suffix |
| ".test" to distinguish them from their driver files, which end in ".exp". |
| |
| The design of the Insight Tcl testsuite parallels Tcl's testsuite. It has |
| many powerful features, including the ability to run ANY test in a given |
| Tcl test file. See the description of utility routines below for more |
| information about running any set of tests from a file. |
| |
| The bulk of the code implementing the Tcl testsuite infrastructure in |
| Insight is contained in the testsuite definitions file, "defs", located |
| in src/gdb/testsuite/gdb.gdbtk. This file contains routines necessary |
| to write tests for Insight. |
| |
| Public functions: |
| proc gdbtk_read_defs {} |
| This function, located in Insight's core Tcl library, attempts to load |
| the testsuite definitions file. If it fails, it will either pop up |
| a dialog box with the error (if running interactively) or it will |
| print the error to stderr and exit (if running non-interactively). |
| |
| If successful, it will return true. |
| |
| proc gdbtk_test_file {filename} |
| This function is used to load the file given by FILENAME into |
| Insight. It will automatically append ".exe" to any FILENAME |
| on Cygwin-hosted systems. |
| |
| If successful, it will load the given file into Insight and |
| return the output of gdb's file command. It will call "error" |
| if it was succesful, therefore all calls to "gdbtk_test_file" |
| should be called using "catch". |
| |
| Test authors should not use "gdb_cmd {file FILENAME}" to load |
| files into gdb unless they are testing interface code between |
| gdb and Insight. |
| |
| proc gdbtk_test_run {{prog_args {}}} |
| gdbtk_test_run runs the previoiusly loaded executable, passing |
| the given arguments to the inferior. Like Insight's Run button, |
| it will do whatever is necessary to get the executable running, |
| including any target initialization (setting baud rate and remote |
| protocol options), downloading the executable to the target, and |
| finally starting execution. |
| |
| Test authors should NEVER use "gdb_cmd {run ARGUMENTS}" to run an |
| executable. Doing so will insure that your tests will only run on |
| a native debugger. |
| |
| It returns true if successful or false otherwise. It will report |
| the error in a dialog box (if running interactively) or it will |
| print the error to stderr. |
| |
| proc gdbtk_test {name description script answer} |
| This is Tcl testsuite equivalent of "expect". "name" is a canonical |
| name of the test, usually of the form "shortname-major.minor". This is |
| the name that is used when running selected tests from a given file. |
| If "name" starts with an asterisk (*), it designates that the test |
| is expected to fail. |
| |
| "description" is a short textual description of the test to help |
| humans understand what it does. |
| |
| "script" is the actual test script to run. The result of this script |
| will be compared against "answer" to determine if the test passed |
| or failed. |
| |
| It calls gdbtk_print_verbose to print out the results to the terminal |
| (if running interactively) or to the log file. |
| |
| proc gdbtk_test_done {} |
| gdbtk_test_done is called at the very end of all tcl tests. It is used |
| to exit Insight and return control back to the dejagnu driver which |
| started the tests. |
| |
| proc gdbtk_dotests {file args} |
| Obsolete. |
| |
| proc do_test {{file {}} {verbose {}} {tests {}}} |
| This procedure is used to invoke the Insight test(s) given in FILE |
| which match the regular expression(s) in TESTS. This is invoked |
| from Insight's console window to run tests interactively. |
| |
| VERBOSE sets the verbosity of the test run. When set to one, |
| the testsuite will report all results in human readable form. |
| When set greater than one, it will print out results as a list, |
| i.e., for passing to gdbtk_analyze_results. If zero, it will only |
| print errors and failures in human readable form. |
| |
| Public global variables: |
| objdir - The objdir from dejagnu. See gdbtk_start for more information. |
| srcdir - The srcdir from dejagnu. See gdbtk_start for more information. |
| test_ran - Indicates whether the last test ran or not. See example below. |
| |
| Private functions: |
| proc gdbtk_test_error {desc} |
| An internal function used to report a framework error in the testsuite. |
| "desc" is a description of the error. It calls gdbtk_test_done. |
| |
| proc gdbtk_print_verbose {status name description script code answer} |
| A helper procedure to gdbtk_test which prints out results to the terminal |
| or the logfile (or both or none). |
| |
| Private global variables: |
| _test - An array used by the testsuite internals. |
| |
| |
| Tcl Test Basics |
| |
| Armed with the basic interface described above, it is possible to test Insight's |
| GUI. Please do not write tests which attempt to imitate a user (moving the |
| mouse and clicking buttons), unless there is no other way to test the functionality. |
| |
| The basic test file (with one test) looks like this (nonsensical one): |
| File: mytest.test |
| 1 if {![gdbtk_read_defs]} { |
| 2 break |
| 3 } |
| 4 |
| 5 global objdir test_ran |
| 6 set program [file join $objdir mytest] |
| 7 if {[catch {gdbtk_test_file $program} t]} { |
| 8 gdbtk_test_error "loading \"$program\": $t" |
| 9 } |
| 10 if {![gdbtk_test_run]} { exit 1 } |
| 11 |
| 12 global foo |
| 13 set foo 1 |
| 14 |
| 15 # Test: mytest-1.1 |
| 16 # Desc: check if a source window was created |
| 17 gdbtk_test mytest-1.1 {source window created} { |
| 18 set window [ManagedWin::find SrcWin] |
| 19 llength $window |
| 20 set foo 13 |
| 21 } {1} |
| 22 |
| 23 if {$test_ran} { |
| 24 set foo 1 |
| 25 } |
| 26 |
| 27 # Done |
| 28 gdbtk_test_done |
| |
| Line 1 calls the Inisght function gdbtk_read_defs to read in the testsuite |
| definitions file. |
| |
| Line 6 then specifies the name of a file (mytest) in the object directory |
| which is loaded into gdb on Line 7. If loading the file into Insight |
| failed, gdbtk_test_error is called to publish the error (and terminate the |
| test run for this file). |
| |
| Line 10 runs the executable on the target, and exits if it was unable |
| to do so. |
| |
| Line 13 simply sets a global variable foo to illustrate the purpose |
| of the global "test_ran". Before the test "mytest-1.1" runs, foo is set to |
| one. In order to support running specific tests, the state of the debugger |
| cannot be altered INSIDE gdbtk_test scripts, since the contents of the |
| script may not be run if the user requested only a specific test to run. |
| |
| Line 20 in the middle of the test modifies the global foo. If subsequent |
| test relied on foo being one, we would have a state violation, since |
| mytest-1.1 may have (or may have not) run. |
| |
| Therefore, we can check if a test ran and reset foo by checking the |
| global "test_ran". If set, we know that the previous test (mytest-1.1) |
| was run, and that foo is now thirteen. We reset the result back to one. |
| |
| (Aside: Some tests do not follow this rule explicitly: they can assume |
| that all tests run sequentially. In these cases, running a specific |
| test in the file will probably fail, since the debugger is not brought |
| to a known state along the way.) |
| |
| Lines 17-21 contain the actual test. The test's name is "mytest-1.1". It |
| is this name that may be referred to when asking the testsuite to run |
| a specific test. The description of this test is "source window created", |
| indicating that mytest-1.1's purpose is to check whether a source window |
| was created. |
| |
| If gdbtk_test determines that this test is to run, it will execute the |
| next part, lines 18-20, and compare the output of that script (llength |
| $window) with "1". If the result is not "1", a failure is recorded. If |
| it is "1", a pass is recorded. |
| |
| Finally, the test file is done and exits on line 28. |