| |
| Understanding the runtest script in DejaGnu |
| |
| Philip A. Wilsey |
| Clifton Labs, Inc |
| philip.wilsey@cliftonlabs.com |
| |
| September 26, 2001 |
| |
| |
| |
| DejaGnu is a tcl program to help run test scripts for batch and |
| interactive programs. If you have not yet read the DejaGnu |
| documentation don't, read this document first and play with this |
| example. If you have, you have my sympathies. Hopefully this document |
| and example will provide a high level overview of what goes on in |
| runtest. You will still have to read the DejaGnu documentation. It |
| contains many details of the low level functions and operation that I do |
| not address in this document. |
| |
| FIRST: I AM NOT A DEJAGNU OR TCL EXPERT. I have simply spent the past |
| week studying and trying to understand the execution behavior of the |
| runtest script. As part of my studies I built the accompanying example |
| that you may (or may not) find helpful in understanding just what is |
| going on with the runtest script. Fortunately tcl is a pretty simply |
| language and a quick scan of Ousterhout's text on tcl has been more than |
| sufficient for me to conduct these experiments. Again, this document |
| only records my observations. |
| |
| SECOND: I am not interested in or concerned with testing |
| cross-compilation systems or anything exotic like remote execution. I |
| was/am simply interested in using a test harness in a general purpose |
| conformance suite. Consequently I have not examined all the switches |
| and variable related to -host_board, -target, or remote execution. I |
| was simply trying to gain an overall understanding of what and when the |
| various tcl/expect files and procedures are executed. |
| |
| THIRD: There are really two parts to my studies, specifically: (i) the |
| study of runtest, and (ii) the use of the auto- tools to realize a |
| "make check" target that will invoke the runtest script for you. As a |
| result, this document is organized into three parts: |
| |
| Part I: An overview of runtest |
| Part II: Building a "make check" target with automake |
| Part III: Running the example |
| |
| For small projects, you may not be interested in the autoconf/automake |
| tools and can safely skip Part II of this document. |
| |
| FOURTH: This example is setup only for recording the files and |
| procedures that runtest uses. Technically it will compile a simple c++ |
| hello world program, however the test system does not verify it's |
| operation. If you run make check, it will simply dump out a bunch of |
| DEBUG statements and exit. Please don't expect anything more. |
| |
| FINALLY: For many years now we have been managing our regression suite |
| using hand rolled perl scripts that require continual maintenance. We |
| turned to DejaGnu in hopes to reduce some of our efforts in this |
| direction. Unfortunately, I found understanding DejaGnu to be very |
| painful and at several times during my study I was ready to chuck it |
| all. Now that I have a better understanding of the system, I believe |
| that in the long run I will be much happier that I stuck with it. |
| Hopefully you will think so as well. |
| |
| NOTE: I have only used/studied runtest on a Debian Linux box. I do not |
| know how well this knowledge will generalize or be useful to you. |
| Please use at your own risk. |
| |
| |
| |
| PART I: AN OVERVIEW OF RUNTEST |
| ------------------------------ |
| |
| I.1 The essence of Runtest |
| -------------------------- |
| |
| runtest is designed to be run from a test subdirectory where all tests |
| are stored in subdirectories with a naming convention of TOOL.TESTNAME. |
| Subdirectories not conforming to the TOOL.TESTNAME naming convention are |
| not examined. runtest examines recursively all the subdirectories under |
| each TOOL.TESTNAME subdirectories for expect scripts to be executed (it |
| assumes that everything with the ".exp" suffix is an expect script). It |
| will then attempt to execute every expect script it finds. For example, |
| let's take a look at the contents of the helloworld.* subdirectories in |
| the testsuite subdirectory of this example: |
| |
| peabody>ls -R testsuite/helloworld.test* |
| ~/test/dejagnu/helloDemo |
| testsuite/helloworld.test1: |
| total 8 |
| 4 helloworld.test1-1/ 4 test1.exp |
| |
| testsuite/helloworld.test1/helloworld.test1-1: |
| total 4 |
| 4 test1-1.exp |
| |
| testsuite/helloworld.test2: |
| total 8 |
| 4 test2.exp 4 test3.exp |
| |
| testsuite/helloworld.test3: |
| total 12 |
| 4 config/ 4 lib/ 4 non_compliant_dir_name/ |
| |
| testsuite/helloworld.test3/config: |
| total 4 |
| 4 config.exp |
| |
| testsuite/helloworld.test3/lib: |
| total 4 |
| 4 lib.exp |
| |
| testsuite/helloworld.test3/non_compliant_dir_name: |
| total 4 |
| 4 non_compliant.exp |
| peabody> |
| |
| Once everything is setup, issuing the command "runtest" from the |
| testsuite subdirectory on my machine causes all of the expect scripts to |
| be executed in this order: |
| |
| Running ./helloworld.test1/helloworld.test1-1/test1-1.exp ... |
| Running ./helloworld.test1/test1.exp ... |
| Running ./helloworld.test2/test2.exp ... |
| Running ./helloworld.test2/test3.exp ... |
| Running ./helloworld.test3/non_compliant_dir_name/non_compliant.exp ... |
| |
| Looks like a depth first search; however, I can find nothing in the |
| documentation that indicates that a specific order will be followed. So |
| you should not count on it. Also notice that the TOOL.TESTNAME naming |
| convention is only enforced at the root directory from where the runtest |
| command is issued -- beneath the first level subdirectories all |
| subdirectories except those named "lib" and "config" are examined for |
| expect scripts. |
| |
| I.2 The nuts and bolts of Runtest |
| --------------------------------- |
| |
| A few more points. runtest is setup to look for scripts to load for |
| configuration information and runtime procedures. In many cases if |
| runtest cannot find a script it will continue. Four important tcl |
| procedures that you should define (in testsuite/lib/TOOL.exp) are: |
| |
| TOOL_init |
| TOOL_finish |
| TOOL_exit |
| TOOL_version |
| |
| TOOL_init is called prior to running each expect script it locates under |
| the testsuite subdirectory. runtest invokes TOOL_finish after executing |
| each expect script. TOOL_exit is invoked after TOOL_finish for the last |
| expect script executes. Finally TOOL_version is invoked to allow the |
| testsute to report the version of the tool that was just tested. |
| |
| If you want to add tool specific arguments on the on the runtest command |
| line you can also define: |
| |
| TOOL_option_proc |
| TOOL_option_help |
| |
| TOOL_option_proc is invoked as runtest parses the command line |
| arguments. TOOL_option_help is invoked when runtest is invoked with the |
| "-help" command line argument (this is actually pretty cool). |
| |
| In general runtest will take the following steps (the expect scripts |
| named in these steps are named relative to the directory where the |
| runtest command is issued): |
| |
| 1. It will look for and load the file ~/.dejagnurc for command line |
| options. I have not experimented with this and cannot comment on |
| what can and cannot be placed in this file. |
| 2. It will try to load load the file ./site.exp (if you use automake, |
| this file will be automagically created for you). Technically this |
| file is organized into two parts, one that is set when the .configure |
| script is executed; the second part can be edited and changed by the |
| user. |
| 3. It will try to load the file ./lib/TOOL.exp (this is probably a good |
| place to locate definitions for the TOOL_init/TOOL_exit/TOOL_version |
| procedures. |
| 4. It will try to load some configuration files. Specifically |
| 4a. It will try to load a base-config.exp file. On my system it |
| searches, in order, the following subdirectories: |
| ./config |
| ./../config |
| ./../../config |
| ./../../../config |
| 4b. It will try to load a system specific configuration file. I believe |
| you can/should use this to setup different configuration information |
| for various operating systems that the test suite is to run in. On |
| my system it searches, in order, for (the search is terminated after |
| the first success): |
| ./config/unix.exp |
| ./config/gnu.exp |
| ./config/default.exp |
| ./config/unknown.exp |
| ./../config/unix.exp |
| ./../config/gnu.exp |
| ./../config/default.exp |
| ./../config/unknown.exp |
| ./../../config/unix.exp |
| ./../../config/gnu.exp |
| ./../../config/default.exp |
| ./../../config/unknown.exp |
| ./../../../config/unix.exp |
| ./../../../config/gnu.exp |
| ./../../../config/default.exp |
| ./../../../config/unknown.exp |
| 5. Now it will recursively search the ./TOOL.* subdirectories and for |
| each expect script it locates it will: |
| 5a. Invoke the procedure TOOL_init giving the relative pathname where |
| the expect script is found as an argument. |
| 5b. Run the expect script. |
| 5c. Invoke the proecture TOOL_finish with no arguments. |
| 6. After running all of the located expect script it will invoke the |
| TOOL_exit procedure. |
| 7. Next the TOOL_version procedure is invoked. |
| 8. Finally, runtest exits. |
| |
| That's it. Sadly it took me almost a week to learn this.... |
| |
| |
| I.3 Some notes I made during my week of study |
| --------------------------------------------- |
| |
| 1. site.exp is loaded twice. Why? I have no clue. |
| |
| 2. As far as I can tell the "testsuite" name is completely artificial. |
| runtest does not appear to depend on it. The autoconf/automake |
| family also do not seem to depend on it. |
| |
| 3. Searching for files to load: There is a method to this madness. |
| However, it's not very regular: for example why does 4a not look into |
| the testsuite subdirectory and 4b does. |
| |
| 3a. Look into srcdir (where the runtest command was issued) for |
| lib/tool.exp; I guess the principle concept is to put tool specific |
| and platform independent expect scripts here. |
| |
| 3b. Next look for for platform scripts in --srcdir/config (on my linux |
| system it looks for one of unix.exp, gnu.exp, and default.exp (in |
| this order and stops with the first one it finds). |
| |
| My confusion on the search patterns of runtest is magnified by the |
| fact that I am trying to issue the runtest command from outside the |
| testsuite subdirectory....don't do this. |
| |
| 4. Executing expect scripts. |
| |
| 4a. If an expect script is named on the command line it (and only it) is |
| run. otherwise.... |
| |
| 4b. runtest looks in the testsuite directory and does a glob on |
| TOOL.*/*.exp (ok, it's a recursive glob TOOL.*/.../*.exp) and |
| executes all expect script it sees (ok, it appears to ignore all |
| subdirectories named "lib" or "config"). Before execution of each |
| script, runtest invokes TOOL_init with the relative path (relative |
| to where the directory where the runtest command was issued) name of |
| that script as it's only argument. After execution of each script |
| runtest invokes TOOL_finish with no arguments. |
| |
| 5. load_lib: load_lib is a tcl script in the DejaGnu system that you can |
| use to load tcl/expect files. It has an odd search path. In |
| particular, on my machine, the search path is: |
| |
| 5a. the dejagnu install directory (/usr/share/dejagnu) and its |
| accompanying library directory (/usr/share/dejagnu/lib) |
| 5b. in a dejagnu/lib directory one above the current directory (where |
| runtest was run). that is in ../dejagnu/lib |
| 5c. in the --srcdir lib |
| 5d. in the directory where runtest was issued |
| 5e and this is the funny one, in a dejagnu/lib directory two directories |
| above (../../dejagnu/lib). |
| |
| 6. hmmmm how does DEJAGNULIBS alter the library search path?? I did not |
| investigate this. |
| |
| 7. COOL: I can name an expect script in the --ignore command line option |
| to prune it from the set of scripts that are executed. |
| |
| |
| I.4 Some gotchas |
| ---------------- |
| |
| 1. As far as I can tell the "testsuite" name is completely artificial |
| and you can store your tests in whatever subdirectory name you like. |
| |
| 2. Don't try to issue the runtest command from outside the testsuite |
| subdirectory using the "--srcdir target" command line argument. |
| Unless you really know what's going on, it will cause you grief. For |
| some reason the runtest will look for some things in the $target |
| location and for others it will look in the directory where the |
| runtest command was issued. |
| |
| 3. The search paths used to look for things varies all over the places. |
| The system has tremendous configurability for building and |
| controlling the expect scripts. However the flexibility for |
| controlling what and where things are loaded by runtest is really |
| quite limited. For example, site.exp must be located in the |
| directory where the runtest command is issued. I can discover no way |
| to alter its location. |
| |
| |
| |
| PART II: BUILDING a "make check" TARGET WITH AUTOMAKE |
| ----------------------------------------------------- |
| |
| NOTE: before I began this study I did not know anything about |
| autoconf/automake and friends (I still don't). |
| |
| 1. If you understand these tools it's really quite simple. In the |
| directory containing the testsuite you must have a Makefile.am to |
| build the "make check" target. You only need to define the dejagnu |
| option for AUTOMAKE_OPTIONS. If you want you can also add runtest |
| command line arguments to the RUNTESTDEFAULTFLAGS variable. |
| |
| The only problem I had with this was realizing that the Makefile.am |
| to locate these definitions in is the one located *in* the testsuite |
| subdirectory. DO NOT make these definitions in the root Makefile.am |
| file (unless you intend to place you TOOL.* subdirectories there). |
| |
| 2. I cannot figure out a good way to add alternate check targets (e.g., |
| make check-test1) to run alternate tests on the command line. Since |
| I really don't know automake very well, this could very likely be |
| very easy to do, but my ignorance prevents me from discovering how. |
| Presently I have simply defined the alternate target by hand (see |
| make check-demo in the testsuite/Makefile.am file). |
| |
| |
| PART III: RUNNING THE EXAMPLE |
| ----------------------------- |
| |
| The example contains a bunch of expect/tcl files, some of them are |
| unused (actually I don't believe I ever used anything outside of |
| standard tcl in any of my scripts). I created this example to explore |
| what was going on so you will see, for example, unix.exp in the lib, |
| testsuite/lib, and testsuite/config subdirectories. I did this to learn |
| where things were loaded from. Consequently you should *not* use my |
| setup to build a testing framework from. This system should only be |
| used to discover the runtime behaviors of runtest. |
| |
| To build this example, you will need a system configured with a c++ |
| compiler and the autoconf and automake tools installed. With these |
| tools in place, the build procedure is simply: |
| |
| autoconf |
| aclocal |
| automake |
| ./configure |
| make check |
| |
| Pretty much that's it (you may or may not need the aclocal step). After |
| the ./configure, the makefile in the testsuite subdirectory has two |
| relevant targets, namely: |
| |
| make check |
| make check-demo |
| |
| Good Luck. |