| <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"> |
| <HTML |
| ><HEAD |
| ><TITLE |
| >A POSIX conforming test framework</TITLE |
| ><META |
| NAME="GENERATOR" |
| CONTENT="Modular DocBook HTML Stylesheet Version 1.76b+ |
| "><LINK |
| REL="HOME" |
| TITLE="DejaGnu" |
| HREF="book1.html"><LINK |
| REL="UP" |
| TITLE="Overview" |
| HREF="c50.html"><LINK |
| REL="PREVIOUS" |
| TITLE="Design Goals" |
| HREF="x107.html"><LINK |
| REL="NEXT" |
| TITLE="Getting DejaGnu up and running" |
| HREF="c203.html"></HEAD |
| ><BODY |
| CLASS="SECT1" |
| BGCOLOR="#FFFFFF" |
| TEXT="#000000" |
| LINK="#0000FF" |
| VLINK="#840084" |
| ALINK="#0000FF" |
| ><DIV |
| CLASS="NAVHEADER" |
| ><TABLE |
| SUMMARY="Header navigation table" |
| WIDTH="100%" |
| BORDER="0" |
| CELLPADDING="0" |
| CELLSPACING="0" |
| ><TR |
| ><TH |
| COLSPAN="3" |
| ALIGN="center" |
| >DejaGnu: The GNU Testing Framework</TH |
| ></TR |
| ><TR |
| ><TD |
| WIDTH="10%" |
| ALIGN="left" |
| VALIGN="bottom" |
| ><A |
| HREF="x107.html" |
| ACCESSKEY="P" |
| ><<< Previous</A |
| ></TD |
| ><TD |
| WIDTH="80%" |
| ALIGN="center" |
| VALIGN="bottom" |
| >Overview</TD |
| ><TD |
| WIDTH="10%" |
| ALIGN="right" |
| VALIGN="bottom" |
| ><A |
| HREF="c203.html" |
| ACCESSKEY="N" |
| >Next >>></A |
| ></TD |
| ></TR |
| ></TABLE |
| ><HR |
| ALIGN="LEFT" |
| WIDTH="100%"></DIV |
| ><DIV |
| CLASS="SECT1" |
| ><H1 |
| CLASS="SECT1" |
| ><A |
| NAME="POSIX" |
| ></A |
| >A POSIX conforming test framework</H1 |
| ><P |
| >DejaGnu conforms to the POSIX 1003.3 standard for test |
| frameworks. Rob Savoye was a member of that committee.</P |
| ><P |
| >The POSIX standard 1003.3 defines what a testing framework needs to |
| provide, in order to permit the creation of POSIX conformance test |
| suites. This standard is primarily oriented to running POSIX conformance |
| tests, but its requirements also support testing of features not related |
| to POSIX conformance. POSIX 1003.3 does not specify a particular testing |
| framework, but at this time there is only one other POSIX conforming test |
| framework: TET. TET was created by Unisoft for a consortium comprised of |
| X/Open, Unix International and the Open Software Foundation.</P |
| ><P |
| >The POSIX documentation refers to <I |
| CLASS="FIRSTTERM" |
| >assertions</I |
| >. |
| An assertion is a description of behavior. For example, if a standard |
| says ``The sun shall shine'', a corresponding assertion might be ``The |
| sun is shining.'' A test based on this assertion would pass or fail |
| depending on whether it is day or night. It is important to note |
| that the standard being tested is never 1003.3; the standard being tested |
| is some other standard, for which the assertions were written.</P |
| ><P |
| >As there is no testsuite to test testing frameworks for POSIX |
| 1003.3 conformance, verifying conformance to this standard is done by |
| repeatedly reading the standard and experimenting. One of the main |
| things 1003.3 does specify is the set of allowed output messages and |
| their definitions. Four messages are supported for a required feature of |
| POSIX conforming systems and a fifth for a conditional feature. DejaGnu |
| supports the use of all five output messages. In this sense a testsuite |
| that uses exactly these messages can be considered POSIX conforming. |
| These definitions specify the output of a test |
| case:</P |
| ><P |
| ></P |
| ><DIV |
| CLASS="VARIABLELIST" |
| ><DL |
| ><DT |
| >PASS</DT |
| ><DD |
| ><P |
| >A test has succeeded. That is, it demonstrated that |
| the assertion is true.</P |
| ></DD |
| ><DT |
| >XFAIL</DT |
| ><DD |
| ><P |
| >POSIX 1003.3 does not incorporate the notion of |
| expected failures, so <I |
| CLASS="EMPHASIS" |
| >PASS</I |
| >, instead of |
| <I |
| CLASS="EMPHASIS" |
| >XPASS</I |
| >, must also be returned for test cases |
| which were expected to fail and did not. This means that |
| <I |
| CLASS="EMPHASIS" |
| >PASS</I |
| > is in some sense more ambiguous than if |
| <I |
| CLASS="EMPHASIS" |
| >XPASS</I |
| > is also used.</P |
| ></DD |
| ><DT |
| >FAIL</DT |
| ><DD |
| ><P |
| >A test has produced the bug it was intended to |
| capture. That is, it has demonstrated that the assertion is false. |
| The <I |
| CLASS="EMPHASIS" |
| >FAIL</I |
| > message is based on the test case only. |
| Other messages are used to indicate a failure of the framework. As |
| with <I |
| CLASS="EMPHASIS" |
| >PASS</I |
| >, POSIX tests must return |
| <I |
| CLASS="EMPHASIS" |
| >FAIL</I |
| > rather than <I |
| CLASS="EMPHASIS" |
| >XFAIL</I |
| > even |
| if a failure was expected.</P |
| ></DD |
| ><DT |
| >UNRESOLVED</DT |
| ><DD |
| ><P |
| >A test produced indeterminate results. Usually, this |
| means the test executed in an unexpected fashion; this outcome |
| requires that a human being go over results, to determine if the test |
| should have passed or failed. This message is also used for any test |
| that requires human intervention because it is beyond the abilities |
| of the testing framework. Any unresolved test should resolved to |
| <I |
| CLASS="EMPHASIS" |
| >PASS</I |
| > or <I |
| CLASS="EMPHASIS" |
| >FAIL</I |
| > before a test |
| run can be considered finished.</P |
| ><P |
| >Note that for POSIX, each assertion must produce a test result |
| code. If the test isn't actually run, it must produce |
| <I |
| CLASS="EMPHASIS" |
| >UNRESOLVED</I |
| > rather than just leaving that test |
| out of the output. This means that you have to be careful when |
| writing tests to not carelessly use Tcl commands like |
| <I |
| CLASS="EMPHASIS" |
| >return</I |
| >---if you alter the flow of control of the |
| Tcl code you must insure that every test still produces some result |
| code.</P |
| ><P |
| >Here are some of the ways a test may wind up |
| <I |
| CLASS="EMPHASIS" |
| >UNRESOLVED</I |
| >:</P |
| ></DD |
| ></DL |
| ></DIV |
| ><P |
| ></P |
| ><UL |
| ><LI |
| STYLE="list-style-type: disc" |
| ><P |
| >A test's execution is |
| interrupted.</P |
| ></LI |
| ><LI |
| STYLE="list-style-type: disc" |
| ><P |
| >A test does not produce a clear |
| result. This is usually because there was an |
| <I |
| CLASS="EMPHASIS" |
| >ERROR</I |
| > from DejaGnu while processing |
| the test, or because there were three or more |
| <I |
| CLASS="EMPHASIS" |
| >WARNING</I |
| > messages. Any |
| <I |
| CLASS="EMPHASIS" |
| >WARNING</I |
| > or <I |
| CLASS="EMPHASIS" |
| >ERROR</I |
| > |
| messages can invalidate the output of the test. This |
| usually requires a human being to examine the output to |
| determine what really happened---and to improve the test |
| case.</P |
| ></LI |
| ><LI |
| STYLE="list-style-type: disc" |
| ><P |
| >A test depends on a previous test, which |
| fails.</P |
| ></LI |
| ><LI |
| STYLE="list-style-type: disc" |
| ><P |
| >The test was set up |
| incorrectly.</P |
| ></LI |
| ></UL |
| ><P |
| ></P |
| ><DIV |
| CLASS="VARIABLELIST" |
| ><DL |
| ><DT |
| >UNTESTED</DT |
| ><DD |
| ><P |
| >A test was not run. This is a place-holder, used |
| when there is no real test case yet.</P |
| ></DD |
| ></DL |
| ></DIV |
| ><P |
| >The only remaining output message left is intended to test |
| features that are specified by the applicable POSIX standard as |
| conditional:</P |
| ><P |
| ></P |
| ><DIV |
| CLASS="VARIABLELIST" |
| ><DL |
| ><DT |
| >UNSUPPORTED</DT |
| ><DD |
| ><P |
| >There is no support for the tested case. This may |
| mean that a conditional feature of an operating system, or of a |
| compiler, is not implemented. DejaGnu also uses this message when |
| a testing environment (often a ``bare board'' target) lacks basic |
| support for compiling or running the test case. For example, a |
| test for the system subroutine <I |
| CLASS="EMPHASIS" |
| >gethostname</I |
| > |
| would never work on a target board running only a boot |
| monitor.</P |
| ></DD |
| ></DL |
| ></DIV |
| ><P |
| >DejaGnu uses the same output procedures to produce these messages |
| for all testsuites and these procedures are already known to conform |
| to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3, |
| you must avoid the <I |
| CLASS="EMPHASIS" |
| >setup</I |
| >xfail} procedure as |
| described in the <I |
| CLASS="EMPHASIS" |
| >PASS</I |
| > section above and you must |
| be careful to return <I |
| CLASS="EMPHASIS" |
| >UNRESOLVED</I |
| > where appropriate, |
| as described in the <I |
| CLASS="EMPHASIS" |
| >UNRESOLVED</I |
| > section |
| above.</P |
| ></DIV |
| ><DIV |
| CLASS="NAVFOOTER" |
| ><HR |
| ALIGN="LEFT" |
| WIDTH="100%"><TABLE |
| SUMMARY="Footer navigation table" |
| WIDTH="100%" |
| BORDER="0" |
| CELLPADDING="0" |
| CELLSPACING="0" |
| ><TR |
| ><TD |
| WIDTH="33%" |
| ALIGN="left" |
| VALIGN="top" |
| ><A |
| HREF="x107.html" |
| ACCESSKEY="P" |
| ><<< Previous</A |
| ></TD |
| ><TD |
| WIDTH="34%" |
| ALIGN="center" |
| VALIGN="top" |
| ><A |
| HREF="book1.html" |
| ACCESSKEY="H" |
| >Home</A |
| ></TD |
| ><TD |
| WIDTH="33%" |
| ALIGN="right" |
| VALIGN="top" |
| ><A |
| HREF="c203.html" |
| ACCESSKEY="N" |
| >Next >>></A |
| ></TD |
| ></TR |
| ><TR |
| ><TD |
| WIDTH="33%" |
| ALIGN="left" |
| VALIGN="top" |
| >Design Goals</TD |
| ><TD |
| WIDTH="34%" |
| ALIGN="center" |
| VALIGN="top" |
| ><A |
| HREF="c50.html" |
| ACCESSKEY="U" |
| >Up</A |
| ></TD |
| ><TD |
| WIDTH="33%" |
| ALIGN="right" |
| VALIGN="top" |
| >Getting DejaGnu up and running</TD |
| ></TR |
| ></TABLE |
| ></DIV |
| ></BODY |
| ></HTML |
| > |
