<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<HTML
><HEAD
><TITLE
>A POSIX conforming test framework</TITLE
><META
NAME="GENERATOR"
CONTENT="Modular DocBook HTML Stylesheet Version 1.76b+
"><LINK
REL="HOME"
TITLE="DejaGnu"
HREF="book1.html"><LINK
REL="UP"
TITLE="Overview"
HREF="c50.html"><LINK
REL="PREVIOUS"
TITLE="Design Goals"
HREF="x107.html"><LINK
REL="NEXT"
TITLE="Getting DejaGnu up and running"
HREF="c203.html"></HEAD
><BODY
CLASS="SECT1"
BGCOLOR="#FFFFFF"
TEXT="#000000"
LINK="#0000FF"
VLINK="#840084"
ALINK="#0000FF"
><DIV
CLASS="NAVHEADER"
><TABLE
SUMMARY="Header navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TH
COLSPAN="3"
ALIGN="center"
>DejaGnu: The GNU Testing Framework</TH
></TR
><TR
><TD
WIDTH="10%"
ALIGN="left"
VALIGN="bottom"
><A
HREF="x107.html"
ACCESSKEY="P"
>&#60;&#60;&#60; Previous</A
></TD
><TD
WIDTH="80%"
ALIGN="center"
VALIGN="bottom"
>Overview</TD
><TD
WIDTH="10%"
ALIGN="right"
VALIGN="bottom"
><A
HREF="c203.html"
ACCESSKEY="N"
>Next &#62;&#62;&#62;</A
></TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="100%"></DIV
><DIV
CLASS="SECT1"
><H1
CLASS="SECT1"
><A
NAME="POSIX"
></A
>A POSIX conforming test framework</H1
><P
>DejaGnu conforms to the POSIX 1003.3 standard for test
      frameworks. Rob Savoye was a member of that committee.</P
><P
>The POSIX standard 1003.3 defines what a testing framework needs to
      provide, in order to permit the creation of POSIX conformance test
      suites. This standard is primarily oriented to running POSIX conformance
      tests, but its requirements also support testing of features not related
      to POSIX conformance.  POSIX 1003.3 does not specify a particular testing
      framework, but at this time there is only one other POSIX conforming test
      framework: TET. TET was created by Unisoft for a consortium comprised of
      X/Open, Unix International and the Open Software Foundation.</P
><P
>The POSIX documentation refers to <I
CLASS="FIRSTTERM"
>assertions</I
>.
      An assertion is a description of behavior.  For example, if a standard
      says ``The sun shall shine'', a corresponding assertion might be ``The
      sun is shining.''  A test based on this assertion would pass or fail
      depending on whether it is day or night.  It is important to note
      that the standard being tested is never 1003.3; the standard being tested
      is some other standard, for which the assertions were written.</P
><P
>As there is no testsuite to test testing frameworks for POSIX
      1003.3 conformance, verifying conformance to this standard is done by
      repeatedly reading the standard and experimenting.  One of the main
      things 1003.3 does specify is the set of allowed output messages and
      their definitions.  Four messages are supported for a required feature of
      POSIX conforming systems and a fifth for a conditional feature. DejaGnu
      supports the use of all five output messages.  In this sense a testsuite
      that uses exactly these messages can be considered POSIX conforming.
      These definitions specify the output of a test
      case:</P
><P
></P
><DIV
CLASS="VARIABLELIST"
><DL
><DT
>PASS</DT
><DD
><P
>A test has succeeded.  That is, it demonstrated that
	  the assertion is true.</P
></DD
><DT
>XFAIL</DT
><DD
><P
>POSIX 1003.3 does not incorporate the notion of
	  expected failures, so <I
CLASS="EMPHASIS"
>PASS</I
>, instead of
	  <I
CLASS="EMPHASIS"
>XPASS</I
>, must also be returned for test cases
	  which were expected to fail and did not.  This means that
	  <I
CLASS="EMPHASIS"
>PASS</I
> is in some sense more ambiguous than if
	  <I
CLASS="EMPHASIS"
>XPASS</I
> is also used.</P
></DD
><DT
>FAIL</DT
><DD
><P
>A test has produced the bug it was intended to
	  capture.  That is, it has demonstrated that the assertion is false.
	  The <I
CLASS="EMPHASIS"
>FAIL</I
> message is based on the test case only.
	  Other messages are used to indicate a failure of the framework. As
	  with <I
CLASS="EMPHASIS"
>PASS</I
>, POSIX tests must return
	  <I
CLASS="EMPHASIS"
>FAIL</I
> rather than <I
CLASS="EMPHASIS"
>XFAIL</I
> even
	  if a failure was expected.</P
></DD
><DT
>UNRESOLVED</DT
><DD
><P
>A test produced indeterminate results.  Usually, this
	  means the test executed in an unexpected fashion; this outcome
	  requires that a human being go over results, to determine if the test
	  should have passed or failed.  This message is also used for any test
	  that requires human intervention because it is beyond the abilities
	  of the testing framework.  Any unresolved test should resolved to
	  <I
CLASS="EMPHASIS"
>PASS</I
> or <I
CLASS="EMPHASIS"
>FAIL</I
> before a test
	  run can be considered finished.</P
><P
>Note that for POSIX, each assertion must produce a test result
	  code.  If the test isn't actually run, it must produce
	  <I
CLASS="EMPHASIS"
>UNRESOLVED</I
> rather than just leaving that test
	  out of the output.  This means that you have to be careful when
	  writing tests to not carelessly use Tcl commands like
	  <I
CLASS="EMPHASIS"
>return</I
>---if you alter the flow of control of the
	  Tcl code you must insure that every test still produces some result
	  code.</P
><P
>Here are some of the ways a test may wind up
	  <I
CLASS="EMPHASIS"
>UNRESOLVED</I
>:</P
></DD
></DL
></DIV
><P
></P
><UL
><LI
STYLE="list-style-type: disc"
><P
>A test's execution is
	    interrupted.</P
></LI
><LI
STYLE="list-style-type: disc"
><P
>A test does not produce a clear
	    result. This is usually because there was an
	    <I
CLASS="EMPHASIS"
>ERROR</I
> from DejaGnu while processing
	    the test, or because there were three or more
	    <I
CLASS="EMPHASIS"
>WARNING</I
> messages. Any
	    <I
CLASS="EMPHASIS"
>WARNING</I
> or <I
CLASS="EMPHASIS"
>ERROR</I
>
	    messages can invalidate the output of the test.  This
	    usually requires a human being to examine the output to
	    determine what really happened---and to improve the test
	    case.</P
></LI
><LI
STYLE="list-style-type: disc"
><P
>A test depends on a previous test, which
	    fails.</P
></LI
><LI
STYLE="list-style-type: disc"
><P
>The test was set up
	    incorrectly.</P
></LI
></UL
><P
></P
><DIV
CLASS="VARIABLELIST"
><DL
><DT
>UNTESTED</DT
><DD
><P
>A test was not run.  This is a place-holder, used
	    when there is no real test case yet.</P
></DD
></DL
></DIV
><P
>The only remaining output message left is intended to test
	  features that are specified by the applicable POSIX standard as
	  conditional:</P
><P
></P
><DIV
CLASS="VARIABLELIST"
><DL
><DT
>UNSUPPORTED</DT
><DD
><P
>There is no support for the tested case.  This may
  	    mean that a conditional feature of an operating system, or of a
  	    compiler, is not implemented.  DejaGnu also uses this message when
  	    a testing environment (often a ``bare board'' target) lacks basic
  	    support for compiling or running the test case.  For example, a
  	    test for the system subroutine <I
CLASS="EMPHASIS"
>gethostname</I
>
  	    would never work on a target board running only a boot
  	    monitor.</P
></DD
></DL
></DIV
><P
>DejaGnu uses the same output procedures to produce these messages
	for all testsuites and these procedures are already known to conform
	to POSIX 1003.3.  For a DejaGnu testsuite to conform to POSIX 1003.3,
	you must avoid the <I
CLASS="EMPHASIS"
>setup</I
>xfail} procedure as
	described in the <I
CLASS="EMPHASIS"
>PASS</I
> section above and you must
	be careful to return <I
CLASS="EMPHASIS"
>UNRESOLVED</I
> where appropriate,
	as described in the <I
CLASS="EMPHASIS"
>UNRESOLVED</I
> section
	above.</P
></DIV
><DIV
CLASS="NAVFOOTER"
><HR
ALIGN="LEFT"
WIDTH="100%"><TABLE
SUMMARY="Footer navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
><A
HREF="x107.html"
ACCESSKEY="P"
>&#60;&#60;&#60; Previous</A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="book1.html"
ACCESSKEY="H"
>Home</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
><A
HREF="c203.html"
ACCESSKEY="N"
>Next &#62;&#62;&#62;</A
></TD
></TR
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
>Design Goals</TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="c50.html"
ACCESSKEY="U"
>Up</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
>Getting DejaGnu up and running</TD
></TR
></TABLE
></DIV
></BODY
></HTML
>