clang-tests/gdb-1472-testsuite/src/dejagnu/doc/overview.sgml - llvm-archive - Git at Google

 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [

 <!-- Begin Document Specific Declarations -->

 <?Fm: Validation Off>

 <!ENTITY version "1.4.4">
 <!ENTITY dj "DejaGnu">

 <!ENTITY dejagnu-copyright "
 	<YEAR>2002</YEAR>
 	<HOLDER>Free Software Foundation, Inc.</HOLDER>">

 <!ENTITY dejagnu-code-copyright "
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 This file documents the GNU Testing Framework ``DejaGnu''

 Copyright (C) 92 - 2001, 2002 Free Software
 Foundation, Inc.

 This text may be freely distributed under the terms of the GNU
 General Public License.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 ">

 <!ENTITY dejagnu-copyright "
 Copyright 92 - 2001, 2002 Free Software
 Foundation, Inc.

 Permission is granted to make and distribute verbatim copies of
 this manual provided the copyright notice and this permission notice
 are preserved on all copies.

 Permission is granted to copy and distribute modified versions of this
 manual under the conditions for verbatim copying, provided also that
 the entire resulting derived work is distributed under the terms of a
 permission notice identical to this one.

 Permission is granted to copy and distribute translations of this manual
 into another language, under the above conditions for modified versions.
 ">

 <!-- The reference material -->
 <!entity ref SYSTEM "ref.sgml">

 <!-- The user manual -->
 <!entity user SYSTEM "user.sgml">

 <!-- End Document Specific Declarations -->

 ]>

 <book>
   <bookinfo>
     <title>&dj;</title>
     <subtitle>The GNU Testing Framework</subtitle>
     <date>2002 Sept 04</date>
     <edition> &version</edition>
     <releaseinfo> New release</releaseinfo>
     <authorgroup>
       <author>
         <firstname>Rob Savoye</firstname>
         <affiliation>
           <orgname>Free Software Foundation</orgname></affiliation>
         <!-- <authorblurb>
           <title>Rob Savoye</title>
           <para>
             His home page is at <ulink>
             URL="http://www.welcomehome.org/rob.html">this
             location</ulink>
             </para>
         </authorblurb>
         -->
       </author>
     </authorgroup>
     <address>
       <email>rob@welcomehome.org</email>
     </address>
     <!-- &cygnus-street-address; -->
     <copyright>&dejagnu-copyright;</copyright>
     <!-- <legalnotice>
       <para> -->
         <!-- [FIXME: must put legal notice here] -->
       <!-- </para> -->
       <!-- &cygnus-legal-notice; -->
     <!-- </legalnotice> -->
     <revhistory>
       <revision>
         <revnumber>0.6.2</revnumber>
         <date>2002-7-16</date>
         <authorinitials>rob@welcomehome.org</authorinitials>
         <revremark>Add new tutorial as a new chapter.</revremark>
       </revision>
       <revision>
         <revnumber>0.6.1</revnumber>
         <date>2001-2-16</date>
         <authorinitials>rob@welcomehome.org</authorinitials>
         <revremark>Add info on the new dejagnu.h file.</revremark>
       </revision>
       <revision>
         <revnumber>0.6</revnumber>
         <date>2001-2-16</date>
         <authorinitials>rob@welcomehome.org</authorinitials>
         <revremark>Updated for new release.</revremark>
       </revision>
       <revision>
         <revnumber>0.5</revnumber>
         <date>2000-1-24</date>
         <authorinitials>rob@welcomehome.org</authorinitials>
         <revremark>Initial version after conversion to DocBook.</revremark>
       </revision>
     </revhistory>

   </bookinfo>

   <toc></toc>

   <preface id=preface>
     <title>Abstract</title>

     <para>This document describes the functionality of DejaGnu, the
     testing framework of the GNU project. DejaGnu is written in
     <productname>Expect</productname>, which uses
     <productname>Tcl</productname> as a command
     language. <productname>Expect</productname> acts as a very
     programmable shell.  As with other Unix command shells, you can
     run any program, but once the program is started, your test script
     has programmable control over its input and output.  This does not
     just apply to the programs under test; <command>expect</command>
     can also run any auxiliary program, such as
     <command>diff</command> or <command>sh</command>, with full
     control over its input and output.</para>

     <para>DejaGnu itself is merely a framework for the creation of
     testsuites. Testsuites are distributed with each
     application.</para>

   </preface>

   <chapter id=overview xreflabel=Overview>
     <title>Overview</title>

     <sect1 id=whatis xreflabel="What is &dj; ?">
       <title>What is &dj; ?</title>

     <para><productname>DejaGnu</productname> is a framework for
 	testing other programs.  Its purpose is to provide a single
 	front end for all tests. Think of it as a custom library of
 	Tcl procedures crafted to support writing a test harness. A
 	<emphasis>Test Harness</emphasis> is the testing
 	infrastructure that is created to support a specific program
 	or tool. Each program can have multiple testsuites, all
 	supported by a single test harness. DejaGnu is written in
 	<productname>Expect</productname>, which in turn uses
 	<productname>Tcl</productname> -- Tool command
 	language. There is more information on Tcl at the <ulink
 	URL="http://www.scriptics.com">Scriptics</ulink> web site and the
 	Expect web site is at <ulink
 	URL="http://expect.nist.gov">NIST</ulink>.</para>

    <para>Julia Menapace first coined the term ``DejaGnu'' to describe
         an earlier testing framework at Cygnus Support she had written
         for <command>GDB</command>. When we replaced it with the
         Expect-based framework, it was like DejaGnu all over again.
         More importantly, it was also named after my daughter, <ulink
         URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink>
         (now 13 years old as of September 2003), who was a toddler
         during DejaGnu's beginnings.</para>

     <para>DejaGnu offers several advantages for testing:</para>

     <itemizedlist mark="bullet" spacing="compact">

       <listitem><para>The flexibility and consistency of the DejaGnu
 	framework make it easy to write tests for any program, with
 	either batch oriented, or interactive programs.</para>
       </listitem>

       <listitem><para>DejaGnu provides a layer of abstraction which
 	allows you to write tests that are portable to any host or
 	target where a program must be tested. For instance, a test
 	for <command>GDB</command> can run from any supported host
 	system on any supported target system. DejaGnu runs tests on
 	many single board computers, whose operating software ranges
 	from a simple boot monitor to a real-time OS.</para>
 	</listitem>

       <listitem><para>All tests have the same output format. This
 	makes it easy to integrate testing into other software
 	development processes. DejaGnu's output is designed to be
 	parsed by other filtering script and it is also human
 	readable.</para>
       </listitem>

       <listitem><para>Using Tcl and Expect, it's easy to create wrappers
       for existing testsuites. By incorporating existing tests under
       DejaGnu, it's easier to have a single set of report analyse
       programs..</para>

       </listitem>
     </itemizedlist>

     <para>Running tests requires two things: the testing framework and
     the testsuites themselves. Tests are usually written in
     <productname>Expect</productname> using Tcl, but you can also use
     a Tcl script to run a testsuite that is not based on
     <productname>Expect</productname>. <productname>Expect</productname>
     script filenames conventionally use <emphasis>.exp</emphasis> as a
     suffix; for example, the main implementation of the DejaGnu test
     driver is in the file
     <productname>runtest.exp</productname>.)</para>

   </sect1>

   <sect1 id=new xreflabel="Release Notes">
     <title>What's New In This Release</title>

     <para>This release has a number of substantial changes over version
     1.3. The most visible change is that the version of Expect and Tcl
     included in the release are up-to-date with the current stable net
     releases. The biggest change is years of modifications to the
     target configuration system, used for cross testing. While this
     greatly improved cross testing, is has made that subsystem very
     complicated. The goal is to have this entirely rewritten using
     <productname>iTcl</productname> by the next release. Other changes
     are:</para>

     <itemizedlist>
       <listitem><para>More built-in support for building target binaries
         with the correct linker flags. Currently this only works with
         <productname>GCC</productname> as the cross compiler,
 	preferably with a target supported by
 	<xref linkend=libgloss>.</para></listitem>

       <listitem><para>Lots of little bug fixes from years of heavy
         use at Cygnus Solutions.</para></listitem>

       <listitem><para>DejaGnu now uses
       <productname>Automake</productname> for Makefile
       configuration.</para></listitem>

       <listitem><para>Updated documentation, now in SGML
 	(using the <ulink
 	URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free
 	GNU DocBook tools</ulink>) format.</para></listitem>

       <listitem><para>Windows support. There is beta level support for
       Windows that is still a work in progress. This requires the
       <ulink URL="http://www.cygwin.com/">Cygwin</ulink> POSIX
       subsystem for Windows.</para></listitem>

     </itemizedlist>

     <sect2 id=cygwin xreflabel="Windows Support">
       <title>Windows Support</title>

       <para>To use DejaGnu on Windows, you need to first install the
 	<ulink URL="http://www.cygwin.com/">Cygwin</ulink>
 	release. This works as of the B20.1 release. Cygwin is a POSIX
 	system for Windows. This covers both utility programs and a library
 	that adds POSIX system calls to Windows. Among them is pseudo tty
 	support for Windows that emulates the POSIX pty standard. The
 	latest Cygwin is always available from <ulink
 	URL="http://www.cygwin.com/">this location</ulink>. This
 	works well enough to run <emphasis>"make check"</emphasis> of
 	the GNU development tree on Windows after a native build. But the
 	nature of ptys on Windows is still evolving. Your mileage may
 	vary.</para>

     </sect2>

   </sect1>

   <sect1 id=designgoals xreflabel="Design Goals">
     <title>Design Goals</title>

     <para>DejaGnu grew out of the internal needs of Cygnus Solutions,
     the company formerly known as Cygnus Support. Cygnus maintained
     and enhanced a variety of free programs in many different
     environments and we needed a testing tool that:</para>

     <itemizedlist mark="bullet">
       <listitem><para>was useful to developers while fixing
       bugs;</para></listitem>
       <listitem><para>automated running many tests during a software
       release process;</para></listitem>
       <listitem><para>was portable among a variety of host
       computers;</para></listitem>
       <listitem><para>supported cross-development
       testing;</para></listitem>
       <listitem><para>permitted testing interactive programs, like
       <command>GDB</command>; and </para></listitem>
       <listitem><para>permitted testing batch oriented programs, like
       <command>GCC</command>.</para></listitem>
     </itemizedlist>

     <para>Some of the requirements proved challenging.  For example,
     interactive programs do not lend themselves very well to automated testing.
     But all the requirements are important: for instance, it is imperative to
     make sure that <command>GDB</command> works as well when cross-debugging
     as it does in a native configuration. </para>

     <para>Probably the greatest challenge was testing in a
     cross-development environment.  Most cross-development
     environments are customized by each developer.  Even when buying
     packaged boards from vendors there are many differences.  The
     communication interfaces vary from a serial line to Ethernet.
     DejaGnu was designed with a modular communication setup, so that
     each kind of communication can be added as required and supported
     thereafter.  Once a communication procedure is coded, any test can
     use it.  Currently DejaGnu can use <command>rsh</command>,
     <command>rlogin</command>, <command>telnet</command>,
     <command>tip</command>, <command>kermit</command> and
     <command>mondfe</command> for remote communications.</para>

     </sect1>

     <sect1 id=posix xreflabel="A POSIX Conforming Test Framework">
       <title>A POSIX conforming test framework</title>

       <para>DejaGnu conforms to the POSIX 1003.3 standard for test
       frameworks. Rob Savoye was a member of that committee.</para>

       <para>The POSIX standard 1003.3 defines what a testing framework needs to
       provide, in order to permit the creation of POSIX conformance test
       suites. This standard is primarily oriented to running POSIX conformance
       tests, but its requirements also support testing of features not related
       to POSIX conformance.  POSIX 1003.3 does not specify a particular testing
       framework, but at this time there is only one other POSIX conforming test
       framework: TET. TET was created by Unisoft for a consortium comprised of
       X/Open, Unix International and the Open Software Foundation.</para>

       <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
       An assertion is a description of behavior.  For example, if a standard
       says ``The sun shall shine'', a corresponding assertion might be ``The
       sun is shining.''  A test based on this assertion would pass or fail
       depending on whether it is day or night.  It is important to note
       that the standard being tested is never 1003.3; the standard being tested
       is some other standard, for which the assertions were written.</para>

       <para>As there is no testsuite to test testing frameworks for POSIX
       1003.3 conformance, verifying conformance to this standard is done by
       repeatedly reading the standard and experimenting.  One of the main
       things 1003.3 does specify is the set of allowed output messages and
       their definitions.  Four messages are supported for a required feature of
       POSIX conforming systems and a fifth for a conditional feature. DejaGnu
       supports the use of all five output messages.  In this sense a testsuite
       that uses exactly these messages can be considered POSIX conforming.
       These definitions specify the output of a test
       case:</para>

       <variablelist>
 	<varlistentry>
 	  <term>PASS</term>
 	  <listitem><para>A test has succeeded.  That is, it demonstrated that
 	  the assertion is true.</para></listitem>
 	</varlistentry>

         <varlistentry>
 	  <term>XFAIL</term>
 	  <listitem><para>POSIX 1003.3 does not incorporate the notion of
 	  expected failures, so <emphasis>PASS</emphasis>, instead of
 	  <emphasis>XPASS</emphasis>, must also be returned for test cases
 	  which were expected to fail and did not.  This means that
 	  <emphasis>PASS</emphasis> is in some sense more ambiguous than if
 	  <emphasis>XPASS</emphasis> is also used.</para></listitem>
 	</varlistentry>

         <varlistentry>
 	  <term>FAIL</term>
 	  <listitem><para>A test has produced the bug it was intended to
 	  capture.  That is, it has demonstrated that the assertion is false.
 	  The <emphasis>FAIL</emphasis> message is based on the test case only.
 	  Other messages are used to indicate a failure of the framework. As
 	  with <emphasis>PASS</emphasis>, POSIX tests must return
 	  <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
 	  if a failure was expected.</para></listitem>
 	</varlistentry>

         <varlistentry>
 	  <term>UNRESOLVED</term>
 	  <listitem><para>A test produced indeterminate results.  Usually, this
 	  means the test executed in an unexpected fashion; this outcome
 	  requires that a human being go over results, to determine if the test
 	  should have passed or failed.  This message is also used for any test
 	  that requires human intervention because it is beyond the abilities
 	  of the testing framework.  Any unresolved test should resolved to
 	  <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
 	  run can be considered finished.</para>

 	  <para>Note that for POSIX, each assertion must produce a test result
 	  code.  If the test isn't actually run, it must produce
 	  <emphasis>UNRESOLVED</emphasis> rather than just leaving that test
 	  out of the output.  This means that you have to be careful when
 	  writing tests to not carelessly use Tcl commands like
 	  <emphasis>return</emphasis>---if you alter the flow of control of the
 	  Tcl code you must insure that every test still produces some result
 	  code.</para>

 	  <para>Here are some of the ways a test may wind up
 	  <emphasis>UNRESOLVED</emphasis>:</para></listitem>

 	</varlistentry>
 	</variablelist>

           <itemizedlist mark=bullet>
 	    <listitem><para>A test's execution is
 	    interrupted.</para></listitem>

 	    <listitem><para>A test does not produce a clear
 	    result. This is usually because there was an
 	    <emphasis>ERROR</emphasis> from DejaGnu while processing
 	    the test, or because there were three or more
 	    <emphasis>WARNING</emphasis> messages. Any
 	    <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
 	    messages can invalidate the output of the test.  This
 	    usually requires a human being to examine the output to
 	    determine what really happened---and to improve the test
 	    case.</para></listitem>

 	    <listitem><para>A test depends on a previous test, which
 	    fails.</para></listitem>

 	    <listitem><para>The test was set up
 	    incorrectly.</para></listitem>
 	  </itemizedlist>

 	<variablelist>
 	  <varlistentry>
 	    <term>UNTESTED</term>
 	    <listitem><para>A test was not run.  This is a place-holder, used
 	    when there is no real test case yet.</para></listitem>
 	  </varlistentry>
 	</variablelist>

 	  <para>The only remaining output message left is intended to test
 	  features that are specified by the applicable POSIX standard as
 	  conditional:</para>

 	<variablelist>
 	  <varlistentry>
   	    <term>UNSUPPORTED</term>
 	    <listitem><para>There is no support for the tested case.  This may
   	    mean that a conditional feature of an operating system, or of a
   	    compiler, is not implemented.  DejaGnu also uses this message when
   	    a testing environment (often a ``bare board'' target) lacks basic
   	    support for compiling or running the test case.  For example, a
   	    test for the system subroutine <emphasis>gethostname</emphasis>
   	    would never work on a target board running only a boot
   	    monitor.</para></listitem>
 	  </varlistentry>
 	</variablelist>

         <para>DejaGnu uses the same output procedures to produce these messages
 	for all testsuites and these procedures are already known to conform
 	to POSIX 1003.3.  For a DejaGnu testsuite to conform to POSIX 1003.3,
 	you must avoid the <emphasis>setup</emphasis>xfail} procedure as
 	described in the <emphasis>PASS</emphasis> section above and you must
 	be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
 	as described in the <emphasis>UNRESOLVED</emphasis> section
 	above.</para>
      </sect1>

   </chapter>

   <!-- include the user manual -->
   &user;

   <!-- include the reference manual -->
   &ref;

 </book>

 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
 sgml-omittag:t
 sgml-shorttag:t
 sgml-namecase-general:t
 sgml-general-insert-case:lower
 sgml-minimize-attributes:nil
 sgml-always-quote-attributes:t
 sgml-indent-step:1
 sgml-indent-data:nil
 sgml-parent-document:nil
 sgml-exposed-tags:nil
 sgml-local-catalogs:nil
 sgml-local-ecat-files:nil
 End:
 -->
	<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [

	<!-- Begin Document Specific Declarations -->

	<?Fm: Validation Off>

	<!ENTITY version "1.4.4">
	<!ENTITY dj "DejaGnu">

	<!ENTITY dejagnu-copyright "
	<YEAR>2002</YEAR>
	<HOLDER>Free Software Foundation, Inc.</HOLDER>">

	<!ENTITY dejagnu-code-copyright "
	%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
	This file documents the GNU Testing Framework ``DejaGnu''

	Copyright (C) 92 - 2001, 2002 Free Software
	Foundation, Inc.

	This text may be freely distributed under the terms of the GNU
	General Public License.
	%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
	">

	<!ENTITY dejagnu-copyright "
	Copyright 92 - 2001, 2002 Free Software
	Foundation, Inc.

	Permission is granted to make and distribute verbatim copies of
	this manual provided the copyright notice and this permission notice
	are preserved on all copies.

	Permission is granted to copy and distribute modified versions of this
	manual under the conditions for verbatim copying, provided also that
	the entire resulting derived work is distributed under the terms of a
	permission notice identical to this one.

	Permission is granted to copy and distribute translations of this manual
	into another language, under the above conditions for modified versions.
	">

	<!-- The reference material -->
	<!entity ref SYSTEM "ref.sgml">

	<!-- The user manual -->
	<!entity user SYSTEM "user.sgml">

	<!-- End Document Specific Declarations -->

	]>

	<book>
	<bookinfo>
	<title>&dj;</title>
	<subtitle>The GNU Testing Framework</subtitle>
	<date>2002 Sept 04</date>
	<edition> &version</edition>
	<releaseinfo> New release</releaseinfo>
	<authorgroup>
	<author>
	<firstname>Rob Savoye</firstname>
	<affiliation>
	<orgname>Free Software Foundation</orgname></affiliation>
	<!-- <authorblurb>
	<title>Rob Savoye</title>
	<para>
	His home page is at <ulink>
	URL="http://www.welcomehome.org/rob.html">this
	location</ulink>
	</para>
	</authorblurb>
	-->
	</author>
	</authorgroup>
	<address>
	<email>rob@welcomehome.org</email>
	</address>
	<!-- &cygnus-street-address; -->
	<copyright>&dejagnu-copyright;</copyright>
	<!-- <legalnotice>
	<para> -->
	<!-- [FIXME: must put legal notice here] -->
	<!-- </para> -->
	<!-- &cygnus-legal-notice; -->
	<!-- </legalnotice> -->
	<revhistory>
	<revision>
	<revnumber>0.6.2</revnumber>
	<date>2002-7-16</date>
	<authorinitials>rob@welcomehome.org</authorinitials>
	<revremark>Add new tutorial as a new chapter.</revremark>
	</revision>
	<revision>
	<revnumber>0.6.1</revnumber>
	<date>2001-2-16</date>
	<authorinitials>rob@welcomehome.org</authorinitials>
	<revremark>Add info on the new dejagnu.h file.</revremark>
	</revision>
	<revision>
	<revnumber>0.6</revnumber>
	<date>2001-2-16</date>
	<authorinitials>rob@welcomehome.org</authorinitials>
	<revremark>Updated for new release.</revremark>
	</revision>
	<revision>
	<revnumber>0.5</revnumber>
	<date>2000-1-24</date>
	<authorinitials>rob@welcomehome.org</authorinitials>
	<revremark>Initial version after conversion to DocBook.</revremark>
	</revision>
	</revhistory>

	</bookinfo>

	<toc></toc>

	<preface id=preface>
	<title>Abstract</title>

	<para>This document describes the functionality of DejaGnu, the
	testing framework of the GNU project. DejaGnu is written in
	<productname>Expect</productname>, which uses
	<productname>Tcl</productname> as a command
	language. <productname>Expect</productname> acts as a very
	programmable shell. As with other Unix command shells, you can
	run any program, but once the program is started, your test script
	has programmable control over its input and output. This does not
	just apply to the programs under test; <command>expect</command>
	can also run any auxiliary program, such as
	<command>diff</command> or <command>sh</command>, with full
	control over its input and output.</para>

	<para>DejaGnu itself is merely a framework for the creation of
	testsuites. Testsuites are distributed with each
	application.</para>

	</preface>

	<chapter id=overview xreflabel=Overview>
	<title>Overview</title>

	<sect1 id=whatis xreflabel="What is &dj; ?">
	<title>What is &dj; ?</title>

	<para><productname>DejaGnu</productname> is a framework for
	testing other programs. Its purpose is to provide a single
	front end for all tests. Think of it as a custom library of
	Tcl procedures crafted to support writing a test harness. A
	<emphasis>Test Harness</emphasis> is the testing
	infrastructure that is created to support a specific program
	or tool. Each program can have multiple testsuites, all
	supported by a single test harness. DejaGnu is written in
	<productname>Expect</productname>, which in turn uses
	<productname>Tcl</productname> -- Tool command
	language. There is more information on Tcl at the <ulink
	URL="http://www.scriptics.com">Scriptics</ulink> web site and the
	Expect web site is at <ulink
	URL="http://expect.nist.gov">NIST</ulink>.</para>

	<para>Julia Menapace first coined the term ``DejaGnu'' to describe
	an earlier testing framework at Cygnus Support she had written
	for <command>GDB</command>. When we replaced it with the
	Expect-based framework, it was like DejaGnu all over again.
	More importantly, it was also named after my daughter, <ulink
	URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink>
	(now 13 years old as of September 2003), who was a toddler
	during DejaGnu's beginnings.</para>

	<para>DejaGnu offers several advantages for testing:</para>

	<itemizedlist mark="bullet" spacing="compact">

	<listitem><para>The flexibility and consistency of the DejaGnu
	framework make it easy to write tests for any program, with
	either batch oriented, or interactive programs.</para>
	</listitem>

	<listitem><para>DejaGnu provides a layer of abstraction which
	allows you to write tests that are portable to any host or
	target where a program must be tested. For instance, a test
	for <command>GDB</command> can run from any supported host
	system on any supported target system. DejaGnu runs tests on
	many single board computers, whose operating software ranges
	from a simple boot monitor to a real-time OS.</para>
	</listitem>

	<listitem><para>All tests have the same output format. This
	makes it easy to integrate testing into other software
	development processes. DejaGnu's output is designed to be
	parsed by other filtering script and it is also human
	readable.</para>
	</listitem>

	<listitem><para>Using Tcl and Expect, it's easy to create wrappers
	for existing testsuites. By incorporating existing tests under
	DejaGnu, it's easier to have a single set of report analyse
	programs..</para>

	</listitem>
	</itemizedlist>

	<para>Running tests requires two things: the testing framework and
	the testsuites themselves. Tests are usually written in
	<productname>Expect</productname> using Tcl, but you can also use
	a Tcl script to run a testsuite that is not based on
	<productname>Expect</productname>. <productname>Expect</productname>
	script filenames conventionally use <emphasis>.exp</emphasis> as a
	suffix; for example, the main implementation of the DejaGnu test
	driver is in the file
	<productname>runtest.exp</productname>.)</para>

	</sect1>

	<sect1 id=new xreflabel="Release Notes">
	<title>What's New In This Release</title>

	<para>This release has a number of substantial changes over version
	1.3. The most visible change is that the version of Expect and Tcl
	included in the release are up-to-date with the current stable net
	releases. The biggest change is years of modifications to the
	target configuration system, used for cross testing. While this
	greatly improved cross testing, is has made that subsystem very
	complicated. The goal is to have this entirely rewritten using
	<productname>iTcl</productname> by the next release. Other changes
	are:</para>

	<itemizedlist>
	<listitem><para>More built-in support for building target binaries
	with the correct linker flags. Currently this only works with
	<productname>GCC</productname> as the cross compiler,
	preferably with a target supported by
	<xref linkend=libgloss>.</para></listitem>

	<listitem><para>Lots of little bug fixes from years of heavy
	use at Cygnus Solutions.</para></listitem>

	<listitem><para>DejaGnu now uses
	<productname>Automake</productname> for Makefile
	configuration.</para></listitem>

	<listitem><para>Updated documentation, now in SGML
	(using the <ulink
	URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free
	GNU DocBook tools</ulink>) format.</para></listitem>

	<listitem><para>Windows support. There is beta level support for
	Windows that is still a work in progress. This requires the
	<ulink URL="http://www.cygwin.com/">Cygwin</ulink> POSIX
	subsystem for Windows.</para></listitem>

	</itemizedlist>

	<sect2 id=cygwin xreflabel="Windows Support">
	<title>Windows Support</title>

	<para>To use DejaGnu on Windows, you need to first install the
	<ulink URL="http://www.cygwin.com/">Cygwin</ulink>
	release. This works as of the B20.1 release. Cygwin is a POSIX
	system for Windows. This covers both utility programs and a library
	that adds POSIX system calls to Windows. Among them is pseudo tty
	support for Windows that emulates the POSIX pty standard. The
	latest Cygwin is always available from <ulink
	URL="http://www.cygwin.com/">this location</ulink>. This
	works well enough to run <emphasis>"make check"</emphasis> of
	the GNU development tree on Windows after a native build. But the
	nature of ptys on Windows is still evolving. Your mileage may
	vary.</para>

	</sect2>

	</sect1>

	<sect1 id=designgoals xreflabel="Design Goals">
	<title>Design Goals</title>

	<para>DejaGnu grew out of the internal needs of Cygnus Solutions,
	the company formerly known as Cygnus Support. Cygnus maintained
	and enhanced a variety of free programs in many different
	environments and we needed a testing tool that:</para>

	<itemizedlist mark="bullet">
	<listitem><para>was useful to developers while fixing
	bugs;</para></listitem>
	<listitem><para>automated running many tests during a software
	release process;</para></listitem>
	<listitem><para>was portable among a variety of host
	computers;</para></listitem>
	<listitem><para>supported cross-development
	testing;</para></listitem>
	<listitem><para>permitted testing interactive programs, like
	<command>GDB</command>; and </para></listitem>
	<listitem><para>permitted testing batch oriented programs, like
	<command>GCC</command>.</para></listitem>
	</itemizedlist>

	<para>Some of the requirements proved challenging. For example,
	interactive programs do not lend themselves very well to automated testing.
	But all the requirements are important: for instance, it is imperative to
	make sure that <command>GDB</command> works as well when cross-debugging
	as it does in a native configuration. </para>

	<para>Probably the greatest challenge was testing in a
	cross-development environment. Most cross-development
	environments are customized by each developer. Even when buying
	packaged boards from vendors there are many differences. The
	communication interfaces vary from a serial line to Ethernet.
	DejaGnu was designed with a modular communication setup, so that
	each kind of communication can be added as required and supported
	thereafter. Once a communication procedure is coded, any test can
	use it. Currently DejaGnu can use <command>rsh</command>,
	<command>rlogin</command>, <command>telnet</command>,
	<command>tip</command>, <command>kermit</command> and
	<command>mondfe</command> for remote communications.</para>

	</sect1>

	<sect1 id=posix xreflabel="A POSIX Conforming Test Framework">
	<title>A POSIX conforming test framework</title>

	<para>DejaGnu conforms to the POSIX 1003.3 standard for test
	frameworks. Rob Savoye was a member of that committee.</para>

	<para>The POSIX standard 1003.3 defines what a testing framework needs to
	provide, in order to permit the creation of POSIX conformance test
	suites. This standard is primarily oriented to running POSIX conformance
	tests, but its requirements also support testing of features not related
	to POSIX conformance. POSIX 1003.3 does not specify a particular testing
	framework, but at this time there is only one other POSIX conforming test
	framework: TET. TET was created by Unisoft for a consortium comprised of
	X/Open, Unix International and the Open Software Foundation.</para>

	<para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
	An assertion is a description of behavior. For example, if a standard
	says ``The sun shall shine'', a corresponding assertion might be ``The
	sun is shining.'' A test based on this assertion would pass or fail
	depending on whether it is day or night. It is important to note
	that the standard being tested is never 1003.3; the standard being tested
	is some other standard, for which the assertions were written.</para>

	<para>As there is no testsuite to test testing frameworks for POSIX
	1003.3 conformance, verifying conformance to this standard is done by
	repeatedly reading the standard and experimenting. One of the main
	things 1003.3 does specify is the set of allowed output messages and
	their definitions. Four messages are supported for a required feature of
	POSIX conforming systems and a fifth for a conditional feature. DejaGnu
	supports the use of all five output messages. In this sense a testsuite
	that uses exactly these messages can be considered POSIX conforming.
	These definitions specify the output of a test
	case:</para>

	<variablelist>
	<varlistentry>
	<term>PASS</term>
	<listitem><para>A test has succeeded. That is, it demonstrated that
	the assertion is true.</para></listitem>
	</varlistentry>

	<varlistentry>
	<term>XFAIL</term>
	<listitem><para>POSIX 1003.3 does not incorporate the notion of
	expected failures, so <emphasis>PASS</emphasis>, instead of
	<emphasis>XPASS</emphasis>, must also be returned for test cases
	which were expected to fail and did not. This means that
	<emphasis>PASS</emphasis> is in some sense more ambiguous than if
	<emphasis>XPASS</emphasis> is also used.</para></listitem>
	</varlistentry>

	<varlistentry>
	<term>FAIL</term>
	<listitem><para>A test has produced the bug it was intended to
	capture. That is, it has demonstrated that the assertion is false.
	The <emphasis>FAIL</emphasis> message is based on the test case only.
	Other messages are used to indicate a failure of the framework. As
	with <emphasis>PASS</emphasis>, POSIX tests must return
	<emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
	if a failure was expected.</para></listitem>
	</varlistentry>

	<varlistentry>
	<term>UNRESOLVED</term>
	<listitem><para>A test produced indeterminate results. Usually, this
	means the test executed in an unexpected fashion; this outcome
	requires that a human being go over results, to determine if the test
	should have passed or failed. This message is also used for any test
	that requires human intervention because it is beyond the abilities
	of the testing framework. Any unresolved test should resolved to
	<emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
	run can be considered finished.</para>

	<para>Note that for POSIX, each assertion must produce a test result
	code. If the test isn't actually run, it must produce
	<emphasis>UNRESOLVED</emphasis> rather than just leaving that test
	out of the output. This means that you have to be careful when
	writing tests to not carelessly use Tcl commands like
	<emphasis>return</emphasis>---if you alter the flow of control of the
	Tcl code you must insure that every test still produces some result
	code.</para>

	<para>Here are some of the ways a test may wind up
	<emphasis>UNRESOLVED</emphasis>:</para></listitem>

	</varlistentry>
	</variablelist>

	<itemizedlist mark=bullet>
	<listitem><para>A test's execution is
	interrupted.</para></listitem>

	<listitem><para>A test does not produce a clear
	result. This is usually because there was an
	<emphasis>ERROR</emphasis> from DejaGnu while processing
	the test, or because there were three or more
	<emphasis>WARNING</emphasis> messages. Any
	<emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
	messages can invalidate the output of the test. This
	usually requires a human being to examine the output to
	determine what really happened---and to improve the test
	case.</para></listitem>

	<listitem><para>A test depends on a previous test, which
	fails.</para></listitem>

	<listitem><para>The test was set up
	incorrectly.</para></listitem>
	</itemizedlist>

	<variablelist>
	<varlistentry>
	<term>UNTESTED</term>
	<listitem><para>A test was not run. This is a place-holder, used
	when there is no real test case yet.</para></listitem>
	</varlistentry>
	</variablelist>

	<para>The only remaining output message left is intended to test
	features that are specified by the applicable POSIX standard as
	conditional:</para>

	<variablelist>
	<varlistentry>
	<term>UNSUPPORTED</term>
	<listitem><para>There is no support for the tested case. This may
	mean that a conditional feature of an operating system, or of a
	compiler, is not implemented. DejaGnu also uses this message when
	a testing environment (often a ``bare board'' target) lacks basic
	support for compiling or running the test case. For example, a
	test for the system subroutine <emphasis>gethostname</emphasis>
	would never work on a target board running only a boot
	monitor.</para></listitem>
	</varlistentry>
	</variablelist>

	<para>DejaGnu uses the same output procedures to produce these messages
	for all testsuites and these procedures are already known to conform
	to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3,
	you must avoid the <emphasis>setup</emphasis>xfail} procedure as
	described in the <emphasis>PASS</emphasis> section above and you must
	be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
	as described in the <emphasis>UNRESOLVED</emphasis> section
	above.</para>
	</sect1>

	</chapter>

	<!-- include the user manual -->
	&user;

	<!-- include the reference manual -->
	&ref;

	</book>

	<!-- Keep this comment at the end of the file
	Local variables:
	mode: sgml
	sgml-omittag:t
	sgml-shorttag:t
	sgml-namecase-general:t
	sgml-general-insert-case:lower
	sgml-minimize-attributes:nil
	sgml-always-quote-attributes:t
	sgml-indent-step:1
	sgml-indent-data:nil
	sgml-parent-document:nil
	sgml-exposed-tags:nil
	sgml-local-catalogs:nil
	sgml-local-ecat-files:nil
	End:
	-->