testsuite/README - openmp - Git at Google

  ============================================================================
 |                   OpenMP Validation Suite  V 3.0                           |
 |              High Performance Computing Center, Stuttgart                  |
 |       High Performance Computing and Tools, University of Houston          |
 |                              Jan. 2012                                     |
  ============================================================================


 TABLE OF CONTENTS

 I    	INTRODUCTION
 I.1.	  Aims and general function
 I.2. 	  Files and directories

 II		USAGE
 II.1.	  First run with make
 II.2	  Where to search for results
 II.3.	  Using the runtest script

 III		Adding and modifying tests
 III.1.	  The template structure

 IV		Known Issues and Workaround

 V       Contact and Support

 ------------------------------------------------------------------------------

 I.  INTRODUCTION
 ==============================================================================

 I.1.  Aims and general function
 --------------------------------

 The OpenMP validation suite is designed to verify the correct implementation
 of OpenMP in compilers. It is capable of checking Fortran as well as c
 compiler.

 Testing the implementation is based on statistics. Each directive is tested
 by computing and verifying the result with the already known value. In most
 cases a wrong implementation can result in the right values. So the tests are
 run several times.

 Additionally, the validation suite creates so called crosstests for each
 directive. These are tests in which either the directive is missing or used
 with different arguments. If this so called crosstest fails, this indicates
 strongly that the previous test is capable of testing the directive.

 Lastly, an orphaned test is also run to determine if the directive being
 tested is able to correctly run when 'orphaned' from the main function.
 Essentially, the directive's code is placed into its own function which is
 called during execution of the main function and often inside a parallel
 region.


 I.2.  Files and directories
 ----------------------------


 d c                         directory containing the templates for the c tests
 d fortran                   directory containing the templates for the Fortran
                             tests
   Makefile                  Makefile containing options for compilation
   common_utility.f
   omp_my_sleep.h            thread save sleep function
   omp_testsuite.f           Fortran header file
   omp_testsuite.h           autogenerated c-header file
   ompts-c.conf              configuration file for the c tests about how often
                             the tests shall be executed or how large the loop size
                             is
   ompts_makeHeader.pl	    perl module for automatically generation of an up
                             to date header file
   ompts_parserFunctions.pm	perl module containing general functions for
 				            the parser.pl script
   ompts_parser.pl	        script for generating the source code out of the templates
   ompts_standaloneProc.c	framework for the c tests
   ompts_standaloneProc.f	framework for the Fortran tests
   README		            the README file you've already found ;-)
   LICENSE                   contains license information
   runtest.pl		        the frame program of the test suite
   testlist-f.txt	        test list containing the available tests for Fortran
   testlist-c.txt	        test list containing the available tests for c


 ------------------------------------------------------------------------------

 II.  USAGE
 ==============================================================================


 II.1. First run with make
 --------------------------


 You can do a first simple run of the testsuite only after one step of
 configuration:

 1) Modify the ompts.conf and ompts-c.conf file, change the number of threads
 and number of repetitions of each test.

 2) Modify the Makefile, uncommenting the CC/FC and CFLAGS/FFLAGS variables for
 the compiler of your choice.

 And now you can run the testsuite either for a C compiler with:

 >	make ctest

 or for a Fortran compiler with:

 >	make ftest


 II.2. Running custom tests
 --------------------------


 In order to run single tests or custom groups of tests, two make commmands
 are defined: make stest and make fstest. These two command reference the file
 customtest.txt when looking for a testlist to use. Simply edit customtest.txt
 to include the desired test or tests. If customtest.txt contains c tests,

 >      make stest

 or for fortran tests

 >      make fstest

 In order to change the number of threads used in the tests, simply edit the
 Makefile variables MINTHREADS and MAXTHREADS. By default, they are configured
 to use 2 threads. To change the number of times each test is run, for c tests
 edit the REPETITIONS variable in the file ompts-c.conf. The LOOPCOUNT and
 SLEEPTIME variables can also be changed here. For fortran tests, edit the file
 omp_testsuite.f to change both the LOOPCOUNT and the number of times each test
 is run.


 II.3. Understanding the results
 ---------------------------------


 When running the testsuite the results will be shown on the screen.

 If you need the results for further purpose you can use the results.txt, which
 is a simple list containing the results for each directive
 in a single line. Each line starts with the name of the directive. Then follows
 the result of the test given in the percentage of the passed tests. If 100% of
 the tests passed successfully, the second number gives the result of the
 corresponding crosstest. Crosstests will only be run if the normal test passes
 with 100% accuracy. If a crosstest was not run or a test does not exist, it is
 denotated by a "-".
 After the results of the normal tests, there follow a series of tests in the
 orphaned mode. If there were no orphaned tests available this is shown by a "-".

 If you run the testsuite with different numbers of threads (e.g. using the
 runtest.pl script) the results are shown in blocks of 4 columns for each number
 of threads.

 If a test fails you can find more detailed information in the ompts.log,
 bin/c/*.out and *.log files. While the ompts.log file contains all compiler
 error messages for all tests, the *.out and *.log files contain detailed inforamtion
 on the execution process of the single tests.
 In the *.out files there are listed all the results of the single executions of
 the tests. In the *.log files there are error messages of the tests itself.


 II.4. Cleaning Up
 -----------------


 Because many files are generated for each tested directive, it is often necessary
 to clean the main directory after a battery of tests. To clean all generated files
 in the main directory including the results and log files,

 >     make clean


 To clean only the logs and out files,

 >     make cleanlogs

 To clean only the results,

 >     make cleanresults


 II.4. Using the runtest script
 -------------------------------


 So for special purpose you can use the runtest.pl script, which allows a lot
 more options for the execution process than the execution with make.

 Using the runtest.pl script is rather easy. You can use the the test suite only
 after two steps of modifications:

 1.) Modify the Makefile to your wishes choosing your compiler and the necessary
     compiler flags.
 2.) If necessary edit one of the test lists (testlist-c.txt) and comment out the
     tests you do not want to run using # at the beginning of a line. Testlists for Fortran end
     with -f.txt while test lists for c with -c.txt.

 And now you can run the test suite either for Fortran using
 	#	>  ./runtest.pl -lang=fortran -d=fortran testlist-f.txt
 or for c
 	#	>  ./runtest.pl -lang=c -d=c testlist-c.txt

 With the --help option you can show the complete list of options and their
 explanations.

 The test results are summarized in cresults or fresults.txt while *.log keep
 details for individual tests. There is also a file (ompts.log) keeping
 compilation information. (see section II.2 )

 If you don't want to test the directives in orphaned mode you can use the
 -norphan option. You also can use the runtest.pl script either to compile all
 tests or run compiled tests e.g. for cross compilation on other platforms. For
 this there are the options -norun and -nocompile.

 Happy testing!


 ------------------------------------------------------------------------------

 III.	How to add new tests / The structure of test templates
 ==============================================================================

 III.1  The template structure
 ------------------------------

 The test suite is based on templates so that you only have one file for test,
 crosstest and the orphaned versions of them.

 	A) Description of the template structure

 The syntax of the templates is much like xml. So each test begins with
 '<ompts:test>' and ends with '</ompts:test>'.

 In between there are several other blocks holding information:

 - <ompts:testdescription> </ompts:testdescription> In between this tag you can
   give a description on what the test checks and how it works.

 - <ompts:ompversion> </ompts:version> This tag is used to specify the
   OpenMP-version which includes the tested directive.

 - <ompts:directive> </ompts:directive> Used to specify the directive how it is
   called in the programming language.

 - <ompts:dependences> </ompts:dependences> With this tag you can specify other
   omp directives which are necessary for the correct execution of your test.
 The directives have to be listed by their directive names as it is called in
 the programming language. Multiple directives are separated by ','.

 - <ompts:testcode> </ompts:testcode> In this tag stands the whole source code
 for the test / crosstest.  Each test has to be written as a function. The
 syntax of the functions differs between C and Fortran: In C it has to take a
 file pointer 'FILE * logFile' and return an int. If the test has been passed
 successful it has to return a value unequal to 0. The file pointer can be used
 to write information into a log file.  In Fortran the function takes no
 argument and the function name must not exceed XX characters.  The return value
 has to be specified using the '<testfunctionname>' tags. It has to be 1 if the
 test succeeded and 0 if the test failed. For details see the example.

 To tell the test suite the name of your test function you have to enclose it
 into the '<ompts:testcode:functionname> </ompts:testcode:functionname>' tag.

 If there are differences between test and crosstest you can use the
 <ompts:check> </ompts:check> and <ompts:crosscheck> </ompts:crosscheck> tag.
 When generating the test the parser will use the code enclosed in
 <ompts:check> tags and cut out the code written in <ompts:crosscheck> tags. So
 you have two possibilities to write your template for test and crosstest: The
 first way you can write the complete code is to write the test in one
 <ompts:check> tag and later the crosstest in one <ompts:crosscheck> tag.  The
 second way is to write both tests only by enclosing differing parts in
 corresponding tags.

 The first method should be preferred if test and crosstest differ much from
 each other. The second e.g. if you only want to change a few options like
 replacing an omp singleprivate clause by an omp private clause or to cut out
 a directive like omp flush.  When you use the first way you have to take care
 of the function name! You have to declare it twice with
 <ompts:testcode:functionname>!

 - <ompts:orphan> </ompts:orphan> This tag can be used if you want to enable
   your test to check the directive in orphan regions, too.  The code enclosed
 in this part will be written to a separate function which will be called
 instead.  If you have variables which are used outside this region you have to
 declare them as global variables enclosed in an <ompts:orphan:vars> tag. For
 further information see the description of the <ompts:orphan:vars> tag.

 - <ompts:orphan:vars> </ompts:orphan:vars> This tag is used to specify global
   variables for an orphan region which allow the exchange of values between
 the main program and the orphaned functions. The usage differs between C and
 Fortran.  In C you have to use a single declaration for each variable. You can
 either declare all variables in one single or in several different regions. You
 must not initialize the variables inside!  In Fortran you have to put all
 declarations in one single tag. Because there exist no global variables as in
 C you have to use common blocks. For further information see the examples.

 III.2.  Adding tests to the test lists
 --------------------------------------

 After you have created a new test you have to add them to a testlist. Simply
 add the function name in a new Line into a file.


 ------------------------------------------------------------------------------


 IV.   Known Issues and Workaround
 ==============================================================================

 The Sun OS has a problem with the -maxdepth option on the 'make cleanall'
 command. This prevents the tests from being removed from the working directory
 and can cause problems with future tests. To remedy, edit the makefile line
 under the clean command:

      -rm [cf]test*.[cf] [cf]crosstest*.[cf] [cf]ctest*.[cf] [cf]orphan*.[cf]

 Change to:

      -rm [cf]test* [cf]crosstest* [cf]ctest* [cf]orphan*


 ------------------------------------------------------------------------------

 V.   Contact and Support
 ==============================================================================

 Contact: http://www.cs.uh.edu/~hpctools
	============================================================================
	\| OpenMP Validation Suite V 3.0 \|
	\| High Performance Computing Center, Stuttgart \|
	\| High Performance Computing and Tools, University of Houston \|
	\| Jan. 2012 \|
	============================================================================


	TABLE OF CONTENTS

	I INTRODUCTION
	I.1. Aims and general function
	I.2. Files and directories

	II USAGE
	II.1. First run with make
	II.2 Where to search for results
	II.3. Using the runtest script

	III Adding and modifying tests
	III.1. The template structure

	IV Known Issues and Workaround

	V Contact and Support

	------------------------------------------------------------------------------

	I. INTRODUCTION
	==============================================================================

	I.1. Aims and general function
	--------------------------------

	The OpenMP validation suite is designed to verify the correct implementation
	of OpenMP in compilers. It is capable of checking Fortran as well as c
	compiler.

	Testing the implementation is based on statistics. Each directive is tested
	by computing and verifying the result with the already known value. In most
	cases a wrong implementation can result in the right values. So the tests are
	run several times.

	Additionally, the validation suite creates so called crosstests for each
	directive. These are tests in which either the directive is missing or used
	with different arguments. If this so called crosstest fails, this indicates
	strongly that the previous test is capable of testing the directive.

	Lastly, an orphaned test is also run to determine if the directive being
	tested is able to correctly run when 'orphaned' from the main function.
	Essentially, the directive's code is placed into its own function which is
	called during execution of the main function and often inside a parallel
	region.


	I.2. Files and directories
	----------------------------


	d c directory containing the templates for the c tests
	d fortran directory containing the templates for the Fortran
	tests
	Makefile Makefile containing options for compilation
	common_utility.f
	omp_my_sleep.h thread save sleep function
	omp_testsuite.f Fortran header file
	omp_testsuite.h autogenerated c-header file
	ompts-c.conf configuration file for the c tests about how often
	the tests shall be executed or how large the loop size
	is
	ompts_makeHeader.pl perl module for automatically generation of an up
	to date header file
	ompts_parserFunctions.pm perl module containing general functions for
	the parser.pl script
	ompts_parser.pl script for generating the source code out of the templates
	ompts_standaloneProc.c framework for the c tests
	ompts_standaloneProc.f framework for the Fortran tests
	README the README file you've already found ;-)
	LICENSE contains license information
	runtest.pl the frame program of the test suite
	testlist-f.txt test list containing the available tests for Fortran
	testlist-c.txt test list containing the available tests for c


	------------------------------------------------------------------------------

	II. USAGE
	==============================================================================


	II.1. First run with make
	--------------------------


	You can do a first simple run of the testsuite only after one step of
	configuration:

	1) Modify the ompts.conf and ompts-c.conf file, change the number of threads
	and number of repetitions of each test.

	2) Modify the Makefile, uncommenting the CC/FC and CFLAGS/FFLAGS variables for
	the compiler of your choice.

	And now you can run the testsuite either for a C compiler with:

	> make ctest

	or for a Fortran compiler with:

	> make ftest


	II.2. Running custom tests
	--------------------------


	In order to run single tests or custom groups of tests, two make commmands
	are defined: make stest and make fstest. These two command reference the file
	customtest.txt when looking for a testlist to use. Simply edit customtest.txt
	to include the desired test or tests. If customtest.txt contains c tests,

	> make stest

	or for fortran tests

	> make fstest

	In order to change the number of threads used in the tests, simply edit the
	Makefile variables MINTHREADS and MAXTHREADS. By default, they are configured
	to use 2 threads. To change the number of times each test is run, for c tests
	edit the REPETITIONS variable in the file ompts-c.conf. The LOOPCOUNT and
	SLEEPTIME variables can also be changed here. For fortran tests, edit the file
	omp_testsuite.f to change both the LOOPCOUNT and the number of times each test
	is run.



	II.3. Understanding the results
	---------------------------------


	When running the testsuite the results will be shown on the screen.

	If you need the results for further purpose you can use the results.txt, which
	is a simple list containing the results for each directive
	in a single line. Each line starts with the name of the directive. Then follows
	the result of the test given in the percentage of the passed tests. If 100% of
	the tests passed successfully, the second number gives the result of the
	corresponding crosstest. Crosstests will only be run if the normal test passes
	with 100% accuracy. If a crosstest was not run or a test does not exist, it is
	denotated by a "-".
	After the results of the normal tests, there follow a series of tests in the
	orphaned mode. If there were no orphaned tests available this is shown by a "-".

	If you run the testsuite with different numbers of threads (e.g. using the
	runtest.pl script) the results are shown in blocks of 4 columns for each number
	of threads.

	If a test fails you can find more detailed information in the ompts.log,
	bin/c/.out and .log files. While the ompts.log file contains all compiler
	error messages for all tests, the .out and .log files contain detailed inforamtion
	on the execution process of the single tests.
	In the *.out files there are listed all the results of the single executions of
	the tests. In the *.log files there are error messages of the tests itself.


	II.4. Cleaning Up
	-----------------


	Because many files are generated for each tested directive, it is often necessary
	to clean the main directory after a battery of tests. To clean all generated files
	in the main directory including the results and log files,

	> make clean


	To clean only the logs and out files,

	> make cleanlogs

	To clean only the results,

	> make cleanresults



	II.4. Using the runtest script
	-------------------------------


	So for special purpose you can use the runtest.pl script, which allows a lot
	more options for the execution process than the execution with make.

	Using the runtest.pl script is rather easy. You can use the the test suite only
	after two steps of modifications:

	1.) Modify the Makefile to your wishes choosing your compiler and the necessary
	compiler flags.
	2.) If necessary edit one of the test lists (testlist-c.txt) and comment out the
	tests you do not want to run using # at the beginning of a line. Testlists for Fortran end
	with -f.txt while test lists for c with -c.txt.

	And now you can run the test suite either for Fortran using
	# > ./runtest.pl -lang=fortran -d=fortran testlist-f.txt
	or for c
	# > ./runtest.pl -lang=c -d=c testlist-c.txt

	With the --help option you can show the complete list of options and their
	explanations.

	The test results are summarized in cresults or fresults.txt while *.log keep
	details for individual tests. There is also a file (ompts.log) keeping
	compilation information. (see section II.2 )

	If you don't want to test the directives in orphaned mode you can use the
	-norphan option. You also can use the runtest.pl script either to compile all
	tests or run compiled tests e.g. for cross compilation on other platforms. For
	this there are the options -norun and -nocompile.

	Happy testing!


	------------------------------------------------------------------------------

	III. How to add new tests / The structure of test templates
	==============================================================================

	III.1 The template structure
	------------------------------

	The test suite is based on templates so that you only have one file for test,
	crosstest and the orphaned versions of them.

	A) Description of the template structure

	The syntax of the templates is much like xml. So each test begins with
	'<ompts:test>' and ends with '</ompts:test>'.

	In between there are several other blocks holding information:

	- <ompts:testdescription> </ompts:testdescription> In between this tag you can
	give a description on what the test checks and how it works.

	- <ompts:ompversion> </ompts:version> This tag is used to specify the
	OpenMP-version which includes the tested directive.

	- <ompts:directive> </ompts:directive> Used to specify the directive how it is
	called in the programming language.

	- <ompts:dependences> </ompts:dependences> With this tag you can specify other
	omp directives which are necessary for the correct execution of your test.
	The directives have to be listed by their directive names as it is called in
	the programming language. Multiple directives are separated by ','.

	- <ompts:testcode> </ompts:testcode> In this tag stands the whole source code
	for the test / crosstest. Each test has to be written as a function. The
	syntax of the functions differs between C and Fortran: In C it has to take a
	file pointer 'FILE * logFile' and return an int. If the test has been passed
	successful it has to return a value unequal to 0. The file pointer can be used
	to write information into a log file. In Fortran the function takes no
	argument and the function name must not exceed XX characters. The return value
	has to be specified using the '<testfunctionname>' tags. It has to be 1 if the
	test succeeded and 0 if the test failed. For details see the example.

	To tell the test suite the name of your test function you have to enclose it
	into the '<ompts:testcode:functionname> </ompts:testcode:functionname>' tag.

	If there are differences between test and crosstest you can use the
	<ompts:check> </ompts:check> and <ompts:crosscheck> </ompts:crosscheck> tag.
	When generating the test the parser will use the code enclosed in
	<ompts:check> tags and cut out the code written in <ompts:crosscheck> tags. So
	you have two possibilities to write your template for test and crosstest: The
	first way you can write the complete code is to write the test in one
	<ompts:check> tag and later the crosstest in one <ompts:crosscheck> tag. The
	second way is to write both tests only by enclosing differing parts in
	corresponding tags.

	The first method should be preferred if test and crosstest differ much from
	each other. The second e.g. if you only want to change a few options like
	replacing an omp singleprivate clause by an omp private clause or to cut out
	a directive like omp flush. When you use the first way you have to take care
	of the function name! You have to declare it twice with
	<ompts:testcode:functionname>!

	- <ompts:orphan> </ompts:orphan> This tag can be used if you want to enable
	your test to check the directive in orphan regions, too. The code enclosed
	in this part will be written to a separate function which will be called
	instead. If you have variables which are used outside this region you have to
	declare them as global variables enclosed in an <ompts:orphan:vars> tag. For
	further information see the description of the <ompts:orphan:vars> tag.

	- <ompts:orphan:vars> </ompts:orphan:vars> This tag is used to specify global
	variables for an orphan region which allow the exchange of values between
	the main program and the orphaned functions. The usage differs between C and
	Fortran. In C you have to use a single declaration for each variable. You can
	either declare all variables in one single or in several different regions. You
	must not initialize the variables inside! In Fortran you have to put all
	declarations in one single tag. Because there exist no global variables as in
	C you have to use common blocks. For further information see the examples.

	III.2. Adding tests to the test lists
	--------------------------------------

	After you have created a new test you have to add them to a testlist. Simply
	add the function name in a new Line into a file.



	------------------------------------------------------------------------------


	IV. Known Issues and Workaround
	==============================================================================

	The Sun OS has a problem with the -maxdepth option on the 'make cleanall'
	command. This prevents the tests from being removed from the working directory
	and can cause problems with future tests. To remedy, edit the makefile line
	under the clean command:

	-rm [cf]test.[cf] [cf]crosstest.[cf] [cf]ctest.[cf] [cf]orphan.[cf]

	Change to:

	-rm [cf]test* [cf]crosstest* [cf]ctest* [cf]orphan*



	------------------------------------------------------------------------------

	V. Contact and Support
	==============================================================================

	Contact: http://www.cs.uh.edu/~hpctools