| Overview |
| ================================================== |
| This directory contains a random test generator for the gmp |
| compatibility layer of imath. The tests cases are randomly generated |
| and run with both gmp and imath. The results are compared and any |
| mismatching results are flagged as failures. |
| |
| You should not see any failures when running these tests. |
| |
| Requirements |
| ================================================== |
| These tests use the python ffi to run the imath and gmp functions. To |
| run these tests you will need the following items |
| |
| * libimath.so |
| * python 3 |
| * gmp library and header files |
| |
| |
| Running |
| ================================================== |
| All the tests cases will be generated and run automatically by the |
| makefile. First, make sure you have built libimath.so in the top level |
| imath directory. |
| |
| Use the following command to generate and run the tests |
| |
| $ make TESTS=random.tests |
| |
| This should generate all the needed wrapper files along with the |
| randomized unit tests. By default the unit tests are output to the |
| random.tests file. The tests can be run by hand using the following |
| command line |
| |
| $ ./runtest random.tests |
| |
| You can also write your own unit tests and run them by passing the |
| file name to the runtest script. |
| |
| |
| Testing Methodology |
| ================================================== |
| The goal of our testing is to ensure that we are compatible with the |
| gmp api. To test our compatibility layer we generate inputs for each |
| gmp api that we wrap and then call the gmp version of the api and |
| compare the results to the imath version. The results should be |
| identical. Any output mismatch is considered an error. |
| |
| |
| Testcase Generation |
| -------------------- |
| The test data generation is inspired by the QuickCheck testing |
| methodolgy. We want to test both random data and important values such |
| as 0,1,-1, etc. We generate input data based on the type of the input |
| parameter. Most apis take either an mpz_t or a signed/unsigned long. |
| |
| For each parameter of the given type we generate the following values: |
| |
| mpz_t: 0,1,-1, all min/max values, all min/max values +/- 1 |
| small numbers: 1-4 digits |
| medium numbers: 5-20 digits |
| large numbers: 20-100 digits |
| long : 0,1,-1 short,int,long min/max values |
| random long values |
| unsigned long: 0,1,-1, unsigned short, unsigned int, unsigned long min/max values |
| random unsigned long values |
| |
| The generated data for each paramater is combined to produce a series |
| of inputs to the function. The input data format looks like: |
| |
| mpz_add|0,1,2 |
| |
| Which represents the call mpz_add(0, 1, 2). Additional test cases can be written by hand. |
| |
| Test Structure |
| -------------------- |
| The tests are run using the python ffi (the ctypes module). We have a |
| single description of each api that is used to generate the following: |
| |
| * intput data |
| * c function wrapper to call the libgmp or libimath function |
| * python wrapper to call both libgmp and libimath wrappers and compare results |
| |
| The test_gmp.so and test_imath.so libraries are loaded at runtime by |
| the python runner scrip and used to run each test input and compare |
| the results. |
| |
| The code generation pattern looks something like this: |
| |
| genctest.py ~~generates~~> gmp_test.c <--includes-- gmp_custom_test.c |
| gmp_test.c ~~generates~~> gmp_tests.so <--links-- libgmp.so |
| genpytest.py ~~generates~~> wrappers.py <--calls-- gmp_test.so |
| gmp_test.so --loads--> runtest.py <--includes-- wrappers.py |
| ^ |
| | |
| reads |
| | |
| gendata.py ~~generates~~> random.tests |
| |
| Adding tests for new APIs |
| ================================================== |
| New apis can be tested by modifying the gmpapy.py file. This file |
| contains a list of all apis that will be tested. Adding an api to the |
| apis list will cause code and input data to be generated for that api. |
| Rerunning make will generate the test data and code to run the test. |
| |