External/sollve_vv/README

SOLLVE OpenMP Offloading Validation & Verification Suite
https://crpl.cis.udel.edu/ompvvsollve/

This directory contains a CMakeLists.txt for the SOLLVE OpenMP
Offloading Validation and Verification Suite so it can be built as part
of the LLVM test-suite. Its sources are not part of the test-suite but
have to be fetched separately from https://github.com/SOLLVE/sollve_vv.

The sources are expected either in ${TEST_SUITE_SOLLVEVV_ROOT} or
${TEST_SUITE_EXTERNALS_DIR}/sollve_vv, where TEST_SUITE_SOLLVEVV_ROOT
and TEST_SUITE_EXTERNALS_DIR are CMake configure variables. If none of
them are set, it will look into
${CMAKE_SOURCE_DIR}/test-suite-externals/sollve_vv where
CMAKE_SOURCE_DIR is the root directory of the test-suite sources.

The CMakeLists.txt will search for all C, C++, and Fortran source files of the
OpenMP V&V suite, compile and run them. That is, running llvm-lit
(or "make check") will require a compatible accelerator on the running
machine.

Fortran source files won't be run unless specified. Add TEST_SUITE_FORTRAN=ON
to the CMake build.

To specify the offloading flags, add TEST_SUITE_OFFLOADING_<lang>_FLAGS and
TEST_SUITE_OFFLOADING_<lang>_LDFLAGS where <lang> is C, CXX, or Fortran to the
CMake build. It defaults to `--offload-arch=native`, which is Clang's command
line option to compile for the GPU(s) installed in the current machine.

Not all features that the SOLLVE V&V suite is testing are currently supported
by Clang/Flang. To avoid tests that are known to always fail, the CMake option
`TEST_SUITE_SYSTEM_GPU` can be used to pre-select a set of tests are are known
to work. Available options are:

 * `-DTEST_SUITE_SYSTEM_GPU=amd` (select tests that should work with Clang/Flang on recent AMD GPUs)
 * `-DTEST_SUITE_SYSTEM_GPU=nvidia` (select tests that should work with Clang/Flang on recent Nvidia GPUs)
 * `-DTEST_SUITE_SYSTEM_GPU=native` (select one of the options above based on which GPU is installed)
 * `-DTEST_SUITE_SYSTEM_GPU=all` (select all SOLLVE V&V tests)
 * `-DTEST_SUITE_SYSTEM_GPU=auto` (Use suitable preselected list, otherwise `all`)


An example run is:

    $ COMMON_OFFLOAD="-fopenmp-targets=nvptx64-nvidia-cuda --cuda-path=/soft/compilers/cuda/cuda-10.1.243 -Xopenmp-target -march=sm_70" \
      cmake ../llvm-test-suite -GNinja -DCMAKE_BUILD_TYPE=Release           \
      -DTEST_SUITE_SOLLVEVV_ROOT=${HOME}/src/sollve_vv                      \
      -DTEST_SUITE_LIT=${HOME}/build/llvm-project/release/bin/llvm-lit      \
      -DCMAKE_C_COMPILER=${HOME}/install/llvm-project/release/bin/clang     \
      -DCMAKE_CXX_COMPILER=${HOME}/install/llvm-project/release/bin/clang++ \
      -DTEST_SUITE_SUBDIRS=External/sollve_vv                               \
      "-DTEST_SUITE_OFFLOADING_C_FLAGS=${COMMON_OFFLOAD}"                   \
      "-DTEST_SUITE_OFFLOADING_CXX_FLAGS=${COMMON_OFFLOAD}"                 \
      "-DTEST_SUITE_OFFLOADING_C_LDFLAGS=${COMMON_OFFLOAD}"                 \
      "-DTEST_SUITE_OFFLOADING_CXX_LDFLAGS=${COMMON_OFFLOAD}"               \
      -DTEST_SUITE_SYSTEM_GPU=nvidia                                        \
      -DTEST_SUITE_LIT_FLAGS=-svj1

    $ LD_LIBRARY_PATH=${HOME}/install/llvm-project/release/lib ninja check

Clang also needs to be compiled with enabled offloading for the chosen
target. A configuration compatible for the commands above is:

    $ cmake ../llvm-project/llvm -GNinja -DCMAKE_BUILD_TYPE=Release   \
      -DCMAKE_INSTALL_PREFIX=${HOME}/install/llvm-project/release     \
      "-DLLVM_ENABLE_PROJECTS=clang"                                  \
      "-DLLVM_ENABLE_RUNTIMES=openmp;offload"                         \
      -DCUDA_TOOLKIT_ROOT_DIR=/soft/compilers/cuda/cuda-10.1.243

    $ ninja install

In this example, Clang is not installed into a default search path such
that the paths have to be specified explicitly. "-DCUDA_TOOLKIT_ROOT_DIR"
and "--cuda-path" point to the CUDA SDK to use. The option
"-DTEST_SUITE_LIT_FLAGS=-j1" is required to not run the tests in parallel;
multiple tests may conflict while running on the same GPU and thus fail.