| ============== |
| Testing libc++ |
| ============== |
| |
| .. contents:: |
| :local: |
| |
| Getting Started |
| =============== |
| |
| libc++ uses LIT to configure and run its tests. The primary way to run the |
| libc++ tests is by using make check-libcxx. However since libc++ can be used |
| in any number of possible configurations it is important to customize the way |
| LIT builds and runs the tests. This guide provides information on how to use |
| LIT directly to test libc++. |
| |
| Please see the `Lit Command Guide`_ for more information about LIT. |
| |
| .. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html |
| |
| Setting up the Environment |
| -------------------------- |
| |
| After building libc++ you must setup your environment to test libc++ using |
| LIT. |
| |
| #. Create a shortcut to the actual lit executable so that you can invoke it |
| easily from the command line. |
| |
| .. code-block:: bash |
| |
| $ alias lit='python path/to/llvm/utils/lit/lit.py' |
| |
| #. Tell LIT where to find your build configuration. |
| |
| .. code-block:: bash |
| |
| $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg |
| |
| Example Usage |
| ------------- |
| |
| Once you have your environment set up and you have built libc++ you can run |
| parts of the libc++ test suite by simply running `lit` on a specified test or |
| directory. For example: |
| |
| .. code-block:: bash |
| |
| $ cd path/to/src/libcxx |
| $ lit -sv test/std/re # Run all of the std::regex tests |
| $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test |
| $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic |
| |
| Sometimes you'll want to change the way LIT is running the tests. Custom options |
| can be specified using the `--param=<name>=<val>` flag. The most common option |
| you'll want to change is the standard dialect (ie -std=c++XX). By default the |
| test suite will select the newest C++ dialect supported by the compiler and use |
| that. However if you want to manually specify the option like so: |
| |
| .. code-block:: bash |
| |
| $ lit -sv test/std/containers # Run the tests with the newest -std |
| $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03 |
| |
| Occasionally you'll want to add extra compile or link flags when testing. |
| You can do this as follows: |
| |
| .. code-block:: bash |
| |
| $ lit -sv --param=compile_flags='-Wcustom-warning' |
| $ lit -sv --param=link_flags='-L/custom/library/path' |
| |
| Some other common examples include: |
| |
| .. code-block:: bash |
| |
| # Specify a custom compiler. |
| $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std |
| |
| # Enable warnings in the test suite |
| $ lit -sv --param=enable_warnings=true test/std |
| |
| # Use UBSAN when running the tests. |
| $ lit -sv --param=use_sanitizer=Undefined |
| |
| |
| LIT Options |
| =========== |
| |
| :program:`lit` [*options*...] [*filenames*...] |
| |
| Command Line Options |
| -------------------- |
| |
| To use these options you pass them on the LIT command line as --param NAME or |
| --param NAME=VALUE. Some options have default values specified during CMake's |
| configuration. Passing the option on the command line will override the default. |
| |
| .. program:: lit |
| |
| .. option:: cxx_under_test=<path/to/compiler> |
| |
| Specify the compiler used to build the tests. |
| |
| .. option:: cxx_stdlib_under_test=<stdlib name> |
| |
| **Values**: libc++, libstdc++ |
| |
| Specify the C++ standard library being tested. Unless otherwise specified |
| libc++ is used. This option is intended to allow running the libc++ test |
| suite against other standard library implementations. |
| |
| .. option:: std=<standard version> |
| |
| **Values**: c++98, c++03, c++11, c++14, c++17, c++2a |
| |
| Change the standard version used when building the tests. |
| |
| .. option:: libcxx_site_config=<path/to/lit.site.cfg> |
| |
| Specify the site configuration to use when running the tests. This option |
| overrides the environment variable LIBCXX_SITE_CONFIG. |
| |
| .. option:: cxx_headers=<path/to/headers> |
| |
| Specify the c++ standard library headers that are tested. By default the |
| headers in the source tree are used. |
| |
| .. option:: cxx_library_root=<path/to/lib/> |
| |
| Specify the directory of the libc++ library to be tested. By default the |
| library folder of the build directory is used. This option cannot be used |
| when use_system_cxx_lib is provided. |
| |
| |
| .. option:: cxx_runtime_root=<path/to/lib/> |
| |
| Specify the directory of the libc++ library to use at runtime. This directory |
| is not added to the linkers search path. This can be used to compile tests |
| against one version of libc++ and run them using another. The default value |
| for this option is `cxx_library_root`. |
| |
| .. option:: use_system_cxx_lib=<bool> |
| |
| **Default**: False |
| |
| Enable or disable testing against the installed version of libc++ library. |
| Note: This does not use the installed headers. |
| |
| .. option:: use_lit_shell=<bool> |
| |
| Enable or disable the use of LIT's internal shell in ShTests. If the |
| environment variable LIT_USE_INTERNAL_SHELL is present then that is used as |
| the default value. Otherwise the default value is True on Windows and False |
| on every other platform. |
| |
| .. option:: compile_flags="<list-of-args>" |
| |
| Specify additional compile flags as a space delimited string. |
| Note: This options should not be used to change the standard version used. |
| |
| .. option:: link_flags="<list-of-args>" |
| |
| Specify additional link flags as a space delimited string. |
| |
| .. option:: debug_level=<level> |
| |
| **Values**: 0, 1 |
| |
| Enable the use of debug mode. Level 0 enables assertions and level 1 enables |
| assertions and debugging of iterator misuse. |
| |
| .. option:: use_sanitizer=<sanitizer name> |
| |
| **Values**: Memory, MemoryWithOrigins, Address, Undefined |
| |
| Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when |
| building libc++ then that sanitizer will be used by default. |
| |
| .. option:: color_diagnostics |
| |
| Enable the use of colorized compile diagnostics. If the color_diagnostics |
| option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is |
| present then color diagnostics will be enabled. |
| |
| .. option:: llvm_unwinder |
| |
| Enable the use of LLVM unwinder instead of libgcc. |
| |
| .. option:: builtins_library |
| |
| Path to the builtins library to use instead of libgcc. |
| |
| |
| Environment Variables |
| --------------------- |
| |
| .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg> |
| |
| Specify the site configuration to use when running the tests. |
| Also see `libcxx_site_config`. |
| |
| .. envvar:: LIBCXX_COLOR_DIAGNOSTICS |
| |
| If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt |
| to use color diagnostic outputs from the compiler. |
| Also see `color_diagnostics`. |
| |
| Benchmarks |
| ========== |
| |
| Libc++ contains benchmark tests separately from the test of the test suite. |
| The benchmarks are written using the `Google Benchmark`_ library, a copy of which |
| is stored in the libc++ repository. |
| |
| For more information about using the Google Benchmark library see the |
| `official documentation <https://github.com/google/benchmark>`_. |
| |
| .. _`Google Benchmark`: https://github.com/google/benchmark |
| |
| Building Benchmarks |
| ------------------- |
| |
| The benchmark tests are not built by default. The benchmarks can be built using |
| the ``cxx-benchmarks`` target. |
| |
| An example build would look like: |
| |
| .. code-block:: bash |
| |
| $ cd build |
| $ cmake [options] <path to libcxx sources> |
| $ make cxx-benchmarks |
| |
| This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be |
| built against the just-built libc++. The compiled tests are output into |
| ``build/benchmarks``. |
| |
| The benchmarks can also be built against the platforms native standard library |
| using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This |
| is useful for comparing the performance of libc++ to other standard libraries. |
| The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and |
| ``<test>.native.out`` otherwise. |
| |
| Also See: |
| |
| * :ref:`Building Libc++ <build instructions>` |
| * :ref:`CMake Options` |
| |
| Running Benchmarks |
| ------------------ |
| |
| The benchmarks must be run manually by the user. Currently there is no way |
| to run them as part of the build. |
| |
| For example: |
| |
| .. code-block:: bash |
| |
| $ cd build/benchmarks |
| $ make cxx-benchmarks |
| $ ./algorithms.libcxx.out # Runs all the benchmarks |
| $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks |
| |
| For more information about running benchmarks see `Google Benchmark`_. |