docs/resources/test.rst - llvm-project/lldb - Git at Google

 Testing
 =======

 .. contents::
    :local:

 Test Suite Structure
 --------------------

 The LLDB test suite consists of three different kinds of test:

 * **Unit tests**: written in C++ using the googletest unit testing library.
 * **Shell tests**: Integration tests that test the debugger through the command
   line. These tests interact with the debugger either through the command line
   driver or through ``lldb-test`` which is a tool that exposes the internal
   data structures in an easy-to-parse way for testing. Most people will know
   these as *lit tests* in LLVM, although lit is the test driver and ShellTest
   is the test format that uses ``RUN:`` lines. `FileCheck
   <https://llvm.org/docs/CommandGuide/FileCheck.html>`_ is used to verify
   the output.
 * **API tests**: Integration tests that interact with the debugger through the
   SB API. These are written in Python and use LLDB's ``dotest.py`` testing
   framework on top of Python's `unittest2
   <https://docs.python.org/2/library/unittest.html>`_.

 All three test suites use ``lit`` (`LLVM Integrated Tester
 <https://llvm.org/docs/CommandGuide/lit.html>`_ ) as the test driver. The test
 suites can be run as a whole or separately.


 Unit Tests
 ``````````

 Unit tests are located under ``lldb/unittests``. If it's possible to test
 something in isolation or as a single unit, you should make it a unit test.

 Often you need instances of the core objects such as a debugger, target or
 process, in order to test something meaningful. We already have a handful of
 tests that have the necessary boiler plate, but this is something we could
 abstract away and make it more user friendly.

 Shell Tests
 ```````````

 Shell tests are located under ``lldb/test/Shell``. These tests are generally
 built around checking the output of ``lldb`` (the command line driver) or
 ``lldb-test`` using ``FileCheck``. Shell tests are generally small and fast to
 write because they require little boilerplate.

 ``lldb-test`` is a relatively new addition to the test suite. It was the first
 tool that was added that is designed for testing. Since then it has been
 continuously extended with new subcommands, improving our test coverage. Among
 other things you can use it to query lldb for symbol files, for object files
 and breakpoints.

 Obviously shell tests are great for testing the command line driver itself or
 the subcomponents already exposed by lldb-test. But when it comes to LLDB's
 vast functionality, most things can be tested both through the driver as well
 as the Python API. For example, to test setting a breakpoint, you could do it
 from the command line driver with ``b main`` or you could use the SB API and do
 something like ``target.BreakpointCreateByName`` [#]_.

 A good rule of thumb is to prefer shell tests when what is being tested is
 relatively simple. Expressivity is limited compared to the API tests, which
 means that you have to have a well-defined test scenario that you can easily
 match with ``FileCheck``.

 Another thing to consider are the binaries being debugged, which we call
 inferiors. For shell tests, they have to be relatively simple. The
 ``dotest.py`` test framework has extensive support for complex build scenarios
 and different variants, which is described in more detail below, while shell
 tests are limited to single lines of shell commands with compiler and linker
 invocations.

 On the same topic, another interesting aspect of the shell tests is that there
 you can often get away with a broken or incomplete binary, whereas the API
 tests almost always require a fully functional executable. This enables testing
 of (some) aspects of handling of binaries with non-native architectures or
 operating systems.

 Finally, the shell tests always run in batch mode. You start with some input
 and the test verifies the output. The debugger can be sensitive to its
 environment, such as the the platform it runs on. It can be hard to express
 that the same test might behave slightly differently on macOS and Linux.
 Additionally, the debugger is an interactive tool, and the shell test provide
 no good way of testing those interactive aspects, such as tab completion for
 example.

 API Tests
 `````````

 API tests are located under ``lldb/test/API``. They are run with the
 ``dotest.py``. Tests are written in Python and test binaries (inferiors) are
 compiled with Make. The majority of API tests are end-to-end tests that compile
 programs from source, run them, and debug the processes.

 As mentioned before, ``dotest.py`` is LLDB's testing framework. The
 implementation is located under ``lldb/packages/Python/lldbsuite``. We have
 several extensions and custom test primitives on top of what's offered by
 `unittest2 <https://docs.python.org/2/library/unittest.html>`_. Those can be
 found  in
 `lldbtest.py <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/lldbtest.py>`_.

 Below is the directory layout of the `example API test
 <https://github.com/llvm/llvm-project/tree/main/lldb/test/API/sample_test>`_.
 The test directory will always contain a python file, starting with ``Test``.
 Most of the tests are structured as a binary being debugged, so there will be
 one or more source files and a ``Makefile``.

 ::

   sample_test
   ├── Makefile
   ├── TestSampleTest.py
   └── main.c

 Let's start with the Python test file. Every test is its own class and can have
 one or more test methods, that start with ``test_``.  Many tests define
 multiple test methods and share a bunch of common code. For example, for a
 fictive test that makes sure we can set breakpoints we might have one test
 method that ensures we can set a breakpoint by address, on that sets a
 breakpoint by name and another that sets the same breakpoint by file and line
 number. The setup, teardown and everything else other than setting the
 breakpoint could be shared.

 Our testing framework also has a bunch of utilities that abstract common
 operations, such as creating targets, setting breakpoints etc. When code is
 shared across tests, we extract it into a utility in ``lldbutil``. It's always
 worth taking a look at  `lldbutil
 <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/lldbutil.py>`_
 to see if there's a utility to simplify some of the testing boiler plate.
 Because we can't always audit every existing test, this is doubly true when
 looking at an existing test for inspiration.

 It's possible to skip or `XFAIL
 <https://ftp.gnu.org/old-gnu/Manuals/dejagnu-1.3/html_node/dejagnu_6.html>`_
 tests using decorators. You'll see them a lot. The debugger can be sensitive to
 things like the architecture, the host and target platform, the compiler
 version etc. LLDB comes with a range of predefined decorators for these
 configurations.

 ::

   @expectedFailureAll(archs=["aarch64"], oslist=["linux"]

 Another great thing about these decorators is that they're very easy to extend,
 it's even possible to define a function in a test case that determines whether
 the test should be run or not.

 ::

   @expectedFailure(checking_function_name)

 In addition to providing a lot more flexibility when it comes to writing the
 test, the API test also allow for much more complex scenarios when it comes to
 building inferiors. Every test has its own ``Makefile``, most of them only a
 few lines long. A shared ``Makefile`` (``Makefile.rules``) with about a
 thousand lines of rules takes care of most if not all of the boiler plate,
 while individual make files can be used to build more advanced tests.

 Here's an example of a simple ``Makefile`` used by the example test.

 ::

   C_SOURCES := main.c
   CFLAGS_EXTRAS := -std=c99

   include Makefile.rules

 Finding the right variables to set can be tricky. You can always take a look at
 `Makefile.rules <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/make/Makefile.rules>`_
 but often it's easier to find an existing ``Makefile`` that does something
 similar to what you want to do.

 Another thing this enables is having different variants for the same test
 case. By default, we run every test for all 3 debug info formats, so once with
 DWARF from the object files, once with gmodules and finally with a dSYM on
 macOS or split DWARF (DWO) on Linux. But there are many more things we can test
 that are orthogonal to the test itself. On GreenDragon we have a matrix bot
 that runs the test suite under different configurations, with older host
 compilers and different DWARF versions.

 As you can imagine, this quickly lead to combinatorial explosion in the number
 of variants. It's very tempting to add more variants because it's an easy way
 to increase test coverage. It doesn't scale. It's easy to set up, but increases
 the runtime of the tests and has a large ongoing cost.

 The key take away is that the different variants don't obviate the need for
 focused tests. So relying on it to test say DWARF5 is a really bad idea.
 Instead you should write tests that check the specific DWARF5 feature, and have
 the variant as a nice-to-have.

 In conclusion, you'll want to opt for an API test to test the API itself or
 when you need the expressivity, either for the test case itself or for the
 program being debugged. The fact that the API tests work with different
 variants mean that more general tests should be API tests, so that they can be
 run against the different variants.

 Running The Tests
 -----------------

 .. note::

    On Windows any invocations of python should be replaced with python_d, the
    debug interpreter, when running the test suite against a debug version of
    LLDB.

 .. note::

    On NetBSD you must export ``LD_LIBRARY_PATH=$PWD/lib`` in your environment.
    This is due to lack of the ``$ORIGIN`` linker feature.

 Running the Full Test Suite
 ```````````````````````````

 The easiest way to run the LLDB test suite is to use the ``check-lldb`` build
 target.

 By default, the ``check-lldb`` target builds the test programs with the same
 compiler that was used to build LLDB. To build the tests with a different
 compiler, you can set the ``LLDB_TEST_COMPILER`` CMake variable.

 It is possible to customize the architecture of the test binaries and compiler
 used by appending ``-A`` and ``-C`` options respectively to the CMake variable
 ``LLDB_TEST_USER_ARGS``. For example, to test LLDB against 32-bit binaries
 built with a custom version of clang, do:

 ::

    $ cmake -DLLDB_TEST_USER_ARGS="-A i386 -C /path/to/custom/clang" -G Ninja
    $ ninja check-lldb

 Note that multiple ``-A`` and ``-C`` flags can be specified to
 ``LLDB_TEST_USER_ARGS``.

 Running a Single Test Suite
 ```````````````````````````

 Each test suite can be run separately, similar to running the whole test suite
 with ``check-lldb``.

 * Use ``check-lldb-unit`` to run just the unit tests.
 * Use ``check-lldb-api`` to run just the SB API tests.
 * Use ``check-lldb-shell`` to run just the shell tests.

 You can run specific subdirectories by appending the directory name to the
 target. For example, to run all the tests in ``ObjectFile``, you can use the
 target ``check-lldb-shell-objectfile``. However, because the unit tests and API
 tests don't actually live under ``lldb/test``, this convenience is only
 available for the shell tests.

 Running a Single Test
 `````````````````````

 The recommended way to run a single test is by invoking the lit driver with a
 filter. This ensures that the test is run with the same configuration as when
 run as part of a test suite.

 ::

    $ ./bin/llvm-lit -sv tools/lldb/test --filter <test>


 Because lit automatically scans a directory for tests, it's also possible to
 pass a subdirectory to run a specific subset of the tests.

 ::

    $ ./bin/llvm-lit -sv tools/lldb/test/Shell/Commands/CommandScriptImmediateOutput


 For the SB API tests it is possible to forward arguments to ``dotest.py`` by
 passing ``--param`` to lit and setting a value for ``dotest-args``.

 ::

    $ ./bin/llvm-lit -sv tools/lldb/test --param dotest-args='-C gcc'


 Below is an overview of running individual test in the unit and API test suites
 without going through the lit driver.

 Running a Specific Test or Set of Tests: API Tests
 ``````````````````````````````````````````````````

 In addition to running all the LLDB test suites with the ``check-lldb`` CMake
 target above, it is possible to run individual LLDB tests. If you have a CMake
 build you can use the ``lldb-dotest`` binary, which is a wrapper around
 ``dotest.py`` that passes all the arguments configured by CMake.

 Alternatively, you can use ``dotest.py`` directly, if you want to run a test
 one-off with a different configuration.

 For example, to run the test cases defined in TestInferiorCrashing.py, run:

 ::

    $ ./bin/lldb-dotest -p TestInferiorCrashing.py

 ::

    $ cd $lldb/test
    $ python dotest.py --executable <path-to-lldb> -p TestInferiorCrashing.py ../packages/Python/lldbsuite/test

 If the test is not specified by name (e.g. if you leave the ``-p`` argument
 off),  all tests in that directory will be executed:


 ::

    $ ./bin/lldb-dotest functionalities/data-formatter

 ::

    $ python dotest.py --executable <path-to-lldb> functionalities/data-formatter

 Many more options that are available. To see a list of all of them, run:

 ::

    $ python dotest.py -h


 Running a Specific Test or Set of Tests: Unit Tests
 ```````````````````````````````````````````````````

 The unit tests are simple executables, located in the build directory under ``tools/lldb/unittests``.

 To run them, just run the test binary, for example, to run all the Host tests:

 ::

    $ ./tools/lldb/unittests/Host/HostTests


 To run a specific test, pass a filter, for example:

 ::

    $ ./tools/lldb/unittests/Host/HostTests --gtest_filter=SocketTest.DomainListenConnectAccept


 Running the Test Suite Remotely
 ```````````````````````````````

 Running the test-suite remotely is similar to the process of running a local
 test suite, but there are two things to have in mind:

 1. You must have the lldb-server running on the remote system, ready to accept
    multiple connections. For more information on how to setup remote debugging
    see the Remote debugging page.
 2. You must tell the test-suite how to connect to the remote system. This is
    achieved using the ``--platform-name``, ``--platform-url`` and
    ``--platform-working-dir`` parameters to ``dotest.py``. These parameters
    correspond to the platform select and platform connect LLDB commands. You
    will usually also need to specify the compiler and architecture for the
    remote system.

 Currently, running the remote test suite is supported only with ``dotest.py`` (or
 dosep.py with a single thread), but we expect this issue to be addressed in the
 near future.

 Running tests in QEMU System Emulation Environment
 ``````````````````````````````````````````````````

 QEMU can be used to test LLDB in an emulation environment in the absence of
 actual hardware. `QEMU based testing <https://lldb.llvm.org/use/qemu-testing.html>`_
 page describes how to setup a emulation environment using QEMU helper scripts
 found under llvm-project/lldb/scripts/lldb-test-qemu. These scripts currently
 work with Arm or AArch64, but support for other architectures can be added easily.

 Debugging Test Failures
 -----------------------

 On non-Windows platforms, you can use the ``-d`` option to ``dotest.py`` which
 will cause the script to wait for a while until a debugger is attached.

 Debugging Test Failures on Windows
 ``````````````````````````````````

 On Windows, it is strongly recommended to use Python Tools for Visual Studio
 for debugging test failures. It can seamlessly step between native and managed
 code, which is very helpful when you need to step through the test itself, and
 then into the LLDB code that backs the operations the test is performing.

 A quick guide to getting started with PTVS is as follows:

 #. Install PTVS
 #. Create a Visual Studio Project for the Python code.
     #. Go to File -> New -> Project -> Python -> From Existing Python Code.
     #. Choose llvm/tools/lldb as the directory containing the Python code.
     #. When asked where to save the .pyproj file, choose the folder ``llvm/tools/lldb/pyproj``. This is a special folder that is ignored by the ``.gitignore`` file, since it is not checked in.
 #. Set test/dotest.py as the startup file
 #. Make sure there is a Python Environment installed for your distribution. For example, if you installed Python to ``C:\Python35``, PTVS needs to know that this is the interpreter you want to use for running the test suite.
     #. Go to Tools -> Options -> Python Tools -> Environment Options
     #. Click Add Environment, and enter Python 3.5 Debug for the name. Fill out the values correctly.
 #. Configure the project to use this debug interpreter.
     #. Right click the Project node in Solution Explorer.
     #. In the General tab, Make sure Python 3.5 Debug is the selected Interpreter.
     #. In Debug/Search Paths, enter the path to your ninja/lib/site-packages directory.
     #. In Debug/Environment Variables, enter ``VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\``.
     #. If you want to enabled mixed mode debugging, check Enable native code debugging (this slows down debugging, so enable it only on an as-needed basis.)
 #. Set the command line for the test suite to run.
     #. Right click the project in solution explorer and choose the Debug tab.
     #. Enter the arguments to dotest.py.
     #. Example command options:

 ::

    --arch=i686
    # Path to debug lldb.exe
    --executable D:/src/llvmbuild/ninja/bin/lldb.exe
    # Directory to store log files
    -s D:/src/llvmbuild/ninja/lldb-test-traces
    -u CXXFLAGS -u CFLAGS
    # If a test crashes, show JIT debugging dialog.
    --enable-crash-dialog
    # Path to release clang.exe
    -C d:\src\llvmbuild\ninja_release\bin\clang.exe
    # Path to the particular test you want to debug.
    -p TestPaths.py
    # Root of test tree
    D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test

 ::

    --arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe -p TestPaths.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --no-multiprocess

 .. [#] `https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName <https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName>`_
	Testing
	=======

	.. contents::
	:local:

	Test Suite Structure
	--------------------

	The LLDB test suite consists of three different kinds of test:

	* Unit tests: written in C++ using the googletest unit testing library.
	* Shell tests: Integration tests that test the debugger through the command
	line. These tests interact with the debugger either through the command line
	driver or through ``lldb-test`` which is a tool that exposes the internal
	data structures in an easy-to-parse way for testing. Most people will know
	these as lit tests in LLVM, although lit is the test driver and ShellTest
	is the test format that uses ``RUN:`` lines. `FileCheck
	<https://llvm.org/docs/CommandGuide/FileCheck.html>`_ is used to verify
	the output.
	* API tests: Integration tests that interact with the debugger through the
	SB API. These are written in Python and use LLDB's ``dotest.py`` testing
	framework on top of Python's `unittest2
	<https://docs.python.org/2/library/unittest.html>`_.

	All three test suites use ``lit`` (`LLVM Integrated Tester
	<https://llvm.org/docs/CommandGuide/lit.html>`_ ) as the test driver. The test
	suites can be run as a whole or separately.


	Unit Tests
	``````````

	Unit tests are located under ``lldb/unittests``. If it's possible to test
	something in isolation or as a single unit, you should make it a unit test.

	Often you need instances of the core objects such as a debugger, target or
	process, in order to test something meaningful. We already have a handful of
	tests that have the necessary boiler plate, but this is something we could
	abstract away and make it more user friendly.

	Shell Tests
	```````````

	Shell tests are located under ``lldb/test/Shell``. These tests are generally
	built around checking the output of ``lldb`` (the command line driver) or
	``lldb-test`` using ``FileCheck``. Shell tests are generally small and fast to
	write because they require little boilerplate.

	``lldb-test`` is a relatively new addition to the test suite. It was the first
	tool that was added that is designed for testing. Since then it has been
	continuously extended with new subcommands, improving our test coverage. Among
	other things you can use it to query lldb for symbol files, for object files
	and breakpoints.

	Obviously shell tests are great for testing the command line driver itself or
	the subcomponents already exposed by lldb-test. But when it comes to LLDB's
	vast functionality, most things can be tested both through the driver as well
	as the Python API. For example, to test setting a breakpoint, you could do it
	from the command line driver with ``b main`` or you could use the SB API and do
	something like ``target.BreakpointCreateByName`` [#]_.

	A good rule of thumb is to prefer shell tests when what is being tested is
	relatively simple. Expressivity is limited compared to the API tests, which
	means that you have to have a well-defined test scenario that you can easily
	match with ``FileCheck``.

	Another thing to consider are the binaries being debugged, which we call
	inferiors. For shell tests, they have to be relatively simple. The
	``dotest.py`` test framework has extensive support for complex build scenarios
	and different variants, which is described in more detail below, while shell
	tests are limited to single lines of shell commands with compiler and linker
	invocations.

	On the same topic, another interesting aspect of the shell tests is that there
	you can often get away with a broken or incomplete binary, whereas the API
	tests almost always require a fully functional executable. This enables testing
	of (some) aspects of handling of binaries with non-native architectures or
	operating systems.

	Finally, the shell tests always run in batch mode. You start with some input
	and the test verifies the output. The debugger can be sensitive to its
	environment, such as the the platform it runs on. It can be hard to express
	that the same test might behave slightly differently on macOS and Linux.
	Additionally, the debugger is an interactive tool, and the shell test provide
	no good way of testing those interactive aspects, such as tab completion for
	example.

	API Tests
	`````````

	API tests are located under ``lldb/test/API``. They are run with the
	``dotest.py``. Tests are written in Python and test binaries (inferiors) are
	compiled with Make. The majority of API tests are end-to-end tests that compile
	programs from source, run them, and debug the processes.

	As mentioned before, ``dotest.py`` is LLDB's testing framework. The
	implementation is located under ``lldb/packages/Python/lldbsuite``. We have
	several extensions and custom test primitives on top of what's offered by
	`unittest2 <https://docs.python.org/2/library/unittest.html>`_. Those can be
	found in
	`lldbtest.py <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/lldbtest.py>`_.

	Below is the directory layout of the `example API test
	<https://github.com/llvm/llvm-project/tree/main/lldb/test/API/sample_test>`_.
	The test directory will always contain a python file, starting with ``Test``.
	Most of the tests are structured as a binary being debugged, so there will be
	one or more source files and a ``Makefile``.

	::

	sample_test
	├── Makefile
	├── TestSampleTest.py
	└── main.c

	Let's start with the Python test file. Every test is its own class and can have
	one or more test methods, that start with ``test_``. Many tests define
	multiple test methods and share a bunch of common code. For example, for a
	fictive test that makes sure we can set breakpoints we might have one test
	method that ensures we can set a breakpoint by address, on that sets a
	breakpoint by name and another that sets the same breakpoint by file and line
	number. The setup, teardown and everything else other than setting the
	breakpoint could be shared.

	Our testing framework also has a bunch of utilities that abstract common
	operations, such as creating targets, setting breakpoints etc. When code is
	shared across tests, we extract it into a utility in ``lldbutil``. It's always
	worth taking a look at `lldbutil
	<https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/lldbutil.py>`_
	to see if there's a utility to simplify some of the testing boiler plate.
	Because we can't always audit every existing test, this is doubly true when
	looking at an existing test for inspiration.

	It's possible to skip or `XFAIL
	<https://ftp.gnu.org/old-gnu/Manuals/dejagnu-1.3/html_node/dejagnu_6.html>`_
	tests using decorators. You'll see them a lot. The debugger can be sensitive to
	things like the architecture, the host and target platform, the compiler
	version etc. LLDB comes with a range of predefined decorators for these
	configurations.

	::

	@expectedFailureAll(archs=["aarch64"], oslist=["linux"]

	Another great thing about these decorators is that they're very easy to extend,
	it's even possible to define a function in a test case that determines whether
	the test should be run or not.

	::

	@expectedFailure(checking_function_name)

	In addition to providing a lot more flexibility when it comes to writing the
	test, the API test also allow for much more complex scenarios when it comes to
	building inferiors. Every test has its own ``Makefile``, most of them only a
	few lines long. A shared ``Makefile`` (``Makefile.rules``) with about a
	thousand lines of rules takes care of most if not all of the boiler plate,
	while individual make files can be used to build more advanced tests.

	Here's an example of a simple ``Makefile`` used by the example test.

	::

	C_SOURCES := main.c
	CFLAGS_EXTRAS := -std=c99

	include Makefile.rules

	Finding the right variables to set can be tricky. You can always take a look at
	`Makefile.rules <https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/make/Makefile.rules>`_
	but often it's easier to find an existing ``Makefile`` that does something
	similar to what you want to do.

	Another thing this enables is having different variants for the same test
	case. By default, we run every test for all 3 debug info formats, so once with
	DWARF from the object files, once with gmodules and finally with a dSYM on
	macOS or split DWARF (DWO) on Linux. But there are many more things we can test
	that are orthogonal to the test itself. On GreenDragon we have a matrix bot
	that runs the test suite under different configurations, with older host
	compilers and different DWARF versions.

	As you can imagine, this quickly lead to combinatorial explosion in the number
	of variants. It's very tempting to add more variants because it's an easy way
	to increase test coverage. It doesn't scale. It's easy to set up, but increases
	the runtime of the tests and has a large ongoing cost.

	The key take away is that the different variants don't obviate the need for
	focused tests. So relying on it to test say DWARF5 is a really bad idea.
	Instead you should write tests that check the specific DWARF5 feature, and have
	the variant as a nice-to-have.

	In conclusion, you'll want to opt for an API test to test the API itself or
	when you need the expressivity, either for the test case itself or for the
	program being debugged. The fact that the API tests work with different
	variants mean that more general tests should be API tests, so that they can be
	run against the different variants.

	Running The Tests
	-----------------

	.. note::

	On Windows any invocations of python should be replaced with python_d, the
	debug interpreter, when running the test suite against a debug version of
	LLDB.

	.. note::

	On NetBSD you must export ``LD_LIBRARY_PATH=$PWD/lib`` in your environment.
	This is due to lack of the ``$ORIGIN`` linker feature.

	Running the Full Test Suite
	```````````````````````````

	The easiest way to run the LLDB test suite is to use the ``check-lldb`` build
	target.

	By default, the ``check-lldb`` target builds the test programs with the same
	compiler that was used to build LLDB. To build the tests with a different
	compiler, you can set the ``LLDB_TEST_COMPILER`` CMake variable.

	It is possible to customize the architecture of the test binaries and compiler
	used by appending ``-A`` and ``-C`` options respectively to the CMake variable
	``LLDB_TEST_USER_ARGS``. For example, to test LLDB against 32-bit binaries
	built with a custom version of clang, do:

	::

	$ cmake -DLLDB_TEST_USER_ARGS="-A i386 -C /path/to/custom/clang" -G Ninja
	$ ninja check-lldb

	Note that multiple ``-A`` and ``-C`` flags can be specified to
	``LLDB_TEST_USER_ARGS``.

	Running a Single Test Suite
	```````````````````````````

	Each test suite can be run separately, similar to running the whole test suite
	with ``check-lldb``.

	* Use ``check-lldb-unit`` to run just the unit tests.
	* Use ``check-lldb-api`` to run just the SB API tests.
	* Use ``check-lldb-shell`` to run just the shell tests.

	You can run specific subdirectories by appending the directory name to the
	target. For example, to run all the tests in ``ObjectFile``, you can use the
	target ``check-lldb-shell-objectfile``. However, because the unit tests and API
	tests don't actually live under ``lldb/test``, this convenience is only
	available for the shell tests.

	Running a Single Test
	`````````````````````

	The recommended way to run a single test is by invoking the lit driver with a
	filter. This ensures that the test is run with the same configuration as when
	run as part of a test suite.

	::

	$ ./bin/llvm-lit -sv tools/lldb/test --filter <test>


	Because lit automatically scans a directory for tests, it's also possible to
	pass a subdirectory to run a specific subset of the tests.

	::

	$ ./bin/llvm-lit -sv tools/lldb/test/Shell/Commands/CommandScriptImmediateOutput


	For the SB API tests it is possible to forward arguments to ``dotest.py`` by
	passing ``--param`` to lit and setting a value for ``dotest-args``.

	::

	$ ./bin/llvm-lit -sv tools/lldb/test --param dotest-args='-C gcc'


	Below is an overview of running individual test in the unit and API test suites
	without going through the lit driver.

	Running a Specific Test or Set of Tests: API Tests
	``````````````````````````````````````````````````

	In addition to running all the LLDB test suites with the ``check-lldb`` CMake
	target above, it is possible to run individual LLDB tests. If you have a CMake
	build you can use the ``lldb-dotest`` binary, which is a wrapper around
	``dotest.py`` that passes all the arguments configured by CMake.

	Alternatively, you can use ``dotest.py`` directly, if you want to run a test
	one-off with a different configuration.

	For example, to run the test cases defined in TestInferiorCrashing.py, run:

	::

	$ ./bin/lldb-dotest -p TestInferiorCrashing.py

	::

	$ cd $lldb/test
	$ python dotest.py --executable <path-to-lldb> -p TestInferiorCrashing.py ../packages/Python/lldbsuite/test

	If the test is not specified by name (e.g. if you leave the ``-p`` argument
	off), all tests in that directory will be executed:


	::

	$ ./bin/lldb-dotest functionalities/data-formatter

	::

	$ python dotest.py --executable <path-to-lldb> functionalities/data-formatter

	Many more options that are available. To see a list of all of them, run:

	::

	$ python dotest.py -h


	Running a Specific Test or Set of Tests: Unit Tests
	```````````````````````````````````````````````````

	The unit tests are simple executables, located in the build directory under ``tools/lldb/unittests``.

	To run them, just run the test binary, for example, to run all the Host tests:

	::

	$ ./tools/lldb/unittests/Host/HostTests


	To run a specific test, pass a filter, for example:

	::

	$ ./tools/lldb/unittests/Host/HostTests --gtest_filter=SocketTest.DomainListenConnectAccept


	Running the Test Suite Remotely
	```````````````````````````````

	Running the test-suite remotely is similar to the process of running a local
	test suite, but there are two things to have in mind:

	1. You must have the lldb-server running on the remote system, ready to accept
	multiple connections. For more information on how to setup remote debugging
	see the Remote debugging page.
	2. You must tell the test-suite how to connect to the remote system. This is
	achieved using the ``--platform-name``, ``--platform-url`` and
	``--platform-working-dir`` parameters to ``dotest.py``. These parameters
	correspond to the platform select and platform connect LLDB commands. You
	will usually also need to specify the compiler and architecture for the
	remote system.

	Currently, running the remote test suite is supported only with ``dotest.py`` (or
	dosep.py with a single thread), but we expect this issue to be addressed in the
	near future.

	Running tests in QEMU System Emulation Environment
	``````````````````````````````````````````````````

	QEMU can be used to test LLDB in an emulation environment in the absence of
	actual hardware. `QEMU based testing <https://lldb.llvm.org/use/qemu-testing.html>`_
	page describes how to setup a emulation environment using QEMU helper scripts
	found under llvm-project/lldb/scripts/lldb-test-qemu. These scripts currently
	work with Arm or AArch64, but support for other architectures can be added easily.

	Debugging Test Failures
	-----------------------

	On non-Windows platforms, you can use the ``-d`` option to ``dotest.py`` which
	will cause the script to wait for a while until a debugger is attached.

	Debugging Test Failures on Windows
	``````````````````````````````````

	On Windows, it is strongly recommended to use Python Tools for Visual Studio
	for debugging test failures. It can seamlessly step between native and managed
	code, which is very helpful when you need to step through the test itself, and
	then into the LLDB code that backs the operations the test is performing.

	A quick guide to getting started with PTVS is as follows:

	#. Install PTVS
	#. Create a Visual Studio Project for the Python code.
	#. Go to File -> New -> Project -> Python -> From Existing Python Code.
	#. Choose llvm/tools/lldb as the directory containing the Python code.
	#. When asked where to save the .pyproj file, choose the folder ``llvm/tools/lldb/pyproj``. This is a special folder that is ignored by the ``.gitignore`` file, since it is not checked in.
	#. Set test/dotest.py as the startup file
	#. Make sure there is a Python Environment installed for your distribution. For example, if you installed Python to ``C:\Python35``, PTVS needs to know that this is the interpreter you want to use for running the test suite.
	#. Go to Tools -> Options -> Python Tools -> Environment Options
	#. Click Add Environment, and enter Python 3.5 Debug for the name. Fill out the values correctly.
	#. Configure the project to use this debug interpreter.
	#. Right click the Project node in Solution Explorer.
	#. In the General tab, Make sure Python 3.5 Debug is the selected Interpreter.
	#. In Debug/Search Paths, enter the path to your ninja/lib/site-packages directory.
	#. In Debug/Environment Variables, enter ``VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\``.
	#. If you want to enabled mixed mode debugging, check Enable native code debugging (this slows down debugging, so enable it only on an as-needed basis.)
	#. Set the command line for the test suite to run.
	#. Right click the project in solution explorer and choose the Debug tab.
	#. Enter the arguments to dotest.py.
	#. Example command options:

	::

	--arch=i686
	# Path to debug lldb.exe
	--executable D:/src/llvmbuild/ninja/bin/lldb.exe
	# Directory to store log files
	-s D:/src/llvmbuild/ninja/lldb-test-traces
	-u CXXFLAGS -u CFLAGS
	# If a test crashes, show JIT debugging dialog.
	--enable-crash-dialog
	# Path to release clang.exe
	-C d:\src\llvmbuild\ninja_release\bin\clang.exe
	# Path to the particular test you want to debug.
	-p TestPaths.py
	# Root of test tree
	D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test

	::

	--arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe -p TestPaths.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --no-multiprocess

	.. [#] `https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName <https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName>`_