llvm/docs/HowToCrossCompileBuiltinsOnArm.md - llvm-project - Git at Google


            
            # How to Cross Compile Compiler-rt Builtins For Arm


            ## Introduction


            This document contains information about building and testing the builtins part

            
            of compiler-rt for an Arm target, from an x86_64 Linux machine.


            While this document concentrates on Arm and Linux, the general principles should

            
            apply to other targets supported by compiler-rt. Further contributions for other

            
            targets are welcome.


            The instructions in this document depend on libraries and programs external to

            
            LLVM. There are many ways to install and configure these dependencies, so you

            
            may need to adapt the instructions here to fit your own situation.


            ## Prerequisites


            In this use case, we will be using cmake on a Debian-based Linux system,

            
            cross-compiling from an x86_64 host to a hard-float Armv7-A target. We will be

            
            using as many of the LLVM tools as we can, but it is possible to use GNU

            
            equivalents.


            You will need:

            
            - A build of LLVM for the llvm-tools and LLVM CMake files.

            
            - A clang executable with support for the `ARM` target.

            
            - `compiler-rt` sources.

            
            - The `qemu-arm` user mode emulator.

            
            - An `arm-linux-gnueabihf` sysroot.


            :::{note}

            
            An existing sysroot is required because some of the builtins include C library

            
            headers and a sysroot is the easiest way to get those.

            
            :::


            In this example, we will be using `ninja` as the build tool.


            See <https://compiler-rt.llvm.org/> for information about the dependencies

            
            on clang and LLVM.


            See [Getting Started with the LLVM System](GettingStarted.md) for information about obtaining

            
            the source for LLVM and compiler-rt.


            `qemu-arm` should be available as a package for your Linux distribution.


            The most complicated of the prerequisites to satisfy is the `arm-linux-gnueabihf`

            
            sysroot. In theory, it is possible to use the Linux distributions multiarch

            
            support to fulfill the dependencies for building but unfortunately due to

            
            `/usr/local/include` being added some host includes are selected.


            The easiest way to supply a sysroot is to download an `arm-linux-gnueabihf`

            
            toolchain from <https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads>.


            ## Building compiler-rt builtins for Arm


            We will be doing a standalone build of compiler-rt. The command is shown below.

            
            Shell variables are used to simplify some of the options:


            ```

            
            LLVM_TOOLCHAIN=<path-to-llvm-install>/

            
            TARGET_TRIPLE=arm-none-linux-gnueabihf

            
            GCC_TOOLCHAIN=<path-to-gcc-toolchain>

            
            SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc

            
            COMPILE_FLAGS="-march=armv7-a"


            cmake ../llvm-project/compiler-rt \

            
              -G Ninja \

            
              -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \

            
              -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \

            
              -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \

            
              -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \

            
              -DCMAKE_SYSROOT="${SYSROOT}" \

            
              -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \

            
              -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \

            
              -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \

            
              -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \

            
              -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \

            
              -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \

            
              -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \

            
              -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \

            
              -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \

            
              -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \

            
              -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \

            
              -DCOMPILER_RT_BUILD_BUILTINS=ON \

            
              -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \

            
              -DCOMPILER_RT_BUILD_MEMPROF=OFF \

            
              -DCOMPILER_RT_BUILD_PROFILE=OFF \

            
              -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \

            
              -DCOMPILER_RT_BUILD_SANITIZERS=OFF \

            
              -DCOMPILER_RT_BUILD_XRAY=OFF \

            
              -DCOMPILER_RT_BUILD_ORC=OFF \

            
              -DCOMPILER_RT_BUILD_CRT=OFF \

            
              -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \

            
              -DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \

            
              -DCOMPILER_RT_INCLUDE_TESTS=ON \

            
              -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \

            
              -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} ${COMPILE_FLAGS} --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"

            
            ```


            :::{note}

            
            The command above also enables tests. Enabling tests is not required, more details

            
            in the testing section.

            
            :::


            `CMAKE_<LANGUAGE>_<OPTION>` options are set so that the correct `--target`,

            
            `--sysroot`, `--gcc-toolchain` and `-march` options will be given to the

            
            compilers.


            The combination of these settings needs to be enough to pass CMake's compiler

            
            checks, compile compiler-rt and build the test cases.


            The flags need to select:

            
            : - The Arm target (`--target arm-none-linux-gnueabihf`)

            
              - The Arm architecture level (`-march=armv7-a`)

            
              - Whether to generate Arm (`-marm`, the default) or Thumb (`-mthumb`) instructions.


            It is possible to pass all these flags to CMake using `CMAKE_<LANGUAGE>_FLAGS`,

            
            but the command above uses standard CMake options instead. If you need to

            
            add flags that CMake cannot generate automatically, add them to

            
            `CMAKE_<LANGUAGE>_FLAGS`.


            When CMake has finished, build with Ninja:


            ```

            
            ninja builtins

            
            ```


            ## Testing compiler-rt builtins using qemu-arm


            The following options are required to enable tests:


            ```

            
            -DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \

            
            -DCOMPILER_RT_INCLUDE_TESTS=ON \

            
            -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \

            
            -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"

            
            ```


            This tells compiler-rt that we want to run tests on `qemu-arm`. If you do not

            
            want to run tests, remove these options from the CMake command.


            Note that `COMPILER_RT_TEST_COMPILER_CFLAGS` contains the equivalent of the

            
            options CMake generated for us with the first command. We must pass them

            
            manually here because standard options like `CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`

            
            do not apply here.


            When CMake has finished, run the tests:


            ```

            
            ninja check-builtins

            
            ```


            ## Troubleshooting


            ### The cmake try compile stage fails


            At an early stage cmake will attempt to compile and link a simple C program to

            
            test if the toolchain is working.


            This stage can often fail at link time if the `--sysroot=`, `--target`, or

            
            `--gcc-toolchain=` options are not passed to the compiler. Check the

            
            `CMAKE_<LANGUAGE>_FLAGS` and `CMAKE_<LANGAUGE>_COMPILER_TARGET` flags along

            
            with any of the specific CMake sysroot and toolchain options.


            It can be useful to build a simple example outside of cmake with your toolchain

            
            to make sure it is working. For example:


            ```

            
            clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c

            
            ```


            ### Clang uses the host header files


            On Debian-based systems, it is possible to install multiarch support for

            
            `arm-linux-gnueabi` and `arm-linux-gnueabihf`. In many cases clang can successfully

            
            use this multiarch support when `--gcc-toolchain=` and `--sysroot=` are not supplied.

            
            Unfortunately clang adds `/usr/local/include` before

            
            `/usr/include/arm-linux-gnueabihf` leading to errors when compiling the hosts

            
            header files.


            The multiarch support is not sufficient to build the builtins you will need to

            
            use a separate `arm-linux-gnueabihf` toolchain.


            ### No target passed to clang


            If clang is not given a target, it will typically use the host target. This will

            
            not understand the Arm assembly language files, resulting in error messages such

            
            as `error: unknown directive .syntax unified`.


            You can check the clang invocation in the error message to see if there is no

            
            `--target` or if it is set incorrectly. The cause is usually

            
            `CMAKE_ASM_FLAGS` not containing `--target` or `CMAKE_ASM_COMPILER_TARGET`

            
            not being present.


            ### Arm architecture not given


            The `--target=arm-linux-gnueabihf` will default to Arm architecture v4t which

            
            cannot assemble the barrier instructions used in the `synch_and_fetch` source

            
            files.


            The cause is usually a missing `-march=armv7a` from the `CMAKE_ASM_FLAGS`.


            ### Compiler-rt builds but the tests fail to build


            The flags used to build the tests are not the same as those used to build the

            
            builtins. The c flags are provided by `COMPILER_RT_TEST_COMPILE_CFLAGS` and

            
            the `CMAKE_C_COMPILER_TARGET`, `CMAKE_ASM_COMPILER_TARGET`,

            
            `CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN` and `CMAKE_SYSROOT` flags are not

            
            applied to tests.


            Make sure that `COMPILER_RT_TEST_COMPILE_CFLAGS` contains all the necessary

            
            flags.


            ## Modifications for other Targets


            ### Arm Soft-Float Target


            The instructions for the Arm hard-float target can be used for the soft-float

            
            target by substituting soft-float equivalents for the sysroot and target. The

            
            target to use is:


            - `-DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi`


            Depending on whether you want to use floating point instructions or not, you

            
            may need extra c-flags such as `-mfloat-abi=softfp` for use of floating-point

            
            instructions, and `-mfloat-abi=soft -mfpu=none` for software floating-point

            
            emulation.


            You will need to use an `arm-linux-gnueabi` GNU toolchain for soft-float.


            ### AArch64 Target


            The instructions for Arm can be used for AArch64 by substituting AArch64

            
            equivalents for the sysroot, emulator and target:


            ```

            
            -DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu

            
            -DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot

            
            ```


            You will also have to update any use of the target triple in compiler flags.

            
            For instance in `CMAKE_C_FLAGS` and `COMPILER_RT_TEST_COMPILER_CFLAGS`.


            ### Armv6-m, Armv7-m and Armv7E-M targets


            To build and test the libraries using a similar method to Armv7-A is possible

            
            but more difficult. The main problems are:


            - There is not a `qemu-arm` user-mode emulator for bare-metal systems.

            
              `qemu-system-arm` can be used, but this is significantly more difficult

            
              to setup. This document does not explain how to do this.

            
            - The targets to compile compiler-rt have the suffix `-none-eabi`. This uses

            
              the BareMetal driver in clang and by default will not find the libraries

            
              needed to pass the cmake compiler check.


            As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructions

            
            that are supported on Armv7-A we can still get most of the value of running the

            
            tests using the same `qemu-arm` that we used for Armv7-A by building and

            
            running the test cases for Armv7-A but using the builtins compiled for

            
            Armv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linked

            
            into a binary and execute the tests correctly, but it will not catch if the

            
            builtins use instructions that are supported on Armv7-A but not on Armv6-M,

            
            Armv7-M and Armv7E-M.


            This requires a second `arm-none-eabi` toolchain for building the builtins.

            
            Using a bare-metal toolchain ensures that the target and C library details are

            
            specific to bare-metal instead of using Linux settings. This means that some

            
            tests may behave differently compared to real hardware, but at least the content

            
            of the builtins library is correct.


            Below is an example that builds the builtins for Armv7-M, but runs the tests

            
            as Armv7-A. It is presented in full, but is very similar to the earlier

            
            command for Armv7-A build and test:


            ```

            
            LLVM_TOOLCHAIN=<path to llvm install>/


            # For the builtins.

            
            TARGET_TRIPLE=arm-none-eabi

            
            GCC_TOOLCHAIN=<path to arm-none-eabi toolchain>/

            
            SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc

            
            COMPILE_FLAGS="-march=armv7-m -mfpu=vfpv2"


            # For the test cases.

            
            A_PROFILE_TARGET_TRIPLE=arm-none-linux-gnueabihf

            
            A_PROFILE_GCC_TOOLCHAIN=<path to arm-none-linux-gnueabihf toolchain>/

            
            A_PROFILE_SYSROOT=${A_PROFILE_GCC_TOOLCHAIN}/${A_PROFILE_TARGET_TRIPLE}/libc


            cmake ../llvm-project/compiler-rt \

            
              -G Ninja \

            
              -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \

            
              -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \

            
              -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \

            
              -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \

            
              -DCMAKE_SYSROOT="${SYSROOT}" \

            
              -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \

            
              -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \

            
              -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \

            
              -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \

            
              -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \

            
              -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \

            
              -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \

            
              -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \

            
              -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \

            
              -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \

            
              -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \

            
              -DCOMPILER_RT_BUILD_BUILTINS=ON \

            
              -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \

            
              -DCOMPILER_RT_BUILD_MEMPROF=OFF \

            
              -DCOMPILER_RT_BUILD_PROFILE=OFF \

            
              -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \

            
              -DCOMPILER_RT_BUILD_SANITIZERS=OFF \

            
              -DCOMPILER_RT_BUILD_XRAY=OFF \

            
              -DCOMPILER_RT_BUILD_ORC=OFF \

            
              -DCOMPILER_RT_BUILD_CRT=OFF \

            
              -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \

            
              -DCOMPILER_RT_EMULATOR="qemu-arm -L ${A_PROFILE_SYSROOT}" \

            
              -DCOMPILER_RT_INCLUDE_TESTS=ON \

            
              -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \

            
              -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${A_PROFILE_TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${A_PROFILE_GCC_TOOLCHAIN} --sysroot=${A_PROFILE_SYSROOT} -fuse-ld=lld" \

            
              -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY \

            
              -DCOMPILER_RT_OS_DIR="baremetal" \

            
              -DCOMPILER_RT_BAREMETAL_BUILD=ON

            
            ```


            :::{note}

            
            The sysroot used for compiling the tests is `arm-linux-gnueabihf`, not

            
            `arm-none-eabi` which is used when compiling the builtins.

            
            :::


            The Armv6-M builtins will use the soft-float ABI. When compiling the tests for

            
            Armv7-A we must include `"-mthumb -mfloat-abi=soft -mfpu=none"` in the

            
            test-c-flags. We must use an Armv7-A soft-float ABI sysroot for `qemu-arm`.


            Depending on the linker used for the test cases, you may encounter BuildAttribute

            
            mismatches between the M-profile objects from compiler-rt and the A-profile

            
            objects from the test. The lld linker does not check the profile

            
            BuildAttribute so it can be used to link the tests by adding `-fuse-ld=lld` to the

            
            `COMPILER_RT_TEST_COMPILER_CFLAGS`.
	# How to Cross Compile Compiler-rt Builtins For Arm

	## Introduction

	This document contains information about building and testing the builtins part
	of compiler-rt for an Arm target, from an x86_64 Linux machine.

	While this document concentrates on Arm and Linux, the general principles should
	apply to other targets supported by compiler-rt. Further contributions for other
	targets are welcome.

	The instructions in this document depend on libraries and programs external to
	LLVM. There are many ways to install and configure these dependencies, so you
	may need to adapt the instructions here to fit your own situation.

	## Prerequisites

	In this use case, we will be using cmake on a Debian-based Linux system,
	cross-compiling from an x86_64 host to a hard-float Armv7-A target. We will be
	using as many of the LLVM tools as we can, but it is possible to use GNU
	equivalents.

	You will need:
	- A build of LLVM for the llvm-tools and LLVM CMake files.
	- A clang executable with support for the `ARM` target.
	- `compiler-rt` sources.
	- The `qemu-arm` user mode emulator.
	- An `arm-linux-gnueabihf` sysroot.

	:::{note}
	An existing sysroot is required because some of the builtins include C library
	headers and a sysroot is the easiest way to get those.
	:::

	In this example, we will be using `ninja` as the build tool.

	See <https://compiler-rt.llvm.org/> for information about the dependencies
	on clang and LLVM.

	See [Getting Started with the LLVM System](GettingStarted.md) for information about obtaining
	the source for LLVM and compiler-rt.

	`qemu-arm` should be available as a package for your Linux distribution.

	The most complicated of the prerequisites to satisfy is the `arm-linux-gnueabihf`
	sysroot. In theory, it is possible to use the Linux distributions multiarch
	support to fulfill the dependencies for building but unfortunately due to
	`/usr/local/include` being added some host includes are selected.

	The easiest way to supply a sysroot is to download an `arm-linux-gnueabihf`
	toolchain from <https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads>.

	## Building compiler-rt builtins for Arm

	We will be doing a standalone build of compiler-rt. The command is shown below.
	Shell variables are used to simplify some of the options:

	```
	LLVM_TOOLCHAIN=<path-to-llvm-install>/
	TARGET_TRIPLE=arm-none-linux-gnueabihf
	GCC_TOOLCHAIN=<path-to-gcc-toolchain>
	SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc
	COMPILE_FLAGS="-march=armv7-a"

	cmake ../llvm-project/compiler-rt \
	-G Ninja \
	-DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \
	-DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \
	-DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \
	-DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \
	-DCMAKE_SYSROOT="${SYSROOT}" \
	-DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \
	-DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \
	-DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \
	-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
	-DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
	-DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \
	-DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \
	-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
	-DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
	-DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \
	-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
	-DCOMPILER_RT_BUILD_BUILTINS=ON \
	-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
	-DCOMPILER_RT_BUILD_MEMPROF=OFF \
	-DCOMPILER_RT_BUILD_PROFILE=OFF \
	-DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \
	-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
	-DCOMPILER_RT_BUILD_XRAY=OFF \
	-DCOMPILER_RT_BUILD_ORC=OFF \
	-DCOMPILER_RT_BUILD_CRT=OFF \
	-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
	-DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \
	-DCOMPILER_RT_INCLUDE_TESTS=ON \
	-DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
	-DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} ${COMPILE_FLAGS} --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"
	```

	:::{note}
	The command above also enables tests. Enabling tests is not required, more details
	in the testing section.
	:::

	`CMAKE_<LANGUAGE>_<OPTION>` options are set so that the correct `--target`,
	`--sysroot`, `--gcc-toolchain` and `-march` options will be given to the
	compilers.

	The combination of these settings needs to be enough to pass CMake's compiler
	checks, compile compiler-rt and build the test cases.

	The flags need to select:
	: - The Arm target (`--target arm-none-linux-gnueabihf`)
	- The Arm architecture level (`-march=armv7-a`)
	- Whether to generate Arm (`-marm`, the default) or Thumb (`-mthumb`) instructions.

	It is possible to pass all these flags to CMake using `CMAKE_<LANGUAGE>_FLAGS`,
	but the command above uses standard CMake options instead. If you need to
	add flags that CMake cannot generate automatically, add them to
	`CMAKE_<LANGUAGE>_FLAGS`.

	When CMake has finished, build with Ninja:

	```
	ninja builtins
	```

	## Testing compiler-rt builtins using qemu-arm

	The following options are required to enable tests:

	```
	-DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \
	-DCOMPILER_RT_INCLUDE_TESTS=ON \
	-DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
	-DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"
	```

	This tells compiler-rt that we want to run tests on `qemu-arm`. If you do not
	want to run tests, remove these options from the CMake command.

	Note that `COMPILER_RT_TEST_COMPILER_CFLAGS` contains the equivalent of the
	options CMake generated for us with the first command. We must pass them
	manually here because standard options like `CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`
	do not apply here.

	When CMake has finished, run the tests:

	```
	ninja check-builtins
	```

	## Troubleshooting

	### The cmake try compile stage fails

	At an early stage cmake will attempt to compile and link a simple C program to
	test if the toolchain is working.

	This stage can often fail at link time if the `--sysroot=`, `--target`, or
	`--gcc-toolchain=` options are not passed to the compiler. Check the
	`CMAKE_<LANGUAGE>_FLAGS` and `CMAKE_<LANGAUGE>_COMPILER_TARGET` flags along
	with any of the specific CMake sysroot and toolchain options.

	It can be useful to build a simple example outside of cmake with your toolchain
	to make sure it is working. For example:

	```
	clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c
	```

	### Clang uses the host header files

	On Debian-based systems, it is possible to install multiarch support for
	`arm-linux-gnueabi` and `arm-linux-gnueabihf`. In many cases clang can successfully
	use this multiarch support when `--gcc-toolchain=` and `--sysroot=` are not supplied.
	Unfortunately clang adds `/usr/local/include` before
	`/usr/include/arm-linux-gnueabihf` leading to errors when compiling the hosts
	header files.

	The multiarch support is not sufficient to build the builtins you will need to
	use a separate `arm-linux-gnueabihf` toolchain.

	### No target passed to clang

	If clang is not given a target, it will typically use the host target. This will
	not understand the Arm assembly language files, resulting in error messages such
	as `error: unknown directive .syntax unified`.

	You can check the clang invocation in the error message to see if there is no
	`--target` or if it is set incorrectly. The cause is usually
	`CMAKE_ASM_FLAGS` not containing `--target` or `CMAKE_ASM_COMPILER_TARGET`
	not being present.

	### Arm architecture not given

	The `--target=arm-linux-gnueabihf` will default to Arm architecture v4t which
	cannot assemble the barrier instructions used in the `synch_and_fetch` source
	files.

	The cause is usually a missing `-march=armv7a` from the `CMAKE_ASM_FLAGS`.

	### Compiler-rt builds but the tests fail to build

	The flags used to build the tests are not the same as those used to build the
	builtins. The c flags are provided by `COMPILER_RT_TEST_COMPILE_CFLAGS` and
	the `CMAKE_C_COMPILER_TARGET`, `CMAKE_ASM_COMPILER_TARGET`,
	`CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN` and `CMAKE_SYSROOT` flags are not
	applied to tests.

	Make sure that `COMPILER_RT_TEST_COMPILE_CFLAGS` contains all the necessary
	flags.

	## Modifications for other Targets

	### Arm Soft-Float Target

	The instructions for the Arm hard-float target can be used for the soft-float
	target by substituting soft-float equivalents for the sysroot and target. The
	target to use is:

	- `-DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi`

	Depending on whether you want to use floating point instructions or not, you
	may need extra c-flags such as `-mfloat-abi=softfp` for use of floating-point
	instructions, and `-mfloat-abi=soft -mfpu=none` for software floating-point
	emulation.

	You will need to use an `arm-linux-gnueabi` GNU toolchain for soft-float.

	### AArch64 Target

	The instructions for Arm can be used for AArch64 by substituting AArch64
	equivalents for the sysroot, emulator and target:

	```
	-DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu
	-DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot
	```

	You will also have to update any use of the target triple in compiler flags.
	For instance in `CMAKE_C_FLAGS` and `COMPILER_RT_TEST_COMPILER_CFLAGS`.

	### Armv6-m, Armv7-m and Armv7E-M targets

	To build and test the libraries using a similar method to Armv7-A is possible
	but more difficult. The main problems are:

	- There is not a `qemu-arm` user-mode emulator for bare-metal systems.
	`qemu-system-arm` can be used, but this is significantly more difficult
	to setup. This document does not explain how to do this.
	- The targets to compile compiler-rt have the suffix `-none-eabi`. This uses
	the BareMetal driver in clang and by default will not find the libraries
	needed to pass the cmake compiler check.

	As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructions
	that are supported on Armv7-A we can still get most of the value of running the
	tests using the same `qemu-arm` that we used for Armv7-A by building and
	running the test cases for Armv7-A but using the builtins compiled for
	Armv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linked
	into a binary and execute the tests correctly, but it will not catch if the
	builtins use instructions that are supported on Armv7-A but not on Armv6-M,
	Armv7-M and Armv7E-M.

	This requires a second `arm-none-eabi` toolchain for building the builtins.
	Using a bare-metal toolchain ensures that the target and C library details are
	specific to bare-metal instead of using Linux settings. This means that some
	tests may behave differently compared to real hardware, but at least the content
	of the builtins library is correct.

	Below is an example that builds the builtins for Armv7-M, but runs the tests
	as Armv7-A. It is presented in full, but is very similar to the earlier
	command for Armv7-A build and test:

	```
	LLVM_TOOLCHAIN=<path to llvm install>/

	# For the builtins.
	TARGET_TRIPLE=arm-none-eabi
	GCC_TOOLCHAIN=<path to arm-none-eabi toolchain>/
	SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc
	COMPILE_FLAGS="-march=armv7-m -mfpu=vfpv2"

	# For the test cases.
	A_PROFILE_TARGET_TRIPLE=arm-none-linux-gnueabihf
	A_PROFILE_GCC_TOOLCHAIN=<path to arm-none-linux-gnueabihf toolchain>/
	A_PROFILE_SYSROOT=${A_PROFILE_GCC_TOOLCHAIN}/${A_PROFILE_TARGET_TRIPLE}/libc

	cmake ../llvm-project/compiler-rt \
	-G Ninja \
	-DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \
	-DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \
	-DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \
	-DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \
	-DCMAKE_SYSROOT="${SYSROOT}" \
	-DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \
	-DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \
	-DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \
	-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
	-DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
	-DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \
	-DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \
	-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
	-DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
	-DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \
	-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
	-DCOMPILER_RT_BUILD_BUILTINS=ON \
	-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
	-DCOMPILER_RT_BUILD_MEMPROF=OFF \
	-DCOMPILER_RT_BUILD_PROFILE=OFF \
	-DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \
	-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
	-DCOMPILER_RT_BUILD_XRAY=OFF \
	-DCOMPILER_RT_BUILD_ORC=OFF \
	-DCOMPILER_RT_BUILD_CRT=OFF \
	-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
	-DCOMPILER_RT_EMULATOR="qemu-arm -L ${A_PROFILE_SYSROOT}" \
	-DCOMPILER_RT_INCLUDE_TESTS=ON \
	-DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
	-DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${A_PROFILE_TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${A_PROFILE_GCC_TOOLCHAIN} --sysroot=${A_PROFILE_SYSROOT} -fuse-ld=lld" \
	-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY \
	-DCOMPILER_RT_OS_DIR="baremetal" \
	-DCOMPILER_RT_BAREMETAL_BUILD=ON
	```

	:::{note}
	The sysroot used for compiling the tests is `arm-linux-gnueabihf`, not
	`arm-none-eabi` which is used when compiling the builtins.
	:::

	The Armv6-M builtins will use the soft-float ABI. When compiling the tests for
	Armv7-A we must include `"-mthumb -mfloat-abi=soft -mfpu=none"` in the
	test-c-flags. We must use an Armv7-A soft-float ABI sysroot for `qemu-arm`.

	Depending on the linker used for the test cases, you may encounter BuildAttribute
	mismatches between the M-profile objects from compiler-rt and the A-profile
	objects from the test. The lld linker does not check the profile
	BuildAttribute so it can be used to link the tests by adding `-fuse-ld=lld` to the
	`COMPILER_RT_TEST_COMPILER_CFLAGS`.