docs/TestingGuide.md

# Compiler-rt Testing Infrastructure Guide

:::{contents}
:local: true
:::

## Overview

This document is the reference manual for the compiler-rt modifications to the
testing infrastructure. Documentation for the infrastructure itself can be found at
{ref}`llvm_testing_guide`.

## LLVM testing infrastructure organization

The compiler-rt testing infrastructure contains regression tests which are run
as part of the usual `make check-all` and are expected to always pass -- they
should be run before every commit.

## Quick start

The regressions tests are in the "compiler-rt" module and are normally checked
out in the directory `llvm/projects/compiler-rt/test`. Use `make check-all`
to run the regression tests after building compiler-rt.

### REQUIRES, XFAIL, etc.

Sometimes it is necessary to restrict a test to a specific target or mark it as
an "expected fail" or XFAIL. This is normally achieved using `REQUIRES:` or
`XFAIL:` and the `target=<target-triple>` feature, typically with a regular
expression matching an appropriate substring of the triple. Unfortunately, the
behaviour of this is somewhat quirky in compiler-rt. There are two main
pitfalls to avoid.

The first pitfall is that these regular expressions may inadvertently match
more triples than expected. For example, `XFAIL: target=mips{{.*}}` matches
`mips-linux-gnu`, `mipsel-linux-gnu`, `mips64-linux-gnu`, and
`mips64el-linux-gnu`. Including a trailing `-` such as in
`XFAIL: target=mips-{{.*}}` can help to mitigate this quirk but even that has
issues as described below.

The second pitfall is that the default target triple is often inappropriate for
compiler-rt tests since compiler-rt tests may be compiled for multiple targets.
For example, a typical build on an `x86_64-linux-gnu` host will often run the
tests for both x86_64 and i386. In this situation `XFAIL: target=x86_64{{{.*}}`
will mark both the x86_64 and i386 tests as an expected failure while
`XFAIL: target=i386{{.*}}` will have no effect at all.

To remedy both pitfalls, compiler-rt tests provide a feature string which can
be used to specify a single target. This string is of the form
`target-is-${arch}` where `${arch}}` is one of the values from the
following lines of the CMake output:

```
-- Compiler-RT supported architectures: x86_64;i386
-- Builtin supported architectures: i386;x86_64
```

So for example `XFAIL: target-is-x86_64` will mark a test as expected to fail
on x86_64 without also affecting the i386 test and `XFAIL: target-is-i386`
will mark a test as expected to fail on i386 even if the default target triple
is `x86_64-linux-gnu`. Directives that use these `target-is-${arch}` string
require exact matches so `XFAIL: target-is-mips`,
`XFAIL: target-is-mipsel`, `XFAIL: target-is-mips64`, and
`XFAIL: target-is-mips64el` all refer to different MIPS targets.