libc/docs/dev/code_coverage.md

(code_coverage)=

# Code Coverage

Code coverage is a software testing metric that measures the proportion of
source code executed while running an automated test suite. It provides insight
into test thoroughness by identifying untested functions, dead code paths, and
unexercised conditional branches across library entrypoints and internal
utilities.

LLVM-libc supports Modified Condition / Decision Coverage (MC/DC). MC/DC
evaluates compound boolean decisions composed of multiple sub-conditions (such
as `if (A && (B || C))`). Under MC/DC criteria, each individual boolean
condition must:
* Evaluate to both true and false across the test suite.
* Demonstrate that it can independently affect the outcome of the overall
  decision while other conditions remain fixed.

This provides rigorous structural verification for safety-critical algorithms
without requiring exhaustive testing of all 2<sup>n</sup> condition
permutations.

## Continuous Profiling Architecture

LLVM-libc uses Clang's [continuous profiling mode][clang-continuous]
(`-fprofile-continuous`) to record execution metrics directly into
memory-mapped profile files during test execution.

[clang-continuous]:
  https://clang.llvm.org/docs/UsersManual.html#cmdoption-fprofile-continuous

### Compiler Counter Relocation
When compiled with `-fprofile-continuous`, Clang configures the LLVM code
generator (`-mllvm -runtime-counter-relocation=true`) so that execution counter
increments reference a dynamic base pointer (`*(bias + &counter) += 1`). Each
branch and basic block counter dynamically resolves to an address within a
dedicated profile buffer mapped at program startup.

### Runtime Memory Mapping
During binary initialization, the profiling runtime (`libclang_rt.profile`)
resolves the target `.profraw` file and maps the execution counter section into
process memory using `mmap` with `MAP_SHARED`. The runtime sets the global bias
pointer to this mapped region, routing live counter increments directly into the
file-backed buffer.

### Kernel Page-Cache Synchronization
Execution counts are written directly to shared memory-mapped pages and
synchronized by the operating system kernel's page cache. Subprocesses created
via `fork()` share the same underlying memory mapping, committing statements
executed across parent and child processes directly to the profile file.

### Build System Integration
Setting `-DLIBC_ENABLE_COVERAGE=ON` in the CMake configuration passes
`-fprofile-instr-generate=libc_cov_%p.profraw`, `-fcoverage-mapping`, and
`-fprofile-continuous` across all LLVM-libc compilation units and test link
steps. When `-fprofile-continuous` is enabled, Clang automatically prepends
`%c` to the profile file template, avoiding duplicate specifier warnings at
runtime. Setting `-DLIBC_ENABLE_COVERAGE_MCDC=ON` additionally enables
`-fcoverage-mcdc`.

## Running Code Coverage Locally

### Prerequisites & Toolchain Setup

Generating coverage reports requires Clang 24, LLVM profile tools, CMake, and
Ninja:

* **Compiler:** Clang 24 (or Clang built from HEAD). Full-build hermetic tests
  link `libclang_rt.profile.a`, which must match the compiler version and cannot
  rely on distro-built libraries with glibc source fortification.
* **LLVM Utilities:** Matching Clang 24 versions of `llvm-profdata` and
  `llvm-cov`.
* **Linker:** `lld` is recommended when configuring full-build mode.
* **Build System:** CMake 3.28+ and Ninja.

#### Toolchain Discovery

If your Linux distribution packages version-suffixed binaries (e.g. `clang-24`,
`llvm-profdata-24`), discover and export them:

```bash
CLANG_MAJOR=$(clang --version | sed -n 's/.*version \([0-9]*\).*/\1/p')
export LLVM_PROFDATA=$(which llvm-profdata-$CLANG_MAJOR 2>/dev/null \
  || which llvm-profdata)
export LLVM_COV=$(which llvm-cov-$CLANG_MAJOR 2>/dev/null \
  || which llvm-cov)
```

If version-agnostic tools are directly available in your `PATH`, export:

```bash
export LLVM_PROFDATA=llvm-profdata
export LLVM_COV=llvm-cov
```

Subsequent merge and report commands reference `$LLVM_PROFDATA` and `$LLVM_COV`.

#### Building Clang 24 with Profiling Support

Full-build hermetic tests link `libclang_rt.profile.a`. Distro-built compiler-rt
packages on distributions like Debian or Ubuntu are built with glibc source
fortification enabled, which LLVM-libc does not support because it introduces
unresolved symbols such as `__vfprintf_chk`. Furthermore, compiler-rt must
match the exact version of the compiler used to build. The recommended approach
is building Clang 24, lld, and compiler-rt from HEAD:

```bash
cmake -G Ninja -S llvm -B build-clang \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$HOME/clang" \
  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld" \
  -DLLVM_ENABLE_RUNTIMES="compiler-rt" \
  -DLLVM_USE_LINKER=lld
ninja -C build-clang install
```

---

## Cleaning Profile Counters

Removes previously generated raw profile counter files (`.profraw`) and merged
profile databases (`.profdata`) so that new coverage runs record clean,
non-aggregated execution data:

```bash
find . -name "libc_cov_*.profraw" -delete 2>/dev/null || true
rm -f libc_full.profdata libc_mcdc.profdata libc_single.profdata \
  profraw_list.txt
```

---

## Standard Statement & Branch Coverage

Standard coverage measures physical line execution and conditional branch
outcomes across all LLVM-libc entrypoints and internal support utilities.

### 1. Clean Prior Profile Artifacts

Removes previously generated raw profile counters and profile data to maintain
a clean baseline:

```bash
rm -f build-cov/libc_cov_*.profraw libc_cov_*.profraw profraw_list.txt libc_full.profdata
```

### 2. Configure Full-Build Coverage with Clang 24

Configures CMake to build LLVM-libc with code coverage enabled using Clang 24
and LLD:

```bash
cmake -G Ninja -S runtimes -B build-cov \
  -DCMAKE_C_COMPILER="$PWD/build-clang/bin/clang" \
  -DCMAKE_CXX_COMPILER="$PWD/build-clang/bin/clang++" \
  -DLLVM_USE_LINKER=lld \
  -DCMAKE_BUILD_TYPE=Debug \
  -DLLVM_ENABLE_RUNTIMES="libc" \
  -DLLVM_LIBC_FULL_BUILD=ON \
  -DLIBC_ENABLE_COVERAGE=ON
```

### 3. Export Environment Variables

Exports the profile output pattern and Clang 24 tool paths:

```bash
export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
export LLVM_PROFDATA="$PWD/build-clang/bin/llvm-profdata"
export LLVM_COV="$PWD/build-clang/bin/llvm-cov"
```

### 4. Build and Execute Hermetic Tests

Compiles and executes the full hermetic test suite:

```bash
ninja -k 0 -C build-cov libc-hermetic-tests
```

:::{note}
The `-k 0` flag ensures Ninja continues executing all remaining test targets
even if an individual edge-case test encounters an error. To only compile test
binaries without executing them, use
`ninja -C build-cov libc-hermetic-tests-build`.
:::

### 5. Merge Profile Counters

Scans the build tree for all generated `.profraw` files and indexes them into a
unified, sparse `.profdata` archive using `$LLVM_PROFDATA`:

```bash
find build-cov -name "libc_cov_*.profraw" > profraw_list.txt
"$LLVM_PROFDATA" merge -sparse -f profraw_list.txt -o libc_full.profdata
```

### 6. Generate Coverage Reports

Collects all compiled test binary paths and invokes `$LLVM_COV` to correlate
recorded profile counters against the libc source tree:

```bash
TEST_BINS=($(find build-cov -type f -executable -name "*__build__"))
OBJECT_FLAGS=()
for bin in "${TEST_BINS[@]:1}"; do
  OBJECT_FLAGS+=("-object=$bin")
done
```

Reports can be generated in different formats:

#### Option 1: Terminal Summary Report

Prints an aggregated terminal summary showing line, region, and branch coverage
percentages for each file:

```bash
"$LLVM_COV" report \
  -instr-profile=libc_full.profdata \
  "${TEST_BINS[0]}" "${OBJECT_FLAGS[@]}" \
  --show-branch-summary \
  -ignore-filename-regex=".*(test|utils).*"
```

To restrict the terminal report to a specific source file:

```bash
"$LLVM_COV" report \
  -instr-profile=libc_full.profdata \
  "${TEST_BINS[0]}" "${OBJECT_FLAGS[@]}" \
  libc/src/string/strlen.cpp
```

#### Option 2: Interactive HTML Dashboard

Generates an interactive HTML dashboard containing sortable directory metrics
and syntax-highlighted source views:

```bash
"$LLVM_COV" show \
  -format=html \
  -output-dir=coverage_html \
  -instr-profile=libc_full.profdata \
  "${TEST_BINS[0]}" "${OBJECT_FLAGS[@]}" \
  --show-directory-coverage \
  --show-branches=count \
  -ignore-filename-regex=".*(test|utils).*"

# Open dashboard in browser
xdg-open coverage_html/index.html
```

---

## Modified Condition / Decision Coverage (MC/DC)

MC/DC evaluates boolean sub-conditions within compound logical expressions (such
as `if (A && B)`). It verifies that each individual sub-condition evaluates to
both true and false and independently affects the outcome of the enclosing
decision.

### 1. Clean Prior Profile Artifacts

Removes previous MC/DC profile counters and profile data:

```bash
rm -f build-cov-mcdc/libc_cov_*.profraw libc_cov_*.profraw profraw_list.txt libc_mcdc.profdata
```

### 2. Configure Full-Build Coverage with MC/DC using Clang 24

Configures CMake with `-DLIBC_ENABLE_COVERAGE_MCDC=ON` alongside profiling
flags using Clang 24 and LLD:

```bash
cmake -G Ninja -S runtimes -B build-cov-mcdc \
  -DCMAKE_C_COMPILER="$PWD/build-clang/bin/clang" \
  -DCMAKE_CXX_COMPILER="$PWD/build-clang/bin/clang++" \
  -DLLVM_USE_LINKER=lld \
  -DCMAKE_BUILD_TYPE=Debug \
  -DLLVM_ENABLE_RUNTIMES="libc" \
  -DLLVM_LIBC_FULL_BUILD=ON \
  -DLIBC_ENABLE_COVERAGE=ON \
  -DLIBC_ENABLE_COVERAGE_MCDC=ON
```

### 3. Export Environment Variables

Exports the profile output pattern and Clang 24 tool paths:

```bash
export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
export LLVM_PROFDATA="$PWD/build-clang/bin/llvm-profdata"
export LLVM_COV="$PWD/build-clang/bin/llvm-cov"
```

### 4. Build and Execute Hermetic Tests

Compiles and executes test executables in parallel with MC/DC instrumentation
enabled:

```bash
ninja -k 0 -C build-cov-mcdc libc-hermetic-tests
```

### 5. Merge Profile Counters

Indexes and merges all MC/DC `.profraw` files into a unified
`libc_mcdc.profdata` archive for report generation:

```bash
find build-cov-mcdc -name "libc_cov_*.profraw" > profraw_list.txt
"$LLVM_PROFDATA" merge -sparse -f profraw_list.txt -o libc_mcdc.profdata
```

### 6. Generate MC/DC Coverage Reports

Maps MC/DC bitmap records to source AST decisions and evaluates condition
independence pairs:

```bash
TEST_BINS=($(find build-cov-mcdc -type f -executable -name "*__build__"))
OBJECT_FLAGS=()
for bin in "${TEST_BINS[@]:1}"; do
  OBJECT_FLAGS+=("-object=$bin")
done
```

Reports can be generated in two formats depending on your needs:

#### Option 1: Terminal Summary Report

Displays the terminal coverage summary including MC/DC Condition and Missed
Condition percentages:

```bash
"$LLVM_COV" report \
  -instr-profile=libc_mcdc.profdata \
  "${TEST_BINS[0]}" "${OBJECT_FLAGS[@]}" \
  --show-branch-summary \
  --show-mcdc-summary \
  -ignore-filename-regex=".*(test|utils).*"
```

#### Option 2: Interactive HTML Dashboard

Produces an HTML report with expandable MC/DC decision truth tables and test
vector coverage breakdowns:

```bash
"$LLVM_COV" show \
  -format=html \
  -output-dir=coverage_mcdc_html \
  -instr-profile=libc_mcdc.profdata \
  "${TEST_BINS[0]}" "${OBJECT_FLAGS[@]}" \
  --show-directory-coverage \
  --show-branches=count \
  --show-mcdc \
  --show-mcdc-summary \
  -ignore-filename-regex=".*(test|utils).*"

# Open dashboard in browser
xdg-open coverage_mcdc_html/index.html
```

---

## Running Coverage for a Single Test

When developing or modifying a specific function, coverage can be collected for
a single hermetic test without building and executing the entire test suite.

The commands below use `libc.test.src.ctype.isalpha_test` (which tests
`libc/src/ctype/isalpha.cpp`) as an example. You can test any other entrypoint
by substituting the target name and source file path:
* **Target pattern:** `libc.test.<path_to_test>.<test_name>`
  (e.g. `libc.test.src.string.strlen_test`)
* **Source path pattern:** `libc/<path_to_source>/<source_file>.cpp`
  (e.g. `libc/src/string/strlen.cpp`)

### 1. Clean Prior Profile Artifacts

Removes previously generated raw profile counters:

```bash
rm -f libc_cov_*.profraw profraw_list.txt libc_single.profdata
```

### 2. Build and Execute the Targeted Test

Compiles and runs only the specified test binary, immediately writing execution
profile counters to disk upon completion:

```bash
export LLVM_PROFILE_FILE="libc_cov_%p.profraw"

# Standard coverage build
ninja -C build-cov libc.test.src.ctype.isalpha_test

# MC/DC coverage build
ninja -C build-cov-mcdc libc.test.src.ctype.isalpha_test
```

### 3. Merge the Profile

Merges the single test's raw profile into an indexed database:

```bash
find build-cov/ build-cov-mcdc/ \
  -name "libc_cov_*.profraw" 2>/dev/null > profraw_list.txt
"$LLVM_PROFDATA" merge -sparse -f profraw_list.txt -o libc_single.profdata
```

### 4. View Coverage Reports

#### Option 1: Summary Table Report

```bash
BIN_DIR="build-cov/libc/test/src/ctype"
"$LLVM_COV" report \
  -instr-profile=libc_single.profdata \
  "$BIN_DIR/libc.test.src.ctype.isalpha_test.__build__" \
  libc/src/ctype/isalpha.cpp
```

#### Option 2: Line-by-Line & Truth Table View

```bash
BIN_DIR="build-cov-mcdc/libc/test/src/ctype"
"$LLVM_COV" show \
  -instr-profile=libc_single.profdata \
  "$BIN_DIR/libc.test.src.ctype.isalpha_test.__build__" \
  --show-branches=count \
  --show-mcdc \
  libc/src/ctype/isalpha.cpp
```

---

## Interpreting Results

For detailed documentation on the LLVM coverage reporting format, refer to the
[official Clang Source-Based Code Coverage documentation][clang-coverage].

[clang-coverage]:
  https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#interpreting-reports

### Coverage Metrics Overview

* **Line Coverage:** Measures whether each physical line of executable source
  code was reached at least once during testing.
* **Branch Coverage:** Measures whether each conditional branch evaluated to
  both its `True` and `False` paths. For example, if an `if (x > 0)` branch is
  taken 10 times but never skipped, branch coverage is 50% because the `False`
  path was never exercised.
* **MC/DC Coverage:** Evaluates compound boolean expressions (such as
  `if (A && B)` or `if (A || B)`). It verifies that each individual condition
  was tested as both True and False, and demonstrated that it could
  independently change the overall outcome of the decision.

### Interpreting Reports

The summary table produced by `llvm-cov report` displays metrics across
individual source files and overall totals:

* **Regions / Missed Regions:** A region is a continuous segment of code (such
  as a function body or basic block). Missed regions indicate code blocks that
  were never executed.
* **Functions / Missed Functions:** The total number of entrypoints or
  subroutines executed vs unexecuted.
* **Lines / Missed Lines:** Physical source lines executed vs unexecuted.
* **Branches / Missed Branches:** The total count of decision directions (both
  True and False) evaluated.
* **MC/DC Conditions / Missed Conditions:** The count of individual boolean
  sub-conditions that demonstrated independent decision control.

### Interpreting MC/DC Truth Tables

When inspecting with `--show-mcdc`, `llvm-cov` displays an MC/DC analysis table
beneath each compound decision. For instance, consider the following decision:

```text
   19|  if (c < 0 || c > cpp::numeric_limits<unsigned char>::max())
  ------------------------------------------------------------------
  | Conditions: C1 = (c < 0)
  |             C2 = (c > cpp::numeric_limits<unsigned char>::max())
  |
  | Executed Test Vectors:
  |    C1, C2    Result
  | 1 { F,  F  = F      }  (tested with c = 'a')
  | 2 { T,  -  = T      }  (tested with c = -1)
  |
  | C1-Pair: covered (1, 2)
  | C2-Pair: not covered
  | MC/DC Coverage: 50.00%
  ------------------------------------------------------------------
```

* **Conditions:** **C1** represents `c < 0` and **C2** represents
  `c > cpp::numeric_limits<unsigned char>::max()`.
* **Executed Vectors:**
  * **Vector 1 (`F, F = F`):** Tested with a valid character (`c = 'a'`). Both
    C1 and C2 evaluated False, producing an overall False result.
  * **Vector 2 (`T, - = T`):** Tested with a negative value (`c = -1`). C1
    evaluated True, producing an overall True result. The hyphen (`-`)
    indicates C2 was short-circuited and not evaluated.
* **Condition Pairs:**
  * **`C1-Pair: covered (1, 2)`:** Comparing Vector 1 and Vector 2 proves that
    changing C1 from False to True directly flipped the result from False to
    True. C1 is fully covered.
  * **`C2-Pair: not covered`:** C2 was never tested in a state where it
    independently turned the result True while C1 was False.
* **Reaching 100% Coverage:** Add a test with a value above 255 (`c = 256`).
  This executes Vector 3 (`F, T = T`), forming the independence pair `(1, 3)`
  for C2 and reaching 100% MC/DC coverage.