LLVM Test Suite Guide

The LLVM test suite contains two major categories of tests: code -fragments and whole programs. Code fragments are in the llvm module -under the llvm/test directory. The whole programs -test suite is in the llvm-test module under the main directory.

The LLVM testing infrastructure contains two major categories of tests: code +fragments and whole programs. Code fragments are referred to as the "DejaGNU +tests" and are in the llvm module in subversion under the +llvm/test directory. The whole programs tests are referred to as the +"Test suite" and are in the test-suite module in subversion. +

The llvm-test suite contains whole programs, which are pieces of +

The test suite contains whole programs, which are pieces of code which can be compiled and linked into a stand-alone program that can be executed. These programs are generally written in high level languages such as C or C++, but sometimes they are written straight in LLVM assembly.

@@ -175,27 +175,26 @@ a way of benchmarking LLVM performance, both in terms of the efficiency of the programs generated as well as the speed with which LLVM compiles, optimizes, and generates code.

All "whole program" tests are located in the test-suite Subversion -module.

The test-suite is located in the test-suite Subversion module.

The tests are located in two separate Subversion modules. The code fragment +

The tests are located in two separate Subversion modules. The DejaGNU tests are in the main "llvm" module under the directory llvm/test (so you get these tests for free with the main llvm tree). - The more comprehensive llvm-test suite that includes whole -programs in C and C++ is in the test-suite module. This module should -be checked out to the llvm/projects directory as llvm-test (don't use -another name, for then the test suite will be run every time you run -make in the main llvm directory). + The more comprehensive test suite that includes whole +programs in C and C++ is in the test-suite module. This module should +be checked out to the llvm/projects directory (don't use another name +then the default "test-suite", for then the test suite will be run every time +you run make in the main llvm directory). When you configure the llvm module, -the llvm-test directory will be automatically configured. +the test-suite directory will be automatically configured. Alternatively, you can configure the test-suite module manually.

@@ -218,8 +217,8 @@ Alternatively, you can configure the test-suite module manually.

To run only a subdirectory of tests in llvm/test using DejaGNU (ie. -Regression/Transforms), just set the TESTSUITE variable to the path of the +

To run only a subdirectory of tests in llvm/test using DejaGNU (ie. +Transforms), just set the TESTSUITE variable to the path of the subdirectory (relative to llvm/test):

@@ -232,8 +231,8 @@ subdirectory (relative to llvm/test):

must have run the complete testsuite before you can specify a subdirectory.

To run only a single test, set TESTONE to its path (relative to -llvm/test) and make the check-one target:

To run only a single test, set TESTONE to its path (relative to +llvm/test) and make the check-one target:

@@ -242,7 +241,7 @@ subdirectory.

llvm-test tests

Test suite

To run the comprehensive test suite (tests that compile and execute whole @@ -251,7 +250,7 @@ programs), first checkout and setup the test-suite module:

 % cd llvm/projects
-% svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
+% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
 % cd ..
 % ./configure --with-llvmgccdir=$LLVM_GCC_DIR

@@ -259,12 +258,12 @@ programs), first checkout and setup the test-suite module:

llvm-gcc, not it's src or obj dir.

Then, run the entire test suite by running make in the llvm-test +

Then, run the entire test suite by running make in the test-suite directory:

-% cd projects/llvm-test
+% cd projects/test-suite
 % gmake

@@ -274,19 +273,19 @@ let it generate a report by running:

-% cd projects/llvm-test
+% cd projects/test-suite
 % gmake TEST=nightly report report.html

Any of the above commands can also be run in a subdirectory of -projects/llvm-test to run the specified test only on the programs in +projects/test-suite to run the specified test only on the programs in that subdirectory.

llvm-test +

Test suite Structure

@@ -684,7 +683,7 @@ others are features that we haven't added yet (or may never add). In DejaGNU, the result for such tests will be XFAIL (eXpected FAILure). In this way, you can tell the difference between an expected and unexpected failure.

The tests in llvm-test have no such feature at this time. If the +

The tests in the test suite have no such feature at this time. If the test passes, only warnings and other miscellaneous output will be generated. If a test fails, a large <program> FAILED message will be displayed. This will help you separate benign warnings from actual test failures.

@@ -692,7 +691,7 @@ will help you separate benign warnings from actual test failures.

@@ -701,7 +700,7 @@ will help you separate benign warnings from actual test failures.

are not executed inside of the LLVM source tree. This is because the test suite creates temporary files during execution.

To run the llvm-test suite, you need to use the following steps:

To run the test suite, you need to use the following steps:

cd into the llvm/projects directory

-% svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
+% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite

This will get the test suite into llvm/projects/llvm-test

Configure the test suite using llvm configure. This will automatically configure llvm-test. - You must do it from the top level otherwise llvm-gcc will not be set which is required to - run llvm-test:
+

Configure the test suite using llvm configure. This will automatically + configure test-suite. You must do it from the top level otherwise llvm-gcc + will not be set which is required to run llvm-test:

 % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure --with-llvmgccdir=$LLVM_GCC_DIR
@@ -728,7 +727,7 @@ test suite creates temporary files during execution.
     installed llvm-gcc, not its src or obj directory.

Change back to the llvm/projects/llvm-test directory you created before +
Change back to the llvm/projects/test-suite directory you created before and run gmake (or just "make" on systems where GNU make is the default, such as linux.

@@ -738,11 +737,11 @@ the test code or configure script changes).

-Configuring external tests

+Configuring external tests

In addition to the regular "whole program" tests, the llvm-test +

In addition to the regular "whole program" tests, the test-suite module also provides a mechanism for compiling the programs in different ways. -If the variable TEST is defined on the gmake command line, the test system will +If the variable TEST is defined on the gmake command line, the test system will include a Makefile named TEST.<value of TEST variable>.Makefile. This Makefile can modify build rules to yield different results.

@@ -797,7 +796,7 @@ LLVM.

-Generating test output

+Generating test output

Assuming you can run llvm-test, (e.g. "gmake TEST=nightly report" +

Assuming you can run the test suite, (e.g. "gmake TEST=nightly report" should work), it is really easy to run optimizations or code generator components against every program in the tree, collecting statistics or running custom checks for correctness. At base, this is how the nightly tester works, @@ -846,10 +845,10 @@ will tally counts of things you care about.

Following this, you can set up a test and a report that collects these and formats them for easy viewing. This consists of two files, an -"llvm-test/TEST.XXX.Makefile" fragment (where XXX is the name of your +"test-suite/TEST.XXX.Makefile" fragment (where XXX is the name of your test) and an "llvm-test/TEST.XXX.report" file that indicates how to format the output into a table. There are many example reports of various -levels of sophistication included with llvm-test, and the framework is very +levels of sophistication included with the test suite, and the framework is very general.

If you are interested in testing an optimization pass, check out the @@ -857,7 +856,7 @@ general.

-% cd llvm/projects/llvm-test/MultiSource/Benchmarks  # or some other level
+% cd llvm/projects/test-suite/MultiSource/Benchmarks  # or some other level
 % make TEST=libcalls report

@@ -888,7 +887,7 @@ Prolangs-C/bison/mybison | 74 | * | You can also use the "TEST=libcalls report.html" target to get the table in HTML form, similarly for report.csv and report.tex.

The source for this is in llvm-test/TEST.libcalls.*. The format is pretty +

The source for this is in test-suite/TEST.libcalls.*. The format is pretty simple: the Makefile indicates how to run the test (in this case, "opt -simplify-libcalls -stats"), and the report contains one line for each column of the output. The first value is the header for the column and the -- cgit v1.1