hlvm/docs/DevelopersGuide.html

<!--#set var="title" value="Developer's Guide" -->
<!--#include virtual="/incl/header.incl" -->
<h1 class="title">HLVM Developer's Guide</h1>
<div class="warning">CAUTION: This document is a work in progress.</div>
<ol>
  <li><a href="#intro">Introduction</a></li>
  <li><a href="#build">Build System</a>
    <ol>
      <li><a href="#scons">About SCONS</a></li>
      <li><a href="#config">Configuring</a></li>
      <li><a href="#rules">Build Rules</a></li>
      <li><a href="#sanity">Sane Build Environment</a></li>
    </ol>
  </li>
  <li><a href="#style">Coding Style</a></li>
</ol>

<div class="author">
  <p>Author: <a href="mailto:rspencer@reidspencer.com">Reid Spencer</a>.</p>
</div>

<h2><a name="intro">Introduction</a></h2>
<div class="text">
  <p>This document is a manual for developers who wish to contribute to the HLVM
  project. It covers topics such as coding style, the build system and project 
  rules. All the <a href="http://llvm.org/docs/DeveloperPolicy.html">policies 
    of the LLVM project</a> apply here as well. This document just contains
  things specific to HLVM.</p>
</div>

<h2><a name="build">Build System</a></h2>
<div class="text">
  <p>This section describes the HLVM build system.</p>
</div>

<h3><a name="scons">About SCONS</a></h3>
<div class="text">
  <p>The HLVM project uses the <a href="http://www.scons.org/">scons</a>
  software construction tool to orchestrate its builds. This gives us superior
  dependency checking and a much more flexible tool for developing the build
  system. We started with "make" but were convinced of SCons superiority after
  trying it for a week. If you're not familiar with it, please read up on it
  and get a general understanding. You don't need detailed understanding because
  all you will most likely need to do is follow the instructions in this
  section.</p>
</div>

<h3><a name="config">Configuring</a></h3>
<div class="text">
  <h4>Configuration On Every Build</h4>
  <p>Unlike some other build systems (e.g. autoconf/automake), the
  configuration and construction phases of building are not separated with
  scons. The configuration parameters are checked every time you build. While
  you might think that will slow things down, it doesn't. The configuration
  information is cached and proceeds quite quickly once your configuration is
  stable. If something changes in your environment, only those pieces affected
  will get re-configured. This saves a lot of time and hassle by telling you of
  any configuration errors on the very next build.</p>
  <h4>The Options Cache</h4>
  <p>Configuration parameters are specified on the scons command line. The
  values of these parameters are stored in the <i>options cache</i> which is
  simply a file named .options_cache in the root source directory. If the file
  doesn't exist, it will be created the first time you run scons. On subsequent
  runs, the values are loaded from this cache and used unless new values are
  given on the command line. In this way, you only need to specify your options
  once and then they "stick" from that point forward.</p>
  <h4>Configuration Options</h4>
  <p>HLVM has a variety of options to assist with configuring HLVM for your
  environment and build choices. Each option has a default that will work in
  many environments but might need adjusting for your environment. The default
  assumes various software packages are in <tt>/usr/local</tt> and that a fully
  debuggable version of HLVM should be built. The table below provides the 
  full list of options.</p>
  <table>
  <tr><th>Option</th><th>Description</th><th>Default</th></tr>
  <tr>
    <td>assertions</td>
    <td>Include assertions in the code. This can be enabled or disabled separate
    from the setting for debug.</td>
    <td>1</td></tr>
  <tr>
    <td>debug</td>
    <td>Build with debug options turned on. This includes #ifdef/#endif code as
      well as instructing the compiler to include debug information (-g)</td>
    <td>1</td></tr>
  <tr>
    <td>inline</td>
    <td>Cause inline code to be inline. This causes inline functions to really
      be inline. Its off by default for better debuggability.</td><td>0</td>
  </tr>
  <tr>
    <td>optimize</td>
    <td>Build object files with optimization. This turns on compiler
      optimization (-O3) to make the code efficient.</td><td>0</td></tr>
  <tr>
    <td>profile</td>
    <td>Generate profiling aware code. This causes the compiler to construct the
      code so that it can be profiled (-pg on Unix).</td><td>0</td></tr>
  <tr>
    <td>small</td>
    <td>Generate smaller code rather than faster. Instructs the code to favor
      smaller object files and executables than faster ones. Might be useful for
      memory constrained platforms.</td>
    <td>0</td></tr>
  <tr>
    <td>strip</td>
    <td>Strip executables of their symbols. This can significantly reduce the
      size of the executable. Useful for release preparation when trying to
      minimize the size of a tarball or RPM package.</td>
    <td>0</td></tr>
  <tr>
    <td>prefix</td>
    <td>Specify where to install HLVM. This is the root directory of where HLVM
      will be installed when you run <tt>scons install</tt></td>
    <td>/usr/local</td></tr>
  <tr>
    <td>confpath</td>
    <td>Specify additional configuration dirs to search. This is a unix style
      path (colon separate) of directories to include in the search for software
      packages. These directories will be searched first. If you put your
      software in a non-standard place like /proj (Reid) or /more (Saem) then
      you'll want to use this option.</td>
    <td>{empty}</td></tr>
  <tr>
    <td>with_llvm</td>
    <td>Specify where LLVM is installed.</td>
    <td>/usr/local</td></tr>
  <tr>
    <td>with_apr</td>
    <td>Specify where apr is installed.</td>
    <td>/usr/local/apr</td></tr>
  <tr>
    <td>with_apru</td>
    <td>Specify where apr-utils is installed.</td>
    <td>/usr/local/apr</td></tr>
  <tr>
    <td>with_libxml2</td>
    <td>Specify where LibXml2 is installed.</td>
    <td>/usr/local</td></tr>
  <tr>
    <td>with_gperf</td>
    <td>Specify where the gperf program is installed.</td>
    <td>/usr/local/bin/gperf</td></tr>
  <tr>
    <td>with_llc</td>
    <td>Specify where the LLVM compiler is installed.</td>
    <td>/usr/local/bin/llc</td></tr>
  <tr>
    <td>with_llvmdis</td>
    <td>Specify where the LLVM disassembler is installed.</td>
    <td>/usr/local/bin/llvm-dis</td></tr>
  <tr>
    <td>with_llvmas</td>
    <td>Specify where the LLVM assembler is installed.</td>
    <td>/usr/local/bin/llvm-as</td></tr>
  <tr>
    <td>with_llvmgcc</td>
    <td>Specify where the LLVM C compiler is installed.</td>
    <td>/usr/local/bin/llvm-gcc</td></tr>
  <tr>
    <td>with_llvmgxx</td>
    <td>Specify where the LLVM C++ compiler is installed.</td>
    <td>/usr/local/bin/llvm-g++</td></tr>
  <tr>
    <td>with_llvmar</td>
    <td>Specify where the LLVM bytecode archiver is installed.</td>
    <td>/usr/local/bin/llvm-g++</td></tr>
  <tr>
    <td>with_llvm2cpp</td>
    <td>Specify where the LLVM llvm2cpp program is installed.</td>
    <td>/usr/local/bin/llvm2cpp</td></tr>
  <tr>
    <td>with_runtest</td>
    <td>Specify where DejaGnu runtest program is installed.</td>
    <td>/usr/local/bin/runtest</td></tr>
  <tr>
    <td>with_doxygen</td>
    <td>Specify where the doxygen program is installed.</td>
    <td>/usr/local/bin/doxygen</td></tr>
  <tr>
    <td>with_xsltproc</td>
    <td>Specify where the XSLT processor is installed.</td>
    <td>/usr/local/bin/xsltproc</td>
  </tr>
  </table>
  <h4>Configuration Prompts</h4>
  <p>When you first run <tt>scons</tt> against HLVM, if you did not specify the
  <tt>confpath</tt> option or the configuration code cannot otherwise find a
  package it needs, you will be prompted to enter the applicable path names
  manually. You only need to do this once as the values you enter will be
  remembered in the <tt>.options_cache</tt> file. These prompts will repeat
  until you enter a path in which the package is properly recognized. If you
  wish to abort this, just enter <tt>exit</tt> or <tt>quit</tt> at any prompt.
  </p>
</div>

<h3><a name="rules">Build Rules</a></h3>
<div class="text">
  <p>This subsection describes the various build rules and extensions to SCons
  that HLVM uses.</p>
  <h4>The <tt>build</tt> python module</h4>
  <p>For convenience, and to unclutter the SConscript files, the top level
  <tt>build</tt> directory contains a python module that provides the build
  facilities for HLVM. This module manipulates an SCons environment to set the
  variables and define the builder rules for HLVM. The public interface to this
  library is in the <tt>hlvm.py</tt> module. No other module should be imported
  into the SConscript files.</p>
  <h4>HLVM Builder Rules</h4>
  <p>HLVM requires some specialized build rules beyond what SCons provides.
  While general construction of static libraries, shared libraries, and
  executables use the standard SCons builders, several more are defined in the
  <tt>build</tt> python module. The table below describes the HLVM specific
  builders.</p>
  <table>
    <tr><th>Builder</th><th>Source</th><th>Target</th><th>Description</th></tr>
    <tr>
      <td>RNGTokenizer</td>
      <td>Relax/NG Schema (.rng)</td>
      <td>*Tokenizer.h, *Tokenizer.cpp, *TokenHash.cpp</td>
      <td>This builder scans a Relax/NG Schema for element, attribute, and
        enumerated value names. It then uses these tokens to build a perfect
        hash function (via <tt>gperf</tt>) for quickly recognizing the tokens
        and converting it to an enumeration value. This is provided to make
        recognizing the element and attributes names of an XML document faster.
      </td>
    </tr>
    <tr>
      <td>RNGQuoteSource</td>
      <td>Relax/NG Schema (.rng)</td>
      <td>C++ Strings For Inclusion (.inc)</td>
      <td>This builder converts a Relax/NG schema into a set of strings that can
        be #included into a C++ program as a literal string. It strips out
        comments, annotations, and converts special characters into the C++
        equivalent. The purpose here is to allow a single source for the schema
        to be incorporated directly into the software so that no file I/O is
        necessaroy to access the schema.</td>
    </tr>
    <tr>
      <td>Bytecode</td>
      <td>C++ Source (.cpp)</td>
      <td>LLVM Bytecode (.bc)</td>
      <td>This builder compiles a C++ source file and produces an equivalent 
        LLVM bytecode file. This allows us to incorporate C++ source into the 
        Runtime as linkable bytecode.</td>
    </tr>
    <tr>
      <td>BytecodeArchive</td>
      <td>LLVM Bytecode (.bc)</td>
      <td>LLVM Bytecode Archive (.bca)</td>
      <td>This builder collects a group of LLVM Bytecode files together and
        creates a bytecode archive from them using <tt>llvm-ar</tt></td>
    </tr>
    <tr>
      <td>Cpp2LLVMCpp</td>
      <td>C++ Source (.cxx)</td>
      <td>C++ Source (.cpp)</td>
      <td>This builder uses the <tt>llvm2cpp</tt> to turn source C++ into
        equivalent C++ calls against the LLVM Intermediate Representation (IR)
        to construct the same program or program fragment. The construction
        environment variable LLVM2CPPFLAGS can be set to control what options
        are passed to <tt>llvm2cpp</tt>. See the <tt>llvm2cpp</tt> manual page
        for details. Examples are in the <tt>hlvm/CodeGen/SConscript</tt>
        file.</td>
    </tr>
    <tr>
      <td>Check</td>
      <td>n/a</td>
      <td>n/a</td>
      <td>This builder is a little special. It is used to run the DejaGnu test
        suite. It is aliased on the command line as the target "check" so that
        the <tt>scons check</tt> command will invoke this build.</td>
    </tr>
    <tr>
      <td>Doxygen</td>
      <td>n/a</td>
      <td>doxygen.tar.gz</td>
      <td>This builder runs the doxygen documentation generator on the source
        code and builds a gzipped tar file containing doxygen's output. This
        allows the HLVM API documentation, generated from source code comments,
        to be packged into a single file.</td>
    </tr>
  </table>
</div>

<h3><a name="sanity">A Sane Build Environment</a></h3>
<div class="text">
  <p>Building HLVM is no small feat. It depends on a lot of software that is
  quite version dependent. To bring a little sanity to the process, here is a
  step-by-step procedure we know to work.</p>
  <h4>Build Separation</h4>
  <p>In building HLVM, you'll be installing compilers and library that may
  already exist on your system. You don't want to overwrite your system versions
  of these things or it will wreak havoc on your system. So, we suggest that you
  start with a fresh directory. In the discussion that follows, we'll call it
  <tt>/proj</tt> (that's what Reid uses).  But, it could be anything you want, 
  as long as its new.  <tt>~/hlvm</tt> would work just as well.</p>
  <p<em class="warning">IMPORTANT: </em>Choosing the disk location for these
  builds should not be taken lightly. You will need upwards of 40GB of storage 
  on a 32-bit machine to build all this software. Don't assume your home 
  directory has enough space!</p>
  <p>Once you've found a suitable location for HLVM, create the following
  directory structure:</p><pre>
  cd /proj
  mkdir gcc libxml2 apr apru gperf scons install</pre>
  <p>In the following sections you will build each of these packages and install
  them into <tt>/proj/install</tt> which will keep it separate from anything
  else in your system.</p>
  <h4>Build GCC 3.4.6</h4>
  <p>First, start with obtaining GCC 3.4.6. This will be the compiler that you
  use for all the remaining compilations. Use the following commands to obtain,
  build and install GCC 3.4.6:</p><pre>
  cd /proj/gcc
  mkdir build
  svn co svn://gcc.gnu.org/svn/gcc/tags/gcc_3_4_6_release src
  cd ../build
  ../src/configure  --prefix=/proj/install
  make 
  make install</pre>
  <h4>Set Environment</h4>
  <p>Now that you have gcc installed in a separate location, you will need to
  change your environment to ensure that that version of gcc is the one used in
  subsequent builds. Details vary from platform to platform, but on Linux, the
  following should work:</p><pre>
  export PATH=/proj/install/bin:$PATH
  export LD_LIBRARY_PATH=/proj/install/lib:$LD_LIBRARY_PATH</pre>
  <p>The essential point is to change your environment so that programs and
  libraries installed into /proj/install will be found first. You should do this
  in any shell environment in which you'll be building HLVM related software.
  </p>
  <h4>Build gperf</h4>
  <p>This package is used for generating perfect hash functions. It is used by
  HLVM for fast recognition of XML element and attribute names. Its easy and
  fast to build:</p><pre>
  cd /proj/gperf
  wget http://mirrors.kernel.org/gnu/gperf/gperf-2.7.2.tar.gz
  tar zxf gperf-2.7.2.tar.gz
  mkdir build
  cd build
  ../gperf-2.7.2/configure --prefix=/proj/install
  make
  make install</pre>
  <p>You can probably use any version after 2.7. We know it works with 2.7.2 and
  3.0.1.</p>
  <h4>Build libxml2</h4>
  <p>This package provides all XML services for HLVM. It is part of GNome and
  many other packages and quite stable. It should build quickly and easily for
  you. Use these commands:</p><pre>
  cd /proj/libxml2
  wget ftp://xmlsoft.org/libxml2/libxml2-2.6.24.tar.gz
  tar zxf libxml2-2.6.24.tar.gz
  mkdir build
  cd build
  ../libxml2-2.6.24/configure --prefix=/proj/install
  make
  make install</pre>
  <h4>Build apr</h4>
  <p>The Apache Portable Runtime is a portability layer used within the Apache
  HTTP Server. Although it is still undergoing active development, stable
  releases are available. HLVM uses APR for portability in the runtime. Build 
  APR with the following commands:</p>
  <pre>
  cd /proj/apr
  wget http://mirror.olnevhost.net/pub/apache/apr/apr-1.2.7.tar.gz
  tar zxf apr-1.2.7.tar.gz
  mkdir build
  cd apr-1.2.7
  ./buildconf
  cd ../build
  ../apr-1.2.7/configure --prefix=/proj/install --enable-debug \
    --enable-threads --enable-other-child
  make
  make install</pre>
  <h4>Build apr-util</h4>
  <p>The apr-util package is some additional utilities that go with APR. Build 
  apr-util with the following commands:</p>
  <pre>
  cd /proj/apru
  wget http://mirror.olnevhost.net/pub/apache/apr/apr-util-1.2.7.tar.gz
  tar zxf apr-1.2.7.tar.gz
  mkdir build
  cd apr-util-1.2.7
  ./buildconf
  cd ../build
  ../apr-util-1.2.7/configure --prefix=/proj/install --enable-debug \
    --enable-threads --enable-other-child
  make
  make install</pre>
  <h4>Build LLVM</h4>
  <p>For now, you must build LLVM from the CVS repository. Although LLVM is
  actively being developed, it is generally stable and this is safe. If you get
  tempted to use a release tarball, it will fail. HLVM depends on post-1.7
  features of LLVM. When you build LLVM, use the "tools-only" target. This will
  avoid building the "runtime" portion of LLVM which was necessary for an older
  version of llvm-gcc (v3). Since we'll be using llvm-gcc4, this is unnecessary
  and will eliminate some chicken-and-egg type problems.</p>
  <pre>
  cd /proj/llvm
  cvs -d :pserver:anon@llvm-cvs.cs.uiuc.edu:/var/cvs/llvm login
  &lt;return&gt;
  cvs -z3 -d :pserver:anon@llvm-cvs.cs.uiuc.edu:/var/cvs/llvm co llvm
  mkdir build
  cd build
  ../llvm/configure --prefix=/proj/install
  make tools-only
  make install
  make ENABLE_OPTIMIZED=1 OPTIMIZE_OPTION=-O2 tools-only</pre>
  <h4>Build llvm-gcc4</h4>
  <p>Apple provides a Subversion repository for llvm-gcc4. You need this latest
  version because HLVM depends on some of the features. You might find this one
  a bit tricky. See the README.LLVM file in the top source directory for
  additional help and late breaking news.</p>
  <pre>
  cd /proj/llvm-gcc4
  svn co svn://anonsvn.opensource.apple.com/svn/llvm cfe
  mkdir build install
  cd build
  ../cfe/configure --prefix=/proj/llvm-gcc4/cfe/install \
    --enable-llvm=/proj/llvm/build --enable-languages=c,c++ --disable-threads \
    --program-prefix=llvm-
  make
  make install</pre>
  <p>Note that the <tt>--disable-threads</tt> option is a temporary workaround
  until LLVM's PR822 (supporting weak-external linkage) is implemented. This
  should be fixed by September, 2006 at which time --disable-threads won't be
  needed any more.</p>
  <p><em class="warning">IMPORTANT: </em>You <em>MUST</em> install llvm-gcc4
  into its own installation area. Do <em>NOT</em> be tempted to install it into
  <tt>/proj/install</tt>. If you do, it will overwrite your gcc 3.4.6
  installation and you will have corrupted your build environment. 
  <i>You've been warned.</i></p>
  <h4>Install SCons</h4>
  <p>As noted earlier, HLVM uses the SCons tool for its builds. You need version
  0.96.92. Install it like this:</p>
  <pre>
  cd /proj/scons
  wget http://internap.dl.sourceforge.net/sourceforge/scons/scons-0.96.92.tar.gz
  tar zxf scons-0.96.92.tar.gz
  python config.py install</pre>
  <p>Alternatively, you could use one of the SCons packages, such as:</p>
  <pre>
  cd /proj/scons
  wget http://internap.dl.sourceforge.net/sourceforge/scons/scons-0.96.92-1.noarch.rpm
  rpm --install scons-0.96.92-1.noarch.rpm</pre>

  <h4>Build HLVM</h4>
  <p>Note in the following that if you've actually used <tt>/proj</tt> as your
  build area then you don't need to provide any arguments to <tt>make</tt> as
  these paths are the default. Also note that the arguments are only needed the
  first time you build HLVM. Subsequently, these options will be remembered. See
  the description of SCons above.</p>
  <pre>
  cd /proj/hlvm
  svn co svn://hlvm.org/hlvm hlvm
  cd hlvm
  make debug MYMODE=Debug MYPREFIX=/proj/install MYPATH=/proj/install \
    MYLLVMGCC=/proj/llvm/cfe/install/bin</pre>
</div>

<h2><a name="style">Coding Style</a></h2>
<div class="text">
  <p>Contributions to HLVM must meet the following Coding Style
  requirements:</p>
  <ul>
    <li>Each file must start with the comment section common to other files in
    HLVM. This comment section starts with a line describing the file, followed
    by the HLVM title line, and a statement about the author and the 
    copyright holder (UIUC).</li>
    <li>Source code that is to be part of an HLVM library or executable must be
    written in C++. We know this isn't the greatest language in the world, but 
    it is the most compatible for meeting HLVM's integration and platform 
    support requirements. Its also necessary for integration with LLVM.</li>
    <li>Utility programs should be written in python, especially if they are
    build related.</li>
    <li>Files must not exceed 80 columns in width</li>
    <li>Indentation is 2 spaces (no tabs).</li>
    <li>Doxygen documentation must be provided in header files, at the very
    least.</li>
  </ul>
</div>

<!--#include virtual="/incl/footer.incl" -->