docs/importing_data.rst

.. _importing_data:

Importing Data
==============

Importing Data in a Text File
-----------------------------

The ``lnt importreport`` command will import data in a simple text file format. The
command takes a space separated key value file and creates an LNT report file,
which can be submitted to a LNT server.  Example input file::

    foo.execution_time 123
    bar.size 456
    foo/bar/baz.size 789

The format is ``test-name.metric value``, so ``execution_time`` and ``size`` must be valid
metrics for the test suite you are submitting to.

Example::

    echo -n "foo.execution_time 25\nbar.score 24.2\nbar/baz.size 110.0\n" > results.txt
    lnt importreport --machine=my-machine-name --order=1234 --testsuite=nts results.txt report.json
    lnt submit http://mylnt.com/db_default/submitRun report.json

Additional information can also be included in runs. This can be useful to include e.g. the
commit information associated to the version of the code being benchmarked or other information
relevant to the run like the system load, etc. This information can be provided to ``lnt importreport``
as key-value pairs, passing ``--run-info`` as many times as necessary::

    lnt importreport --machine=my-machine-name --order=1234 --testsuite=nts \
                     --run-info commit="COMMIT INFORMATION"                 \
                     --run-info machine_load="LOAD INFORMATION"             \
                     results.txt report.json

.. _json_format:

LNT Report File Format
----------------------

The ``lnt importreport`` tool is an easy way to import data into LNT's test format.
You can also create LNTs report data directly for additional flexibility.

First, make sure you've understood the underlying :ref:`concepts` used by LNT.

.. code-block:: none

  {
      "format_version": "2",
      "machine": {
          "name": _String_      // machine name, mandatory
          (_String_: _String_)* // optional extra info
      },
      "run": {
          ("start_time": "%Y-%m-%dT%H:%M:%S",)? // optional, ISO8061 timestamp
          ("end_time": "%Y-%m-%dT%H:%M:%S",)?   // optional, ISO8061 timestamp, can equal start_time if not known.
          (_String_: _String_,)* // optional extra info about the run.
          // At least one of the extra fields is used as ordering and is
          // mandatory. For the 'nts' and 'Compile' schemas this is the
          // 'llvm_project_revision' field.
      },
      "tests": [
          {
              "name": _String_,   // test name mandatory
              (_String_: _Data_)* // List of metrics, _Data_ allows:
                                  // number, string or list of numbers
          }+
      ]
  }

Any optional fields provided in the ``run`` section will be associated to that run and visible
in the UI when looking at that run. This allows annotating runs with useful additional information,
such as the commit information related to this run or similar. This data will also be visible on
charts when viewing historical results. Arbitrary data can be provided, however including too much
run-related information can cause very large amounts of data to have to be transfered to display
graphs.

A concrete small example is

.. literalinclude:: report-example.json
    :language: json


Given how simple it is to make your own results and send them to LNT, it is common to
not use the LNT client application at all, and just have a custom script run your tests
and submit the data to the LNT server in JSON format.


.. _nts_suite:

Default Test Suite (NTS)
------------------------

The default test-suite schema is called NTS. It was originally designed for
nightly test runs of the llvm test-suite. However it should fit many other
benchmark suites as well. The following metrics are supported for a test:

* ``execution_time``: Execution time in seconds; lower is better.
* ``score``: Benchmarking score; higher is better.
* ``compile_time``: Compiling time in seconds; lower is better.
* ``hash``: A string with the executable hash (usually md5sum of the stripped binary)
* ``mem_bytes``: Memory usage in bytes during execution; lower is better.
* ``code_size``: Code size (usually the size of the text segment) in bytes;
  lower is better.
* ``execution_status``: A non zero value represents an execution failure.
* ``compile_status``: A non zero value represents a compilation failure.
* ``hash_status``: A non zero value represents a failure computing the
  executable hash.

The `run` information is expected to contain this:

* ``llvm_project_revision``: The revision or version of the compiler
  used for the tests. Used to sort runs.

.. _custom_testsuites:

Custom Test Suites
------------------

LNT test suite schemas define which metrics can be tracked for a test and what
extra information is known about runs and machines.  You can define your own
test suite schemas in a yaml file. The LNT administrator can upload the schema
to the server using the ``lnt admin test-suite add`` command (see :ref:`admin_tools`).

Example:

.. literalinclude:: my_suite.yaml
    :language: yaml

* LNT currently supports the following metric types:

  - ``Real``: 8-byte IEEE floating point values.
  - ``Hash``: String values; limited to 256, sqlite is not enforcing the limit.
  - ``Status``: StatusKind enum values (limited to 'PASS', 'FAIL', 'XFAIL' right
    now).

* You need to mark at least 1 of the run fields as ``order: true`` so LNT knows
  how to sort runs.
* Note that runs are not be limited to the fields defined in the schema for
  the run and machine information. The fields in the schema merely declare which
  keys get their own column in the database and a prefered treatment in the UI.