This tutorial is going to walk through creating a TableGen backend using Python.
We are using Python to better fit into a notebook, but backends in LLVM are written in C++. The principles you learn here will still apply and you could port this tutorial to any language that has a JSON parser.
This is the process in LLVM, using a C++ backend:
TableGen source -> llvm-tblgen -> backend (within llvm-tblgen) -> results
This is what we will be doing:
TableGen source -> llvm-tblgen -> JSON -> Python -> results
The backend here is ported from one of several in “SQLGen” which was written by Min-Yih Hsu.
I encourage you to use those resources to supplement this notebook.
Unlike the other tutorial notebooks we are not using the TableGen kernel. This is an iPython notebook and we're going to run llvm-tblgen as a subprocess.
First let's find it, in the same way the TableGen kernel does.
import os import shutil def find_tblgen(): path = os.environ.get("LLVM_TBLGEN_EXECUTABLE") if path is not None and os.path.isfile(path) and os.access(path, os.X_OK): return path else: path = shutil.which("llvm-tblgen") if path is None: raise OSError("llvm-tblgen not found") return path _ = find_tblgen()
If the above cell raises an exception, either put llvm-tblgen on your PATH or point to it using the LLVM_TBLGEN_EXECUTABLE environment variable. Alternatively, edit the code to use whatever path you want.
Then we need to compile some TableGen by passing it to llvm-tblgen's stdin. We will be using the option --dump-json and returning the JSON as a Python dictionary if the compilation succeeds. If it fails, we raise an exception.
import subprocess import tempfile import json def run_tblgen(src): # Passing to stdin requires a file like object. with tempfile.TemporaryFile("w+") as f: f.write(src) f.seek(0) got = subprocess.run( [find_tblgen(), "--dump-json"], stdin=f, stderr=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, ) if got.stderr: raise RuntimeError("llvm-tblgen failed with stderr: " + got.stderr) return json.loads(got.stdout) print(json.dumps(run_tblgen("class Foo {}"), indent=4))
{
"!instanceof": {
"Foo": []
},
"!tablegen_json_version": 1
}
This backend is going to generate SQL queries. The general form of a SQL query is:
SELECT <some field names> FROM <table name> WHERE <conditions> ORDER BY <field tags>;
query_tblgen = """\ def all; def fields; def none; def eq; def ne; def gt; def ge; def and; def or; """
Normally you'd write this to a .td file but here we have it in a Python string to fit into this notebook. We will add to this string to produce the final source.
This section defines some constants. First are the fields we want to get back from the query:
all - Return all fields.fields - Means that we will provide a list of fields we are interested in.The second set are the logical operators for what will become the WHERE clause (called condition in the TableGen). These are string versions of various symbols. For example ne means !=, which in SQL is <>.
Finally none is used to mean there is no condition to the query (no WHERE).
query_tblgen += """\ class Query <string table, dag query_fields = (all), dag condition = (none)> { string TableName = table; dag Fields = query_fields; dag WhereClause = condition; list<string> OrderedBy = []; } """
Then the Query class. Its arguments are:
table - The name of the table to query (FROM <table>).query_fields - The fields you want returned (SELECT <fields>).all meaning return all fields.condition - Logic to select entries (WHERE <conditions>).none meaning there is no condition, or in other words select all entries in the table.full_tblgen = query_tblgen + """\ def : Query<"Customer">; def : Query<"Orders", (fields "Person", "Amount")>; def : Query<"Customer", (fields "Affiliation"), (eq "Name", "Mary Blackburn":$str)>; def : Query<"Orders", (fields "ProductName"), (gt "Amount", 8)>; def : Query<"Orders", (fields "ProductName":$name, "Person"), (and (gt "Amount", 8), (ne "Person", 1))> { let OrderedBy = ["$name"]; } """
Now we can define some queries. Let's go go over the last one in detail.
def : Query<"Orders", (fields "ProductName":$name, "Person"),
(and (gt "Amount", 8), (ne "Person", 1))> {
let OrderedBy = ["$name"];
}
Orders.ProductName and Person.ProductName with $name.Amount must be greater than 8 and Person must not be equal to 1.$name, which is ProductName.The condition being of DAG type (Directed Acyclic Graph) allows us to describe nested conditions. You might write this condition in Python as:
if (Amount > 8) and (Person != 1):
Putting that into a graph form:
|------|and|------|
| |
| Amount > 8 | | Person != 1 |
Which is what we're describing with the DAG in TableGen.
full_json = run_tblgen(full_tblgen) print(json.dumps(full_json, indent=4))
{
"!instanceof": {
"Query": [
"anonymous_0",
"anonymous_1",
"anonymous_2",
"anonymous_3",
"anonymous_4"
]
},
"!tablegen_json_version": 1,
"all": {
"!anonymous": false,
"!fields": [],
"!name": "all",
"!superclasses": []
},
"and": {
"!anonymous": false,
"!fields": [],
"!name": "and",
"!superclasses": []
},
"anonymous_0": {
"!anonymous": true,
"!fields": [],
"!name": "anonymous_0",
"!superclasses": [
"Query"
],
"Fields": {
"args": [],
"kind": "dag",
"operator": {
"def": "all",
"kind": "def",
"printable": "all"
},
"printable": "(all)"
},
"OrderedBy": [],
"TableName": "Customer",
"WhereClause": {
"args": [],
"kind": "dag",
"operator": {
"def": "none",
"kind": "def",
"printable": "none"
},
"printable": "(none)"
}
},
"anonymous_1": {
"!anonymous": true,
"!fields": [],
"!name": "anonymous_1",
"!superclasses": [
"Query"
],
"Fields": {
"args": [
[
"Person",
null
],
[
"Amount",
null
]
],
"kind": "dag",
"operator": {
"def": "fields",
"kind": "def",
"printable": "fields"
},
"printable": "(fields \"Person\", \"Amount\")"
},
"OrderedBy": [],
"TableName": "Orders",
"WhereClause": {
"args": [],
"kind": "dag",
"operator": {
"def": "none",
"kind": "def",
"printable": "none"
},
"printable": "(none)"
}
},
"anonymous_2": {
"!anonymous": true,
"!fields": [],
"!name": "anonymous_2",
"!superclasses": [
"Query"
],
"Fields": {
"args": [
[
"Affiliation",
null
]
],
"kind": "dag",
"operator": {
"def": "fields",
"kind": "def",
"printable": "fields"
},
"printable": "(fields \"Affiliation\")"
},
"OrderedBy": [],
"TableName": "Customer",
"WhereClause": {
"args": [
[
"Name",
null
],
[
"Mary Blackburn",
"str"
]
],
"kind": "dag",
"operator": {
"def": "eq",
"kind": "def",
"printable": "eq"
},
"printable": "(eq \"Name\", \"Mary Blackburn\":$str)"
}
},
"anonymous_3": {
"!anonymous": true,
"!fields": [],
"!name": "anonymous_3",
"!superclasses": [
"Query"
],
"Fields": {
"args": [
[
"ProductName",
null
]
],
"kind": "dag",
"operator": {
"def": "fields",
"kind": "def",
"printable": "fields"
},
"printable": "(fields \"ProductName\")"
},
"OrderedBy": [],
"TableName": "Orders",
"WhereClause": {
"args": [
[
"Amount",
null
],
[
8,
null
]
],
"kind": "dag",
"operator": {
"def": "gt",
"kind": "def",
"printable": "gt"
},
"printable": "(gt \"Amount\", 8)"
}
},
"anonymous_4": {
"!anonymous": true,
"!fields": [],
"!name": "anonymous_4",
"!superclasses": [
"Query"
],
"Fields": {
"args": [
[
"ProductName",
"name"
],
[
"Person",
null
]
],
"kind": "dag",
"operator": {
"def": "fields",
"kind": "def",
"printable": "fields"
},
"printable": "(fields \"ProductName\":$name, \"Person\")"
},
"OrderedBy": [
"$name"
],
"TableName": "Orders",
"WhereClause": {
"args": [
[
{
"args": [
[
"Amount",
null
],
[
8,
null
]
],
"kind": "dag",
"operator": {
"def": "gt",
"kind": "def",
"printable": "gt"
},
"printable": "(gt \"Amount\", 8)"
},
null
],
[
{
"args": [
[
"Person",
null
],
[
1,
null
]
],
"kind": "dag",
"operator": {
"def": "ne",
"kind": "def",
"printable": "ne"
},
"printable": "(ne \"Person\", 1)"
},
null
]
],
"kind": "dag",
"operator": {
"def": "and",
"kind": "def",
"printable": "and"
},
"printable": "(and (gt \"Amount\", 8), (ne \"Person\", 1))"
}
},
"eq": {
"!anonymous": false,
"!fields": [],
"!name": "eq",
"!superclasses": []
},
"fields": {
"!anonymous": false,
"!fields": [],
"!name": "fields",
"!superclasses": []
},
"ge": {
"!anonymous": false,
"!fields": [],
"!name": "ge",
"!superclasses": []
},
"gt": {
"!anonymous": false,
"!fields": [],
"!name": "gt",
"!superclasses": []
},
"ne": {
"!anonymous": false,
"!fields": [],
"!name": "ne",
"!superclasses": []
},
"none": {
"!anonymous": false,
"!fields": [],
"!name": "none",
"!superclasses": []
},
"or": {
"!anonymous": false,
"!fields": [],
"!name": "or",
"!superclasses": []
}
}
The backend is going to walk the JSON we decoded. You can see the full output above in case you want to browse but for now don't read the whole thing. We will highlight the key aspects of it as we go along.
print(full_json["!instanceof"])
{'Query': ['anonymous_0', 'anonymous_1', 'anonymous_2', 'anonymous_3', 'anonymous_4']}
Any key beginning with ! is some sort of metadata about the other keys. Here this is a list of all instances of certain classes. We just have Query which lists all the queries we defined.
print(full_json["anonymous_0"]["!superclasses"])
['Query']
On each def there is also a !superclasses that gives you the same information. Meaning you could use !instanceof to get a list of keys to lookup, or you could walk all keys and check !superclasses.
print(full_json["anonymous_0"]["Fields"])
{'args': [], 'kind': 'dag', 'operator': {'def': 'all', 'kind': 'def', 'printable': 'all'}, 'printable': '(all)'}
From a def object you can find its attributes. Here we have the fields we want the query to show, which is all of them.
The core of a backend is looping over all defs of a certain class and outputting some text based on their properties.
Here we're going to loop over all defs of type Query and emit SQL queries for them.
def find_all_queries(j): queries = [] for key in j: # ! means it is some metadata, not a def. if not key.startswith("!"): value = full_json[key] # If we inherit from Query. if "Query" in value["!superclasses"]: queries.append(value) return queries queries = find_all_queries(full_json) print([q["!name"] for q in queries])
['anonymous_0', 'anonymous_1', 'anonymous_2', 'anonymous_3', 'anonymous_4']
Why are the names anonymous_...? When we defined them we did def : and missed out the name. This is allowed and llvm-tblgen just came up with a name for us. For this purpose the names are irrelevant.
Now we have the relevant classes we need to “emit” them. Meaning produce something from them, in this case a SQL query.
def emit_operator(operator): return { 'gt': ' > ', 'ge': ' >= ', 'lt': ' < ', 'le': ' <= ', 'ne': ' <> ', 'eq': ' = ', 'or': ' OR ', 'and': ' AND ' }[operator] print(emit_operator('and'))
AND
The maps our TableGen constants to the equivalent SQL logical operation.
def emit_fields(args): # Return a comma separated list of arg names. return ", ".join([arg[0] for arg in args]) print(emit_fields([["Abc", None], ["Def", None]]))
Abc, Def
This emits the the fields we are selecting. Each field has a name (arg[0]) and an optional tag that we will use later.
from collections.abc import Mapping def emit_where_clause(where_clause): output = "" num_args = len(where_clause["args"]) for idx, arg in enumerate(where_clause["args"]): arg_name, arg_type = arg if isinstance(arg_name, Mapping): # This is a nested where clause. output += emit_where_clause(arg_name) else: # This is some condition. if arg_type == "str": # String types must be emitted with "" around them. output += '"' + arg_name + '"' else: output += str(arg_name) # If this is not the last arg, emit the condition. if idx != (num_args-1): output += emit_operator(where_clause["operator"]["def"]) return output print(emit_where_clause({ "args": [["Name",None], ["Mary Blackburn", "str"]], "kind": "dag", "operator": { "def": "eq", "kind": "def", "printable": "eq" }}))
Name = "Mary Blackburn"
This emits the condition that goes with the WHERE. The condition is a DAG, which means that we will find a possible mix of conditions and other DAGS. We recurse to handle the latter case.
For each part of the condition we print the name of the thing you're checking, then the condition (=, <>, etc.). The value to check against is last and that goes on the end.
def emit_ordered_by(ordered_by, field_tag_map): # No ORDER BY statement to emit. if not ordered_by: return "" output = "\n ORDER BY " num_ordered_by = len(ordered_by) for idx, field_name in enumerate(ordered_by): # If it is a tag if field_name.startswith('$'): # Find the corresponding field name tag_name = field_name[1:] field_name = field_tag_map.get(tag_name) if field_name is None: raise RuntimeError('Unrecognized tag "{}"'.format( tag_name)) # Separate each tag after the first with ", ". if idx != 0: output += ", " output += field_name return output print(emit_ordered_by(["$abc", "$def"], {'abc':"ABC", 'def':"DEF"}))
ORDER BY ABC, DEF
emit_ordered_by produces the ORDER BY text. If there is no ordering return nothing, otherwise loop over all the fields we want to order by and emit their names.
If the name is a tag, we look that up in a map to get the real field name. Here's how we build that map:
def build_tag_map(arguments): # Args are [Name, Tag]. Reverse this so we have [Tag, Name]. # Add each one to a dictionary where Tag is the key and Name is the value. return dict([reversed(a) for a in arguments]) print(build_tag_map([["ABC", "abc"], ["DEF", "def"]]))
{'abc': 'ABC', 'def': 'DEF'}
def emit_query(q): fields_init = q["Fields"] field_op_name = fields_init["operator"]["def"] if not field_op_name in ["all", "fields"]: raise RuntimeError("Invalid dag operator " + field_op_name) field_tag_map = build_tag_map(fields_init["args"]) where_clause = q["WhereClause"] has_where = where_clause["operator"]["def"] != "none" ret = "SELECT " if field_op_name == "all": ret += "*" ret += emit_fields(fields_init["args"]) ret += " FROM " + q["TableName"] if has_where: ret += "\n WHERE " + emit_where_clause(where_clause) ret += emit_ordered_by(q["OrderedBy"], field_tag_map) ret += ";" return ret
Finally the main function. It emits the skeleton of the query and calls the helpers we defined earlier to fill in the gaps.
for q in queries: print(emit_query(q) + "\n")
SELECT * FROM Customer; SELECT Person, Amount FROM Orders; SELECT Affiliation FROM Customer WHERE Name = "Mary Blackburn"; SELECT ProductName FROM Orders WHERE Amount > 8; SELECT ProductName, Person FROM Orders WHERE Amount > 8 AND Person <> 1 ORDER BY ProductName;
Now we run emit_query and print out the results. There you have it, that's a TableGen backend!
You‘ve seen the core concepts. Loop over all the defs of a certain class and then emit some other structure based on the fields of each one. In this case it was SQL queries. In LLVM it’s most often C++ code but it can be anything you want.
If you want to see the same thing done with a C++ backend (one written in C++ that is, not producing it), check out the links at the start of this notebook.