orc-rt/docs/Design.md

# ORC Runtime Design

The ORC runtime provides APIs for *executor* processes in an ORC JIT session
(as opposed to the LLVM ORC libraries which provide APIs for *controller*
processes). This includes support for both JIT'd code itself, and for users
of JIT'd code.

## Background

LLVM's On Request Compilation (ORC) APIs support cross-process loading of JIT'd
code. We call the process that defines and links the JIT'd code the *controller*
and the process that executes JIT'd code the *executor*. Controller processes
will link LLVM's ORC library, and construct a JIT'd program using an
llvm::orc::ExecutionSession instance (typically through an convenience wrapper
like llvm::orc::LLJIT). Executor processes construct an `orc_rt::Session`
object to manage resources for, and access to, JIT'd code within the executor
process.

## APIs

### Session

The Session object is the root object for a JIT'd program. It owns the
Service instances that manage services and resources supporting JIT'd code
(e.g. JIT'd memory, unwind info registrations, dynamic library handles, etc.).

The Session object must be constructed prior to adding any JIT'd code, and must
outlive execution of any JIT'd code.

An executor may have more than one Session object, in which case each Session
object must outlive execution of any JIT'd code added to that specific session.

### ControllerAccess

ControllerAccess objects support bidirectional RPC between JIT'd code in the
executor and the ExecutionSession in the controller.

Calls in both directions are to "wrapper functions" with a fixed signature (a
function that takes a blob of bytes and returns a blob of bytes as its result).
ControllerAccess objects can not generally assume anything about the format of
the bytes being sent (their interpretation is up to the called function). The
RPC is not fully symmetric: Calls from the controller to the executor specify
wrapper function *addresses* (i.e. the controller can invoke any code in the
executor). Calls from the executor to the controller specify *tags*, which are
addresses in the executor processes that are associated with handlers in the
controller. This ensures that the executing process can only call deliberately
exposed entry points in the controller.

ControllerAccess objects may be detached before the session ends, at which point
JIT'd code may continue executing, but will receive no further calls from the
controller and can make no further calls to the controller.

### Service

`Service` is an interface for classes that provide services to the Session.
E.g. memory managers, or dynamic library loaders.

The `Service` interface provides two operations: `onDetach` and `onShutdown`.
`onDetach` signals that controller access is permanently unavailable. It is
always called before `onShutdown`, regardless of how the Session reaches
shutdown -- including when no controller was ever attached. Since no further
requests will be made by the controller after `onDetach`, Services may use it
to abandon any fine-grained book-keeping that is only needed to service
controller requests. Many Services will implement `onDetach` as a no-op.

The `onShutdown` operation will be called at `Session` destruction time, after
all outstanding keepalives have been released. Services should release all
held resources during `onShutdown`.

### Keepalives and shutdown

Session teardown must not proceed while JIT'd code is still live on a stack:
freeing the JIT'd code (and the resources it runs against) out from under those
frames would crash the moment control returns to them.

To help enforce this the Session carries a keepalive `TaskGroup`. Code that
is about to run JIT'd code obtains a `TaskGroup::Token` -- a *keepalive* -- from
the group to bracket that execution, and the group delays Session shutdown until
every keepalive has been released. `Session::callWithKeepalive` acquires and
holds one for you around a synchronous call; keepalives can also be acquired
manually from the `TokenSource` returned by `Session::keepaliveTokenSource`.

A keepalive brackets a single span of execution on a stack -- *not* a whole
chain of asynchronous operations. When an asynchronous operation captures a
continuation, whoever later invokes that continuation must obtain a fresh
keepalive to bracket the invocation (the continuation can't do it itself: its
entry point may already be JIT'd code).

Keepalive acquisition fails once Session shutdown has been requested -- and it
can fail even for a nested or resumed call whose caller already holds one. So
every caller of JIT'd code needs a way to abort and unwind when acquisition is
denied; `callWithKeepalive` reports denial through its return value, which
callers must check.

### TaskDispatcher

Runs Tasks within the ORC runtime. In particular, calls originating from the
controller (via ControllerAccess) will be dispatched as Tasks.

TaskDispatchers are responsible for ensuring that all dispatched Tasks have
completed or been destroyed during Session shutdown.

### WrapperFunction

A wrapper function is any function with the following C signature:

```c
void (orc_rt_SessionRef Session, uint64_t CallId,
      orc_rt_WrapperFunctionReturn Return,
      orc_rt_WrapperFunctionBuffer ArgBytes);
```

where `orc_rt_WrapperFunctionReturn` and `orc_rt_WrapperFunctionBuffer` are
defined as:

```c
typedef struct {
  orc_rt_WrapperFunctionBufferDataUnion Data;
  size_t Size;
} orc_rt_WrapperFunctionBuffer;

/**
 * Asynchronous return function for an orc-rt wrapper function.
 */
typedef void (*orc_rt_WrapperFunctionReturn)(
    orc_rt_SessionRef Session, uint64_t CallId,
    orc_rt_WrapperFunctionBuffer ResultBytes);
```

The orc_rt::WrapperFunction class provides APIs for implementing and calling
wrapper functions.

### SPSWrapperFunction

An SPS wrapper function is a wrapper function that uses the
SimplePackedSerialization scheme (see documentation in
orc-rt/include/orc-rt/SimplePackedSerialization.h).

## TODO:

Document...

* C API
* Error handling
* RTTI
* ExecutorAddr / ExecutorAddrRange
* SimpleNativeMemoryMap
* Memory Access (unimplemented)
* Platform classes (unimplemented)
* Other utilities