<!--===- docs/OpenMP-declare-target.md

   Part of the LLVM Project, under the Apache License v2.0 with LLVM
   Exceptions.
   See https://llvm.org/LICENSE.txt for license information.
   SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

-->

# Introduction to Declare Target

In OpenMP `declare target` is a directive that can be applied to a function or
variable (primarily global) to notate to the compiler that it should be 
generated in a particular device's environment. In essence whether something 
should be emitted for host or device, or both. An example of its usage for 
both data and functions can be seen below.

```Fortran
module test_0
    integer :: sp = 0
!$omp declare target link(sp)
end module test_0

program main
    use test_0
!$omp target map(tofrom:sp)
    sp = 1
!$omp end target
end program
```

In the above example, we create a variable in a separate module, mark it 
as `declare target` and then map it, embedding it into the device IR and 
assigning to it. 


```Fortran
function func_t_device() result(i)
    !$omp declare target to(func_t_device) device_type(nohost)
        INTEGER :: I
        I = 1
end function func_t_device

program main
!$omp target
    call func_t_device()
!$omp end target
end program
```

In the above example, we are stating that a function is required on device
utilising `declare target`, and that we will not be utilising it on host, 
so we are in theory free to remove or ignore it there. A user could also 
in this case, leave off the `declare target` from the function and it 
would be implicitly marked `declare target any` (for both host and device), 
as it's been utilised within a target region.

# Declare Target as represented in the OpenMP Dialect

In the OpenMP Dialect `declare target` is not represented by a specific 
`operation`. Instead, it's an OpenMP dialect specific `attribute` that can be 
applied to any operation in any dialect, which helps to simplify the 
utilisation of it. Rather than replacing or modifying existing global or 
function `operations` in a dialect, it applies to it as extra metadata that
the lowering can use in different ways as is necessary. 

The `attribute` is composed of multiple fields representing the clauses you 
would find on the `declare target` directive i.e. device type (`nohost`, 
`any`, `host`) or the capture clause (`link` or `to`). A small example of 
`declare target` applied to a Fortran `real` can be found below:

```
fir.global internal @_QFEi {omp.declare_target = 
#omp.declaretarget<device_type = (any), capture_clause = (to)>} : f32 {
    %0 = fir.undefined f32
    fir.has_value %0 : f32
}
```

This would look similar for function style `operations`.

The application and access of this attribute is aided by an OpenMP Dialect 
MLIR Interface named `DeclareTargetInterface`, which can be utilised on 
operations to access the appropriate interface functions, e.g.:

```C++
auto declareTargetGlobal = 
llvm::dyn_cast<mlir::omp::DeclareTargetInterface>(Op.getOperation());
declareTargetGlobal.isDeclareTarget();
```

# Declare Target Fortran OpenMP Lowering

The initial lowering of `declare target` to MLIR for both use-cases is done
inside of the usual OpenMP lowering in flang/lib/Lower/OpenMP.cpp. However, 
some direct calls to `declare target` related functions from Flang's 
lowering bridge in flang/lib/Lower/Bridge.cpp are made.

The marking of operations with the declare target attribute happens in two 
phases, the second one optional and contingent on the first failing. The 
initial phase happens when the declare target directive and its clauses 
are initially processed, with the primary data gathering for the directive and
clause happening in a function called `getDeclareTargetInfo`. This is then used
to feed the `markDeclareTarget` function, which does the actual marking 
utilising the `DeclareTargetInterface`. If it encounters a variable or function
that has been marked twice over multiple directives with two differing device
types (e.g. `host`, `nohost`), then it will swap the device type to `any`.

Whenever we invoke `genFIR` on an `OpenMPDeclarativeConstruct` from the 
lowering bridge, we are also invoking another function called 
`gatherOpenMPDeferredDeclareTargets`, which gathers information relevant to the
application of the `declare target` attribute. This information 
includes the symbol that it should be applied to, device type clause, 
and capture clause, and it is stored in a vector that is part of the lowering
bridge's instantiation of the `AbstractConverter`. It is only stored if we 
encounter a function or variable symbol that does not have an operation 
instantiated for it yet. This cannot happen as part of the 
initial marking as we must store this data in the lowering bridge and we 
only have access to the abstract version of the converter via the OpenMP 
lowering. 

The information produced by the first phase is used in the second phase, 
which is a form of deferred processing of the `declare target` marked 
operations that have delayed generation and cannot be proccessed in the 
first phase. The main notable case this occurs currently is when a 
Fortran function interface has been marked. This is 
done via the function 
`markOpenMPDeferredDeclareTargetFunctions`, which is called from the lowering
bridge at the end of the lowering process allowing us to mark those where 
possible. It iterates over the data previously gathered by 
`gatherOpenMPDeferredDeclareTargets` 
checking if any of the recorded symbols have now had their corresponding 
operations instantiated and applying the declare target attribute where 
possible utilising `markDeclareTarget`. However, it must be noted that it 
is still possible for operations not to be generated for certain symbols, 
in particular the case of function interfaces that are not directly used 
or defined within the current module. This means we cannot emit errors in 
the case of left-over unmarked symbols. These must (and should) be caught 
by the initial semantic analysis.

NOTE: `declare target` can be applied to implicit `SAVE` attributed variables.
However, by default Flang does not represent these as `GlobalOp`'s, which means
we cannot tag and lower them as `declare target` normally. Instead, similarly
to the way `threadprivate` handles these cases, we raise and initialize the 
variable as an internal `GlobalOp` and apply the attribute. This occurs in the
flang/lib/Lower/OpenMP.cpp function `genDeclareTargetIntGlobal`.

# Declare Target Transformation Passes for Flang

There are currently two passes within Flang that are related to the processing 
of `declare target`:
* `MarkDeclareTarget` - This pass is in charge of marking functions captured
(called from) in `target` regions or other `declare target` marked functions as
`declare target`. It does so recursively, i.e. nested calls will also be 
implicitly marked. It currently will try to mark things as conservatively as 
possible, e.g. if captured in a `target` region it will apply `nohost`, unless
it encounters a `host` `declare target` in which case it will apply the `any` 
device type. Functions are handled similarly, except we utilise the parent's 
device type where possible.
* `FunctionFiltering` - This is executed after the `MarkDeclareTarget`
pass, and its job is to conservatively remove host functions from
the module where possible when compiling for the device. This helps make 
sure that most incompatible code for the host is not lowered for the 
device. Host functions with `target` regions in them need to be preserved 
(e.g. for lowering the `target region`(s) inside). Otherwise, it removes 
any function marked as a `declare target host` function and any uses will be 
replaced with `undef`'s so that  the remaining host code doesn't become broken. 
Host functions with `target` regions are marked with a `declare target host` 
attribute so they will be removed after outlining the target regions contained
inside.

While this infrastructure could be generally applicable to more than just Flang, 
it is only utilised in the Flang frontend, so it resides there rather than in 
the OpenMP dialect codebase. 

# Declare Target OpenMP Dialect To LLVM-IR Lowering

The OpenMP dialect lowering of `declare target` is done through the 
`amendOperation` flow, as it's not an `operation` but rather an 
`attribute`. This is triggered immediately after the corresponding
operation has been lowered to LLVM-IR. As it is applicable to
different types of operations, we must specialise this function for 
each operation type that we may encounter. Currently, this is 
`GlobalOp`'s and `FuncOp`'s.

`FuncOp` processing is fairly simple. When compiling for the device, 
`host` marked functions are removed, including those that could not 
be removed earlier due to having `target` directives within. This 
leaves `any`, `device` or indeterminable functions left in the 
module to lower further. When compiling for the host, no filtering is 
done because `nohost` functions must be available as a fallback 
implementation.

For `GlobalOp`'s, the processing is a little more complex. We 
currently leverage the `registerTargetGlobalVariable` and 
`getAddrOfDeclareTargetVar` `OMPIRBuilder` functions shared with Clang. 
These two functions invoke each other depending on the clauses and options 
provided to the `OMPIRBuilder` (in particular, unified shared memory). Their
main purposes are the generation of a new global device pointer with a 
"ref_" prefix on the device and enqueuing metadata generation by the 
`OMPIRBuilder` to be produced at module finalization time. This is done 
for both host and device and it links the newly generated device global 
pointer and the host pointer together across the two modules.

Similarly to other metadata (e.g. for `TargetOp`) that is shared across
both host and device modules, processing of `GlobalOp`'s in the device 
needs access to the previously generated host IR file, which is done 
through another `attribute` applied to the `ModuleOp` by the compiler 
frontend. The file is loaded in and consumed by the `OMPIRBuilder` to 
populate it's `OffloadInfoManager` data structures, keeping host and 
device appropriately synchronised.

The second (and more important to remember) is that as we effectively replace
the original LLVM-IR generated for the `declare target` marked `GlobalOp` we
have some corrections we need to do for `TargetOp`'s (or other region 
operations that use them directly) which still refer to the original lowered
global operation. This is done via `handleDeclareTargetMapVar` which is invoked
as the final function and alteration to the lowered `target` region, it's only
invoked for device as it's only required in the case where we have emitted the
"ref" pointer , and it effectively replaces all uses of the originally lowered
global symbol, with our new global ref pointer's symbol. Currently we do not
remove or delete the old symbol, this is due to the fact that the same symbol
can be utilised across multiple target regions, if we remove it, we risk 
breaking lowerings of target regions that will be processed at a later time. 
To appropriately delete these no longer necessary symbols we would need a 
deferred removal process at the end of the module, which is currently not in 
place. It may be possible to store this information in the OMPIRBuilder and 
then perform this cleanup process on finalization, but this is open for 
discussion and implementation still.

# Current Support

For the moment, `declare target` should work for:
* Marking functions/subroutines and function/subroutine interfaces for 
generation on host, device or both.
* Implicit function/subroutine capture for calls emitted in a `target` region 
or explicitly marked `declare target` function/subroutine. Note: Calls made 
via arguments passed to other functions must still be themselves marked 
`declare target`, e.g. passing a `C` function pointer and invoking it, then 
the interface and the `C` function in the other module must be marked 
`declare target`, with the same type of marking as indicated by the 
specification.
* Marking global variables with `declare target`'s `link` clause and mapping 
the data to the device data environment utilising `declare target`. This may 
not work for all types yet, but for scalars and arrays of scalars, it 
should.

Doesn't work for, or needs further testing for:
* Marking the following types with `declare target link` (needs further 
testing):
    * Descriptor based types, e.g. pointers/allocatables.
    * Derived types.
    * Members of derived types (use-case needs legality checking with OpenMP
specification).
* Marking global variables with `declare target`'s `to` clause. A lot of the
lowering should exist, but it needs further testing and likely some further 
changes to fully function.