[jira] [Created] (TRAFODION-51) LP Blueprint: cmp-tmudf-compile-time-interface - Complete work for TMUDF compile time interface

["Alice Chen (JIRA)" <ji...@apache.org>]

Alice Chen created TRAFODION-51:
-----------------------------------

             Summary: LP Blueprint: cmp-tmudf-compile-time-interface - Complete work for TMUDF compile time interface
                 Key: TRAFODION-51
                 URL: https://issues.apache.org/jira/browse/TRAFODION-51
             Project: Apache Trafodion
          Issue Type: New Feature
          Components: sql-cmp
            Reporter: Hans Zeller
            Assignee: Hans Zeller
             Fix For: 1.1 (pre-incubation)


Suresh and others implemented much of a compile time interface for TMUDFs (Table-Mapping UDFs). Such an interface allows a TMUDF to be polymorphic (input and output columns decided at compile time, not at create time) and to do optimizations like elimination of unneeded columns, pushing predicates below or into the TMUDF, getting better cardinality and cost estimates, and using sort order and partitioning of input tables.

A TMUDF can have zero or more input tables. Using more than one input table is not tested and
supported at the moment, but the design should allow it.

The interface will be a C++ class that the TMUDF writer can derive from. Implementing a compiler
interface for a TMUDF is completely optional and is done by overriding virtual methods of the
default implementation. In a later step, we also want to replace the C interface
at runtime with this C++ interface, designed a few years ago. We should be able to define a
Java compile time interface that's fairly similar to the one in C++. A TMUDF writer only needs to
override those interfaces that need to be different from the default implementation. For example,
a TMUDF could define its output columns through the compiler interface, but it might not
support pushing predicates into the TMUDF. Another example could be a TMUDF that only
implements the compiler interface that determines the degree of parallelism.

There are three main classes in this interface, all are defined in file core/sql/sqludr/sqludr.h:

UDRInvocationInfo: This is similar to SQLUDR_TMUDFINFO in the C interface. It describes
the metadata of the TMUDF, scalar input parameters, the table-valued result, PARTITION BY
and ORDER BY clauses specified for table-valued inputs, etc. There is one of these for every
TMUDF invocation in a query. In some cases, the compiler may create additional
UDRInvocationInfo objects when it transforms the TMUDF, for example by placing it under
a nested join, with a different set of predicates to be pushed down. There are additional
classes to describe parameters, table-valued inputs and outputs, data types, similar to
the existing C interface.

UDRPlanInfo: There are zero or more UDRPlanInfo objects for every UDRInvocationInfo
object. The optimizer creates one for every optimization goal (context) where it needs to
call the TMUDF interface.

TMUDRInterface: This class represents the code associated with a TMUDF. The class itself
represents the default behavior of a TMUDF without a compile time interface. UDF writers
can define a derived class and implement virtual methods to customize the optimizer
interface. Trafodion tries to find a C function
<UDF external function name>_CreateCompilerInterfaceObject
If that function exists in the UDF library, it is assumed to return an object of a class
derived from TMUDRInterface, and the compiler will call the virtual methods, some
could be defined in the derived class, some could be in the base class, and the derived
class also might call the base class method to do part of the work.

Here are the methods we plan to support:

- Validate scalar input parameters, possibly allow those parameters to deviate from the
  parameter list declared at DDL time.
- Allow the compiler interface to look at constant values that are passed in as input
  parameters.
- Define the table-valued result columns, based on scalar parameters and column
  layout of the input (child) table(s).
- Eliminate unneeded columns from the TMUDF result and also from the input tables.
- Allow predicates to be pushed down through he TMUDF operator to the child table(s).
- Allow predicates to be absorbed into the TMUDF.
- Return a cost estimate of the TMUDF, based on information available at compile time.
- Influence the degree of parallelism chosen for the TMUDF.
- Make use of natural partitioning and sort order of input (child) tables to produce
  partitioned and sorted results.
- Gather the necessary information based on the compile time interaction that is
  needed at runtime.
- At any time in the process, the compile time interface can raise an exception. If
  it does, the compilation will fail and an error message provided by the TMUDF
  writer will be returned in the diagnostics area.

Some more design choices:

The C++ interface uses its own C++ namespace, to avoid naming collisions and
to make it look more similar to Java. Objects are allocated on the system heap and
are deleted after statement compilation is finished. The interface does not use any
of the Trafodion objects like NAHeap, ComDiagsArea, etc. The interface does use
C++ STL for strings and collection templates, again with the goal to stay close
to Java.

NAString ==> std::string
NAHeap ==> C++ system heap
ComDiagsArea ==> Throw exception with an attached SQLSTATE and error message

In this first implementation, we only support a C++ interface and the compiler will
call that interface directly, without going through the tdm_udrserv process. We may
need a special privilege to allow a user to define code that's executed in the
Trafodion process (the privilege to create a TMUDF that has a compile time interface).
In the longer term we hope to support the following flavors:

- C++ and Java interfaces for the TMUDF, both at compile time and run time.
- Trusted and isolated modes, both at compile and run time.

Example code for a "sessionize" TMUDF that expects a single table input and
passes all columns through to the result, in addition to the session id column
(assume that session id column is defined as the only output column in the DDL):

class SessionizeUDFInterface : public TMUDRInterface
{
  // override any methods where the UDF author would
  // like to change the default behavior

  void describeParamsAndColumns(UDRInvocationInfo &info);

};

void SessionizeUDFInterface::describeParamsAndColumns(
     UDRInvocationInfo &info)
{
      // sessionize is intended to work with a single input table
      if (info.getNumTableInputs() != 1)
        throw UDRException(38001,
                           "Expecting one table-valued input, got %d",
                           info.getNumTableInputs());

      // add all input table columns as output columns
      info.addPassThruColumns(0);

}

extern "C" TMUDRInterface * SESSIONIZE_CreateCompilerInterfaceObject(
     const UDRInvocationInfo *info)
{
  return new SessionizeUDFInterface();
}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)