[jira] [Created] (GRIFFIN-213) Support pluggable datasource connectors

["Nikolay Sokolov (JIRA)" <ji...@apache.org>]

Nikolay Sokolov created GRIFFIN-213:
---------------------------------------

             Summary: Support pluggable datasource connectors
                 Key: GRIFFIN-213
                 URL: https://issues.apache.org/jira/browse/GRIFFIN-213
             Project: Griffin (Incubating)
          Issue Type: Improvement
            Reporter: Nikolay Sokolov


As of Griffin 0.3, code modification is required, in order to add new data connectors.

Proposal is to add new data connector type, CUSTOM, that would allow to specify class name of data connector implementation to use. Additional jars with custom connector implementations would be provided in spark configuration template.

Class name would be specified in "class" config of data connector. For example:
{code:json}
"connectors": [
        {
          "type": "CUSTOM",
          "config": {
            "class": "org.example.griffin.JDBCConnector"
            // extra connector-specific parameters
          }
        }
      ]
{code}

Proposed contract for implementations is based on current convention:
 - for batch
 ** class should be a subclass of BatchDataConnector
 ** if should have method with signature:
{code:java}
public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
{code}

 - for streaming
 ** class should be a subclass of StreamingDataConnector
 ** it should have method with signature:
{code:java}
public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
{code}

Signatures of context objects:
{code:scala}
case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
                                     dcParam: DataConnectorParam,
                                     timestampStorage: TimestampStorage)

case class StreamingDataConnectorContext(@transient sparkSession: SparkSession,
                                         @transient ssc: StreamingContext,
                                         dcParam: DataConnectorParam,
                                         timestampStorage: TimestampStorage,
                                         streamingCacheClientOpt: Option[StreamingCacheClient])

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)