[jira] [Commented] (GRIFFIN-213) Support pluggable datasource connectors

["Nikolay Sokolov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690695#comment-16690695 ] 

Nikolay Sokolov commented on GRIFFIN-213:
-----------------------------------------

[~Lionel_3L]  [~guoyp] do you have any thoughts on that proposal?

> Support pluggable datasource connectors
> ---------------------------------------
>
>                 Key: GRIFFIN-213
>                 URL: https://issues.apache.org/jira/browse/GRIFFIN-213
>             Project: Griffin (Incubating)
>          Issue Type: Improvement
>            Reporter: Nikolay Sokolov
>            Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to specify class name of data connector implementation to use. Additional jars with custom connector implementations would be provided in spark configuration template.
> Class name would be specified in "class" config of data connector. For example:
> {code:json}
> "connectors": [
>         {
>           "type": "CUSTOM",
>           "config": {
>             "class": "org.example.griffin.JDBCConnector"
>             // extra connector-specific parameters
>           }
>         }
>       ]
> {code}
> Proposed contract for implementations is based on current convention:
>  - for batch
>  ** class should be a subclass of BatchDataConnector
>  ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
>  - for streaming
>  ** class should be a subclass of StreamingDataConnector
>  ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
>                                      dcParam: DataConnectorParam,
>                                      timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession: SparkSession,
>                                          @transient ssc: StreamingContext,
>                                          dcParam: DataConnectorParam,
>                                          timestampStorage: TimestampStorage,
>                                          streamingCacheClientOpt: Option[StreamingCacheClient])
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)