[jira] [Assigned] (DRILL-5976) Kafka MessageReader config optimization

["Arina Ielchiieva (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arina Ielchiieva reassigned DRILL-5976:
---------------------------------------

    Assignee:     (was: AnilKumar B)

> Kafka MessageReader config optimization
> ---------------------------------------
>
>                 Key: DRILL-5976
>                 URL: https://issues.apache.org/jira/browse/DRILL-5976
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: AnilKumar B
>            Priority: Major
>
> Kafka storage plugin currently supports JSON message format, but going forward it will support Avro etc.  
> Also there might be multiple scenarios where user have to implement their own MessageReader (Corresponding their custom Deserializer/Decoder). 
> So, this JIRA is to brainstorm on whether to go with MessageReader as a *Plugin config* OR system/session option.
> Paul's suggestion as part review comment is as below.
> {noformat}
> Suppose this is two or three releases from now and we support other forms of Kafka messages. Different topics use different formats.
> If the message format is a system/session option, then I need to switch the option before each query. Very cumbersome and error prone.
> Instead, perhaps this information should be part of the storage plugin config. Then, I can define different plugins: one for each message format.
> Further, can I have multiple Kafka servers? If so, would I need different plugin configs for each?
> So, should we be thinking about encoding most properties as plugin config properties?
> Now, the plugin might have a format property, one of which is json. The JSON config properties would be defined in the json format within the overall storage plugin config.
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)