[jira] [Created] (DRILL-5976) Kafka MessageReader config optimization

["B Anil Kumar (JIRA)" <ji...@apache.org>]

B Anil Kumar created DRILL-5976:
-----------------------------------

             Summary: Kafka MessageReader config optimization
                 Key: DRILL-5976
                 URL: https://issues.apache.org/jira/browse/DRILL-5976
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: B Anil Kumar
            Assignee: B Anil Kumar


Kafka storage plugin currently supports JSON message format, but going forward it will support Avro etc.  

Also there might be multiple scenarios where user have to implement their own MessageReader (Corresponding their custom Deserializer/Decoder). 

So, this JIRA is to brainstorm on whether to go with MessageReader as a *Plugin config* OR system/session option.

Paul's suggestion as part review comment is as below.

{noformat}
Suppose this is two or three releases from now and we support other forms of Kafka messages. Different topics use different formats.

If the message format is a system/session option, then I need to switch the option before each query. Very cumbersome and error prone.

Instead, perhaps this information should be part of the storage plugin config. Then, I can define different plugins: one for each message format.

Further, can I have multiple Kafka servers? If so, would I need different plugin configs for each?

So, should we be thinking about encoding most properties as plugin config properties?

Now, the plugin might have a format property, one of which is json. The JSON config properties would be defined in the json format within the overall storage plugin config.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)