[jira] [Created] (APEXMALHAR-2459) KafkaInputoperator using 0.10.* Kafka consumer API

["Chaitanya (JIRA)" <ji...@apache.org>]

Chaitanya created APEXMALHAR-2459:
-------------------------------------

             Summary: KafkaInputoperator using 0.10.* Kafka consumer API
                 Key: APEXMALHAR-2459
                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2459
             Project: Apache Apex Malhar
          Issue Type: Task
            Reporter: Chaitanya
            Assignee: Chaitanya


Changes in 0.10.0.0 version of Kafka
-------------------------------------------------
   Message format in 0.10.0 has changed. Messages now include a time-stamp field and compressed messages include relative offsets. 
    
   Please refer the below link about the changes in Kafka 0.10.0.0 version:
https://kafka.apache.org/documentation/#upgrade_10_breaking.

Design
---------
  I'm proposing the below design for new Kafka connectors by refactoring the existing Kafka connector(0.9.* version)
1) Split up the malhar-kafka to malhar-kafka-common, malhar-kafka_0.9, malhar-kafka_0.10. Directory structure would be as follows:
  malhar-kafka
          |--malhar-kafka-common
          |--malhar-kafka_0.9
          |--malhar-kafka_0.10
If possible, will add the connectors for 0.8.x version.
2) Convert the KafkaConsumerWrapper to AbstractKafkaConsumerWrapper by adding the abstract methods over the consumer API's.
3) "malhar-kafka-common" package consists of the abstract classes and by default it builds over the 0.9.x version of Kafka.
4) "malhar-kafka-0.*" consists of concrete classes and its corresponding KafkaConsumerWrapper.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)