[jira] [Comment Edited] (FLUME-2799) Kafka Source - Message Offset and Partition add to headers

["Michael Andre Pearce (IG) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097959#comment-15097959 ] 

Michael Andre Pearce (IG) edited comment on FLUME-2799 at 1/14/16 10:32 AM:
----------------------------------------------------------------------------

The core requirement here is that not all the metadata transformed / transferred to the FlumeEvent. So that interceptors and sinks can use the data.

This is a blocking issue for the organisation I work for to implement flume in our workflows where we use Kafka. (I am also aware of another org). 

RE The Solution:
The intent was to split consumption from transformation into separate entities, it keeps the two concerns separate. 
The design actually as per ticket, mimics that of the more mature JMS solution already in Flume. Its intent is not as an interceptor but to convert from source object event to FlumeEvent.
Happy to merge the two, but I think it is cleaner to separate the transform from source system event to flume event and the consume code, and I think the solution in JMS in cleaner and neater.

As for the number of headers this is far few headers / meta data than will come from sources such as JMS. Also to note, the data will already will be in memory as in MessageAndMetaData that we transform from.


RE Support for Kafka 0.9.0
This ticket has been open since sept, as patch is developed, and solves the issue and all existing tests pass I do not see any reason to hold off merging the functionality. Unless the new consumer is ready for next release with these features supported I do not see why one would hold off. 

Also need to be aware kafka 0.9.0 only just release many organisations will not upgrade instantly, as such using the old consumer for flume for some period of time will be needed for compatibility. 


was (Author: michael.andre.pearce):
The core requirement here is that not all the metadata transformed / transferred to the FlumeEvent. So that interceptors and sinks can use the data.

This is a blocking issue for the organisation I work for to implement flume in our workflows where we use Kafka. (I am also aware of another org). 

RE The Solution:
The intent was to split consumption from transformation into separate entities, it keeps the two concerns separate. 
The design actually as per ticket, mimics that of the more mature JMS solution already in Flume. Its intent is not as an interceptor but to convert from source object event to FlumeEvent.
Happy to merge the two, but I think it is cleaner to separate the transform from source system event to flume event and the consume code, and I think the solution in JMS in cleaner and neater.

As for the number of headers this is far few headers / meta data than will come from sources such as JMS. Also to note, the data will already will be in memory as in MessageAndMetaData that we transform from.


RE Support for Kafka 0.9.0
Holding off, this ticket has been open since sept, as patch is developed, and solves the issue and all existing tests pass I do see any reason to hold off merging the functionality. Unless the new consumer is ready for next release with these features supported I do not see why one would hold off. 

Also need to be aware kafka 0.9.0 only just release many organisations will not upgrade instantly, as such using the old consumer for flume for some period of time will be needed for compatibility. 

> Kafka Source - Message Offset and Partition add to headers
> ----------------------------------------------------------
>
>                 Key: FLUME-2799
>                 URL: https://issues.apache.org/jira/browse/FLUME-2799
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>    Affects Versions: v1.6.0
>            Reporter: Michael Andre Pearce (IG)
>            Priority: Minor
>              Labels: easyfix, patch
>             Fix For: v1.7.0
>
>         Attachments: FLUME-2799-0.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently Kafka source only persists the original kafka message's topic into the Flume event headers.
> For downstream interceptors and sinks that may want to have available to them the partition and the offset , we need to add these.
> Also it is noted that the conversion from MessageAndMetaData to FlumeEvent is not configurable unlike other sources such as JMS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)