You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Mike Percy <mp...@gmail.com> on 2012/05/01 06:53:44 UTC
Re: Review Request: Support modification of event Headers through
configuration
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4826/#review7413
-----------------------------------------------------------
Hey; On further reflection, this approach just isn't great fit. The use case makes sense. However what we need is a decorator framework, not a sink-specific solution, and that is a relatively ambitious undertaking at this point.
Let's hold off on pursuing sink-specific decorators until a framework is put into place. I'm working on a source/sink-agnostic approach in FLUME-1157 and will provide updates on that JIRA soon.
- Mike
On 2012-04-20 10:33:14, Inder Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4826/
> -----------------------------------------------------------
>
> (Updated 2012-04-20 10:33:14)
>
>
> Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan.
>
>
> Summary
> -------
>
> Overview of Changes
>
> 1.EventHeaderDecorator -> generic class having processHeader()
> 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.
> 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config
> 4.AbstractSink -> has a eventHeaderDecortaor
> 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.
>
> Sample Configuration for testing
> ----------------------------------
> agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator
>
>
> This addresses bug FLUME-1097.
> https://issues.apache.org/jira/browse/FLUME-1097
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad
>
> Diff: https://reviews.apache.org/r/4826/diff
>
>
> Testing
> -------
>
> Testing Done
> -------------
> 1. Unit tests
> 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink
>
>
> Thanks,
>
> Inder
>
>
Re: Review Request: Support modification of event Headers through
configuration
Posted by Inder Singh <in...@gmail.com>.
> On 2012-05-01 04:53:44, Mike Percy wrote:
> > Hey; On further reflection, this approach just isn't great fit. The use case makes sense. However what we need is a decorator framework, not a sink-specific solution, and that is a relatively ambitious undertaking at this point.
> >
> > Let's hold off on pursuing sink-specific decorators until a framework is put into place. I'm working on a source/sink-agnostic approach in FLUME-1157 and will provide updates on that JIRA soon.
Sure Mike, will wait to get the design from the JIRA
- Inder
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4826/#review7413
-----------------------------------------------------------
On 2012-04-20 10:33:14, Inder Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4826/
> -----------------------------------------------------------
>
> (Updated 2012-04-20 10:33:14)
>
>
> Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan.
>
>
> Summary
> -------
>
> Overview of Changes
>
> 1.EventHeaderDecorator -> generic class having processHeader()
> 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.
> 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config
> 4.AbstractSink -> has a eventHeaderDecortaor
> 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.
>
> Sample Configuration for testing
> ----------------------------------
> agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator
>
>
> This addresses bug FLUME-1097.
> https://issues.apache.org/jira/browse/FLUME-1097
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad
>
> Diff: https://reviews.apache.org/r/4826/diff
>
>
> Testing
> -------
>
> Testing Done
> -------------
> 1. Unit tests
> 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink
>
>
> Thanks,
>
> Inder
>
>