A plugin: Kafka Sink and Source for flume

[Jack Ye <ja...@jasper.com>]

Does this work for kafka 0.8.1? thank you

https://github.com/baniuyao/flume-kafka

Re: A plugin: Kafka Sink and Source for flume

[Gwen Shapira <gs...@cloudera.com>]

OK, I can't resist a bit more detail about my favorite project :)

Here's a good blog post on how to use the Flume-Kafka source, sink and channel:
http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/

One of the main differences from Baniuyao's version is that we support
committing offsets to zookeeper on every batch (rather than
automatically) and batch sizes are configurable so you can tune
throughput / latency / cpu utilization / zookeeper load, etc.

Gwen

On Fri, Dec 12, 2014 at 6:10 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> I'd use the one in Flume trunk (it was far better tested) and we also
> have Kafka-channel :)
> And it works with 0.8.1.
>
> https://github.com/apache/flume/tree/trunk/flume-ng-sources/flume-kafka-source
> https://github.com/apache/flume/tree/trunk/flume-ng-sinks/flume-ng-kafka-sink
> https://github.com/apache/flume/tree/trunk/flume-ng-channels/flume-kafka-channel
>
> Gwen
>
> On Fri, Dec 12, 2014 at 5:33 PM, Jack Ye <ja...@jasper.com> wrote:
>> Does this work for kafka 0.8.1? thank you
>>
>> https://github.com/baniuyao/flume-kafka
>>

Re: A plugin: Kafka Sink and Source for flume

[Gwen Shapira <gs...@cloudera.com>]

I'd use the one in Flume trunk (it was far better tested) and we also
have Kafka-channel :)
And it works with 0.8.1.

https://github.com/apache/flume/tree/trunk/flume-ng-sources/flume-kafka-source
https://github.com/apache/flume/tree/trunk/flume-ng-sinks/flume-ng-kafka-sink
https://github.com/apache/flume/tree/trunk/flume-ng-channels/flume-kafka-channel

Gwen

On Fri, Dec 12, 2014 at 5:33 PM, Jack Ye <ja...@jasper.com> wrote:
> Does this work for kafka 0.8.1? thank you
>
> https://github.com/baniuyao/flume-kafka
>