[jira] [Commented] (KAFKA-1705) Add MR layer to Kafka

["Gwen Shapira (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171720#comment-14171720 ] 

Gwen Shapira commented on KAFKA-1705:
-------------------------------------

Looking for community comments on this. Do others see this as a good thing to add?

> Add MR layer to Kafka
> ---------------------
>
>                 Key: KAFKA-1705
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1705
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Gwen Shapira
>            Assignee: Gwen Shapira
>
> Many NoSQL-type storage systems (HBase, Mongo,
> Cassandra) and file formats (Avro, Parquet) provide is a MapReduce
> integration layer - usually an InputFormat, OutputFormat and a utility
> class. Sometimes there's also an abstract Job and Mapper that do more
> setup, which can make things even more convenient.
> This is different than the existing Hadoop contrib project or Camus in that an MR layer will be providing components for use in MR jobs, not an entire job that ingests data from Kafka to HDFS.
> The benefits I see for a MapReduce layer are:
> * Developers can create their own jobs, processing the data as it is
> ingested - rather than having to process it in two steps.
> * There's reusable components for developers looking to integrate with
> Kafka, rather than having everyone implement their own solution.
> * Hadoop developers expect projects to have this layer.
> * Spark reuses Hadoop's InputFormat and OutputFormat - so we get Spark
> integration for free.
> * There's a layer to plug the delegation token code into and make it
> invisible to MapReduce developers. Without this, everyone who writes
> MR jobs will need to think about how to implement authentication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)