You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@avro.apache.org by "Bartłomiej Romański (JIRA)" <ji...@apache.org> on 2014/11/25 16:34:12 UTC

[jira] [Created] (AVRO-1611) Avro-mapred should provide CombineAvroKeyInputFormat

Bartłomiej Romański created AVRO-1611:
-----------------------------------------

             Summary: Avro-mapred should provide CombineAvroKeyInputFormat
                 Key: AVRO-1611
                 URL: https://issues.apache.org/jira/browse/AVRO-1611
             Project: Avro
          Issue Type: Improvement
            Reporter: Bartłomiej Romański
            Priority: Minor
         Attachments: CombineAvroKeyInputFormat.java

A serious issue with Hadoop is dealing with a huge number of small files (they slow down processing, overload namenode etc...). 

A common remedy for this is to use CombineFileInputFormat. However, this is an abstract class to need to be conretized for each InputFormat it is wrapping. I believe Avro should provide CombineAvroKeyInputFormat like Hadoop is providing CombineSequenceFileInputFormat and CombineTextInputFormat.

I've attached a basic implementation.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)