Providing bzip2 as codecs in Hadoop Core

[Abdul Qadeer <qa...@gmail.com>]

Hi,

We recently submitted the code to support bzip2 as
another default codecs at JIRA entry
(https://issues.apache.org/jira/browse/HADOOP-3646)
It would be great to have your comments/suggestions in this
regard.  I am copying the description of the above mentioned
JIRA entry for your quick reference.

Thanks,
Abdul Qadeer

///////////////Description of JIRA entry 3646/////////////////////
Hadoop recognizes gzip compressed input and automatically decompresses the
data before providing it to the mapper. But Hadoop can not split a gzip
stream due to the very nature of the gzip compression. Consequently one gzip
stream (e.g a whole file) can go to only one mapper. On the contrary Bzip2
compressed stream can be split across its block delimiters.

We are interested in extending Hadoop to support splittable bzip2 with a
codec. (https://issues.apache.org/jira/browse/HADOOP-1823 uses input reader
to split the bzip2 files, which must be provided by the user and can handle
FileInputFormat. If a user wants to use some other input format or wants to
do custom record handling, he must write a new input reader!)

We have a patch now that provides a basic bzip2 codec equivalent to the
current gzip codec. We are in the process of extending that to support
splitting.