You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ken Weiner <ke...@gumgum.com> on 2009/03/10 05:05:59 UTC
Support for zipped input files
I have a lot of large zipped (not gzipped) files sitting in an Amazon S3
bucket that I want to process. What is the easiest way to process them with
a Hadoop map-reduce job? Do I need to write code to transfer them out of
S3, unzip them, and then move them to HDFS before running my job, or does
Hadoop have support for processing zipped input files directly from S3?
Re: Support for zipped input files
Posted by tim robertson <ti...@gmail.com>.
There is LZO support with a patch:
http://blog.oskarsson.nu/2009/03/hadoop-feat-lzo-save-disk-space-and.html
Cheers
Tim
On Tue, Mar 10, 2009 at 12:23 PM, Tom White <to...@cloudera.com> wrote:
> Hi Ken,
>
> Unfortunately, Hadoop doesn't yet support MapReduce on zipped files
> (see https://issues.apache.org/jira/browse/HADOOP-1824), so you'll
> need to write a program to unzip them and write them into HDFS first.
>
> Cheers,
> Tom
>
> On Tue, Mar 10, 2009 at 4:11 AM, jason hadoop <ja...@gmail.com> wrote:
>> Hadoop has support for S3, the compression support is handled at another
>> level and should also work.
>>
>>
>> On Mon, Mar 9, 2009 at 9:05 PM, Ken Weiner <ke...@gumgum.com> wrote:
>>
>>> I have a lot of large zipped (not gzipped) files sitting in an Amazon S3
>>> bucket that I want to process. What is the easiest way to process them
>>> with
>>> a Hadoop map-reduce job? Do I need to write code to transfer them out of
>>> S3, unzip them, and then move them to HDFS before running my job, or does
>>> Hadoop have support for processing zipped input files directly from S3?
>>>
>>
>>
>>
>> --
>> Alpha Chapters of my book on Hadoop are available
>> http://www.apress.com/book/view/9781430219422
>>
>
Re: Support for zipped input files
Posted by Ken Weiner <ke...@gumgum.com>.
Thanks very much, Tom. You saved me a lot of time by confirming that it
isn't available yet. I'll go vote for HADOOP-1824.
On Tue, Mar 10, 2009 at 3:23 AM, Tom White <to...@cloudera.com> wrote:
> Hi Ken,
>
> Unfortunately, Hadoop doesn't yet support MapReduce on zipped files
> (see https://issues.apache.org/jira/browse/HADOOP-1824), so you'll
> need to write a program to unzip them and write them into HDFS first.
>
> Cheers,
> Tom
>
> On Tue, Mar 10, 2009 at 4:11 AM, jason hadoop <ja...@gmail.com>
> wrote:
> > Hadoop has support for S3, the compression support is handled at another
> > level and should also work.
> >
> >
> > On Mon, Mar 9, 2009 at 9:05 PM, Ken Weiner <ke...@gumgum.com> wrote:
> >
> >> I have a lot of large zipped (not gzipped) files sitting in an Amazon S3
> >> bucket that I want to process. What is the easiest way to process them
> >> with
> >> a Hadoop map-reduce job? Do I need to write code to transfer them out
> of
> >> S3, unzip them, and then move them to HDFS before running my job, or
> does
> >> Hadoop have support for processing zipped input files directly from S3?
> >>
> >
> >
> >
> > --
> > Alpha Chapters of my book on Hadoop are available
> > http://www.apress.com/book/view/9781430219422
> >
>
Re: Support for zipped input files
Posted by Tom White <to...@cloudera.com>.
Hi Ken,
Unfortunately, Hadoop doesn't yet support MapReduce on zipped files
(see https://issues.apache.org/jira/browse/HADOOP-1824), so you'll
need to write a program to unzip them and write them into HDFS first.
Cheers,
Tom
On Tue, Mar 10, 2009 at 4:11 AM, jason hadoop <ja...@gmail.com> wrote:
> Hadoop has support for S3, the compression support is handled at another
> level and should also work.
>
>
> On Mon, Mar 9, 2009 at 9:05 PM, Ken Weiner <ke...@gumgum.com> wrote:
>
>> I have a lot of large zipped (not gzipped) files sitting in an Amazon S3
>> bucket that I want to process. What is the easiest way to process them
>> with
>> a Hadoop map-reduce job? Do I need to write code to transfer them out of
>> S3, unzip them, and then move them to HDFS before running my job, or does
>> Hadoop have support for processing zipped input files directly from S3?
>>
>
>
>
> --
> Alpha Chapters of my book on Hadoop are available
> http://www.apress.com/book/view/9781430219422
>
Re: Support for zipped input files
Posted by jason hadoop <ja...@gmail.com>.
Hadoop has support for S3, the compression support is handled at another
level and should also work.
On Mon, Mar 9, 2009 at 9:05 PM, Ken Weiner <ke...@gumgum.com> wrote:
> I have a lot of large zipped (not gzipped) files sitting in an Amazon S3
> bucket that I want to process. What is the easiest way to process them
> with
> a Hadoop map-reduce job? Do I need to write code to transfer them out of
> S3, unzip them, and then move them to HDFS before running my job, or does
> Hadoop have support for processing zipped input files directly from S3?
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422