Has anyone experienced a major lag reading gzipped files?

[montag <mk...@about.com>]

Hi all,

  Has anyone been having issues with Hadoop jobs involving a large
collection of gzipped files, specifically on EC2?  I currently have a job
set up in which I take as input about 360 gzipped log files within HDFS,
totalling to about 30GB of compressed data.  I have noticed that if I leave
these files compressed, the JobClient hangs at 0% map, 0% reduce, and will
eventually jump to 100% map and reduce (after a rather long time) without
having reported any progress in between.

However, if I unzip the files before pushing them to HDFS, the job starts
almost immediately.  I currently have a Cascading script that unzips the log
files from a local directory while pushing them into HDFS.  This approach,
however, is rather brute force and takes an incredibly long time.  

Has anyone else witnessed this kind of behavior?  I'm running the job up on
EC2 using 10 large instances and version 0.17.0, but I've noticed this issue
locally as well.  Is there, perhaps, a flag I'm not setting in my JobConf,
or an idiosyncrasy  within the way gzipped files are read within HDFS that
would account for this?  I haven't noticed anything unusual in my code or in
the logs that would signify a problem.

I know this question is rather vague, but any help/input would be greatly
appreciated.

Thanks,
Mike
-- 
View this message in context: http://www.nabble.com/Has-anyone-experienced-a-major-lag-reading-gzipped-files--tp19058967p19058967.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Has anyone experienced a major lag reading gzipped files?

[Alex Feinberg <al...@socialmedia.com>]

Smaller maps would finish faster.

>  Interesting, the one map per gzipped file would definitely explain the lag.
> However, how would compressing smaller chunks of the data help?  Wouldn't a
> single map process be launched per chunk, thus resulting in the same problem
> as before?  Sorry if I'm missing something completely obvious.

-- 
Alex Feinberg
Platform Engineer, SocialMedia Networks

Re: Has anyone experienced a major lag reading gzipped files?

[montag <mk...@about.com>]

  Interesting, the one map per gzipped file would definitely explain the lag. 
However, how would compressing smaller chunks of the data help?  Wouldn't a
single map process be launched per chunk, thus resulting in the same problem
as before?  Sorry if I'm missing something completely obvious.  

  Furthermore, is there a way to circumvent the rule programmatically
through the JobConf?  I don't really have a say in how the gzipped files are
initially stored.  

Thanks again,
Mike

Alex Feinberg wrote:
> 
> Hi,
> 
> When using a gzipped file, there is a hard rule of a single map process
> per file
> -- whereas when using a plain text file, the file is "sharded" into
> the amount of maps
> that hadoop finds to be optimal.
> 
> It may help you to (if you want to keep using compressed files), to divide
> the
> files into optimal size chunks (with the number of records that hadoop
> itself would
> divide the file into) and then compress these chunks.
> 
> Thanks,
> 
>>  Has anyone been having issues with Hadoop jobs involving a large
>> collection of gzipped files, specifically on EC2?  I currently have a job
>> set up in which I take as input about 360 gzipped log files within HDFS,
>> totalling to about 30GB of compressed data.  I have noticed that if I
>> leave
>> these files compressed, the JobClient hangs at 0% map, 0% reduce, and
>> will
>> eventually jump to 100% map and reduce (after a rather long time) without
>> having reported any progress in between.
>>
>> However, if I unzip the files before pushing them to HDFS, the job starts
>> almost immediately.  I currently have a Cascading script that unzips the
>> log
>> files from a local directory while pushing them into HDFS.  This
>> approach,
>> however, is rather brute force and takes an incredibly long time.
>>
> 
> -- 
> Alex Feinberg
> Platform Engineer, SocialMedia Networks
> 
> 

-- 
View this message in context: http://www.nabble.com/Has-anyone-experienced-a-major-lag-reading-gzipped-files--tp19058967p19061527.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Has anyone experienced a major lag reading gzipped files?

[Alex Feinberg <al...@socialmedia.com>]

Hi,

When using a gzipped file, there is a hard rule of a single map process per file
-- whereas when using a plain text file, the file is "sharded" into
the amount of maps
that hadoop finds to be optimal.

It may help you to (if you want to keep using compressed files), to divide the
files into optimal size chunks (with the number of records that hadoop
itself would
divide the file into) and then compress these chunks.

Thanks,

>  Has anyone been having issues with Hadoop jobs involving a large
> collection of gzipped files, specifically on EC2?  I currently have a job
> set up in which I take as input about 360 gzipped log files within HDFS,
> totalling to about 30GB of compressed data.  I have noticed that if I leave
> these files compressed, the JobClient hangs at 0% map, 0% reduce, and will
> eventually jump to 100% map and reduce (after a rather long time) without
> having reported any progress in between.
>
> However, if I unzip the files before pushing them to HDFS, the job starts
> almost immediately.  I currently have a Cascading script that unzips the log
> files from a local directory while pushing them into HDFS.  This approach,
> however, is rather brute force and takes an incredibly long time.
>

-- 
Alex Feinberg
Platform Engineer, SocialMedia Networks