You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ravi Gummadi (JIRA)" <ji...@apache.org> on 2009/10/27 14:02:59 UTC
[jira] Created: (HADOOP-6335) Support reading of concatenated gzip
files
Support reading of concatenated gzip files
------------------------------------------
Key: HADOOP-6335
URL: https://issues.apache.org/jira/browse/HADOOP-6335
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6335) Support reading of concatenated gzip
and bzip2 files
Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Ciemiewicz updated HADOOP-6335:
-------------------------------------
Summary: Support reading of concatenated gzip and bzip2 files (was: Support reading of concatenated gzip files)
bzip2 also supports concatenation of individually compressed files.
bzcat has absolutely no problem reading all of the data in a single concatenated file.
Unfortuantely, M/R loses records. My Hadoop Streaming and Pig jobs saw only 2% of the data in the concatenated file. That's a 98% data loss. Not very good. There should be no data loss in reading these concatenated files.
> Support reading of concatenated gzip and bzip2 files
> ----------------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6335) Support reading of concatenated
gzip and bzip2 files
Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871854#action_12871854 ]
David Ciemiewicz commented on HADOOP-6335:
------------------------------------------
@Chris Douglas
Yes, I have verified that in the version of hadoop that I am using, this is not fixed: Hadoop 0.20.10.0.1004192217
I cannot speak as to whether or not this is fixed in the trunk.
I created two files file1.bz2 and file2.bz2 and concatenated them into file12.bz2
-bash-3.1$ bzcat file12.bz2
contents of file1.bz2
contents of file2.bz2
I then run a simple pig script to dump the contents of this file:
-bash-3.1$ cat concat.pig
A = load 'file12.bz2' using PigStorage();
dump A;
The output below shows that only the first file in the concatenation is read. The subsequent file is not read.
-bash-3.1$ pig -Dmapred.job.queue.name=... concat.pig
USING: /grid/0/gs/pig/current
2010-05-26 17:54:06,501 [main] INFO org.apache.pig.Main - Logging error messages to: /homes/ciemo/.../pig_1274896446499.log
2010-05-26 17:54:06,750 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://...:8020
2010-05-26 17:54:07,001 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: ...:50300
2010-05-26 17:54:07,830 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2010-05-26 17:54:07,830 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2010-05-26 17:54:08,804 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2010-05-26 17:54:08,835 [Thread-9] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2010-05-26 17:54:09,834 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Cannot get jobid for this job
2010-05-26 17:54:32,745 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2010-05-26 17:55:09,412 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2010-05-26 17:55:09,412 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Successfully stored result in: "hdfs://...
2010-05-26 17:55:11,158 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Records written : 1
2010-05-26 17:55:11,159 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Bytes written : 34
2010-05-26 17:55:11,159 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
(contents of file1.bz2)
The dump should have shown both file1.bz2 and file2.bz2
> Support reading of concatenated gzip and bzip2 files
> ----------------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6335) Support reading of concatenated gzip
and bzip2 files
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved HADOOP-6335.
-----------------------------------
Assignee: (was: Ravi Gummadi)
Resolution: Duplicate
Duplicate of MAPREDUCE-469
> Support reading of concatenated gzip and bzip2 files
> ----------------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6335) Support reading of concatenated
gzip files
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772183#action_12772183 ]
Koji Noguchi commented on HADOOP-6335:
--------------------------------------
Is this a duplicate of MAPREDUCE-469 ?
Also, can we make a fix for the native compression as well (maybe in a new Jira)?
> Support reading of concatenated gzip files
> ------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6335) Support reading of concatenated
gzip and bzip2 files
Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871924#action_12871924 ]
David Ciemiewicz commented on HADOOP-6335:
------------------------------------------
Actually Pig 0.7 which is top of trunk does not use it's own bzip2 implementation anymore - it's supposed to be using the Hadoop map-reduce underpinnings. So it would benefit. But it also exhibits the problem.
I don't do my own builds of hadoop so don't have any means for verifying that it is fixed in the trunk myself.
> Support reading of concatenated gzip and bzip2 files
> ----------------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6335) Support reading of concatenated
gzip and bzip2 files
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871533#action_12871533 ]
Chris Douglas commented on HADOOP-6335:
---------------------------------------
Have you verified that concatenated bzip2 files are not correctly handled in trunk? HADOOP-4012 claims to have fixed this.
> Support reading of concatenated gzip and bzip2 files
> ----------------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6335) Support reading of concatenated
gzip and bzip2 files
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871904#action_12871904 ]
Chris Douglas commented on HADOOP-6335:
---------------------------------------
Hi David-
This is known not to work in 0.20 (and its variants). My suggestion was that you verify that this is fixed in trunk and 0.21 by HADOOP-4012. IIRC, Pig uses its own bzip2 implementation, so it would not benefit either way.
> Support reading of concatenated gzip and bzip2 files
> ----------------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6335) Support reading of concatenated
gzip files
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770489#action_12770489 ]
Ravi Gummadi commented on HADOOP-6335:
--------------------------------------
This is a known issue that java's GZIPInputStream can't handle concatenated gzip files fully. The bug is http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4691425
There is a workaround(MultiMemberGZIPInputStream) mentioned there on that bug. Can we take that workaround and change GzipCodec.GzipInputStream to support concatenated gzip files ? Any licensing issues if we use it in this project ?
> Support reading of concatenated gzip files
> ------------------------------------------
>
> Key: HADOOP-6335
> URL: https://issues.apache.org/jira/browse/HADOOP-6335
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.