You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ben Maurer (JIRA)" <ji...@apache.org> on 2009/02/19 03:57:01 UTC
[jira] Created: (HADOOP-5281) GzipCodec fails second time it is
used in a process
GzipCodec fails second time it is used in a process
---------------------------------------------------
Key: HADOOP-5281
URL: https://issues.apache.org/jira/browse/HADOOP-5281
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.19.0
Environment: 0.19.0 / Linux / amd64
Reporter: Ben Maurer
Priority: Blocker
The following code raises:
java.io.IOException: incorrect header check
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
{quote}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.MapFile;
import org.apache.hadoop.io.SequenceFile.CompressionType;
import org.apache.hadoop.io.compress.GzipCodec;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class GZt extends Configured implements Tool {
private FileSystem fileSystem;
MapFile.Writer newMap(Path p) throws IOException {
return new MapFile.Writer(getConf(), fileSystem, p.toString(),
IntWritable.class, IntWritable.class, CompressionType.BLOCK,
new GzipCodec(), null);
}
@Override
public int run(String[] args) throws Exception {
Path p = new Path(args[0]);
fileSystem = FileSystem.get(getConf());
fileSystem.mkdirs(p);
for (int i = 0; i < 2; i++) {
MapFile.Writer w = newMap(new Path(p, "b" + i));
for (int j = 0; j < 100; j++) {
w.append(new IntWritable(j), new IntWritable(j));
}
w.close();
}
MapFile.Reader rdr = new MapFile.Reader(fileSystem, new Path(p, "b1")
.toString(), getConf());
rdr.get(new IntWritable(1), new IntWritable());
return 0;
}
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(new Configuration(), new GZt(), args);
System.exit(res);
}
}
{quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this.
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 5281-0.patch, 5281-1.patch, 5281-2.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned HADOOP-5281:
-------------------------------------
Assignee: Chris Douglas
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Ben Maurer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682428#action_12682428 ]
Ben Maurer commented on HADOOP-5281:
------------------------------------
No, only with GzipCodec.
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: GZt.java
>
>
> The following code raises:
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {quote}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.MapFile;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> import org.apache.hadoop.io.compress.GzipCodec;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
> public class GZt extends Configured implements Tool {
> private FileSystem fileSystem;
> MapFile.Writer newMap(Path p) throws IOException {
> return new MapFile.Writer(getConf(), fileSystem, p.toString(),
> IntWritable.class, IntWritable.class, CompressionType.BLOCK,
> new GzipCodec(), null);
> }
> @Override
> public int run(String[] args) throws Exception {
> Path p = new Path(args[0]);
> fileSystem = FileSystem.get(getConf());
> fileSystem.mkdirs(p);
> for (int i = 0; i < 2; i++) {
> MapFile.Writer w = newMap(new Path(p, "b" + i));
> for (int j = 0; j < 100; j++) {
> w.append(new IntWritable(j), new IntWritable(j));
> }
> w.close();
> }
> MapFile.Reader rdr = new MapFile.Reader(fileSystem, new Path(p, "b1")
> .toString(), getConf());
> rdr.get(new IntWritable(1), new IntWritable());
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new GZt(), args);
> System.exit(res);
> }
> }
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682745#action_12682745 ]
Arun C Murthy commented on HADOOP-5281:
---------------------------------------
+1
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, 5281-2.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Ben Maurer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ben Maurer updated HADOOP-5281:
-------------------------------
Attachment: GZt.java
Sorry, the file got munged.. here's a full version of the test
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: GZt.java
>
>
> The following code raises:
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {quote}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.MapFile;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> import org.apache.hadoop.io.compress.GzipCodec;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
> public class GZt extends Configured implements Tool {
> private FileSystem fileSystem;
> MapFile.Writer newMap(Path p) throws IOException {
> return new MapFile.Writer(getConf(), fileSystem, p.toString(),
> IntWritable.class, IntWritable.class, CompressionType.BLOCK,
> new GzipCodec(), null);
> }
> @Override
> public int run(String[] args) throws Exception {
> Path p = new Path(args[0]);
> fileSystem = FileSystem.get(getConf());
> fileSystem.mkdirs(p);
> for (int i = 0; i < 2; i++) {
> MapFile.Writer w = newMap(new Path(p, "b" + i));
> for (int j = 0; j < 100; j++) {
> w.append(new IntWritable(j), new IntWritable(j));
> }
> w.close();
> }
> MapFile.Reader rdr = new MapFile.Reader(fileSystem, new Path(p, "b1")
> .toString(), getConf());
> rdr.get(new IntWritable(1), new IntWritable());
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new GZt(), args);
> System.exit(res);
> }
> }
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682395#action_12682395 ]
Owen O'Malley commented on HADOOP-5281:
---------------------------------------
Does this problem still happen when you use DefaultCodec instead of GzipCodec?
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: GZt.java
>
>
> The following code raises:
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {quote}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.MapFile;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> import org.apache.hadoop.io.compress.GzipCodec;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
> public class GZt extends Configured implements Tool {
> private FileSystem fileSystem;
> MapFile.Writer newMap(Path p) throws IOException {
> return new MapFile.Writer(getConf(), fileSystem, p.toString(),
> IntWritable.class, IntWritable.class, CompressionType.BLOCK,
> new GzipCodec(), null);
> }
> @Override
> public int run(String[] args) throws Exception {
> Path p = new Path(args[0]);
> fileSystem = FileSystem.get(getConf());
> fileSystem.mkdirs(p);
> for (int i = 0; i < 2; i++) {
> MapFile.Writer w = newMap(new Path(p, "b" + i));
> for (int j = 0; j < 100; j++) {
> w.append(new IntWritable(j), new IntWritable(j));
> }
> w.close();
> }
> MapFile.Reader rdr = new MapFile.Reader(fileSystem, new Path(p, "b1")
> .toString(), getConf());
> rdr.get(new IntWritable(1), new IntWritable());
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new GZt(), args);
> System.exit(res);
> }
> }
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682579#action_12682579 ]
Hadoop QA commented on HADOOP-5281:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402340/5281-1.patch
against trunk revision 755057.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/93/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/93/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/93/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/93/console
This message is automatically generated.
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, 5281-2.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Attachment: (was: 5281-2.patch)
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, 5281-2.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Attachment: 5281-1.patch
Patch returns different Compressor/Decompressor types for GzipCodec. While one could probably use end() and init() to share ZlibCompressor instances between the codecs, this keeps the existing semantics.
{noformat}
09/03/17 03:28:45 INFO compress.CodecPool: Got brand-new compressor 1049443 GzipCodec/GzipZlibCompressor
09/03/17 03:28:45 INFO compress.CodecPool: Got brand-new compressor 1043272 DefaultCodec/ZlibCompressor
09/03/17 03:28:45 INFO compress.CodecPool: Returned GzipZlibCompressor/1049443
09/03/17 03:28:45 INFO compress.CodecPool: Returned ZlibCompressor/1043272
09/03/17 03:28:45 INFO compress.CodecPool: Got recycled compressor 1049443 GzipCodec/GzipZlibCompressor
09/03/17 03:28:45 INFO compress.CodecPool: Got recycled compressor 1043272 DefaultCodec/ZlibCompressor
{noformat}
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682542#action_12682542 ]
Chris Douglas commented on HADOOP-5281:
---------------------------------------
The compressors are being recycled, but GzipCodec and DefaultCodec are improperly exchanging ZlibCompressors. In a trace printing out identityHashCode for all compressors:
{quote}
09/03/17 01:44:46 INFO compress.CodecPool: Got brand-new compressor *12182618 GzipCodec/ZlibCompressor*
09/03/17 01:44:46 INFO compress.CodecPool: Got brand-new compressor 12376621 DefaultCodec/ZlibCompressor
09/03/17 01:44:46 INFO compress.CodecPool: Returned ZlibCompressor/12182618
09/03/17 01:44:46 INFO compress.CodecPool: Returned ZlibCompressor/12376621
09/03/17 01:44:46 INFO compress.CodecPool: Got recycled compressor 12376621 GzipCodec/ZlibCompressor
09/03/17 01:44:46 INFO compress.CodecPool: Got recycled compressor *12182618 DefaultCodec/ZlibCompressor*
{quote}
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: 5281-0.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Attachment: 5281-2.patch
Added a unit test
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, 5281-2.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Attachment: 5281-2.patch
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, 5281-2.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683086#action_12683086 ]
Hudson commented on HADOOP-5281:
--------------------------------
Integrated in Hadoop-trunk #783 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/783/])
. Prevent sharing incompatible ZlibCompressor instances between GzipCodec and DefaultCodec.
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 5281-0.patch, 5281-1.patch, 5281-2.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Affects Version/s: (was: 0.19.0)
0.18.0
Status: Patch Available (was: Open)
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: 5281-0.patch, 5281-1.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Description:
The attached code (GZt.java) raises:
{noformat}
java.io.IOException: incorrect header check
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
{noformat}
was:
The following code raises:
java.io.IOException: incorrect header check
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
{quote}
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.MapFile;
import org.apache.hadoop.io.SequenceFile.CompressionType;
import org.apache.hadoop.io.compress.GzipCodec;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class GZt extends Configured implements Tool {
private FileSystem fileSystem;
MapFile.Writer newMap(Path p) throws IOException {
return new MapFile.Writer(getConf(), fileSystem, p.toString(),
IntWritable.class, IntWritable.class, CompressionType.BLOCK,
new GzipCodec(), null);
}
@Override
public int run(String[] args) throws Exception {
Path p = new Path(args[0]);
fileSystem = FileSystem.get(getConf());
fileSystem.mkdirs(p);
for (int i = 0; i < 2; i++) {
MapFile.Writer w = newMap(new Path(p, "b" + i));
for (int j = 0; j < 100; j++) {
w.append(new IntWritable(j), new IntWritable(j));
}
w.close();
}
MapFile.Reader rdr = new MapFile.Reader(fileSystem, new Path(p, "b1")
.toString(), getConf());
rdr.get(new IntWritable(1), new IntWritable());
return 0;
}
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(new Configuration(), new GZt(), args);
System.exit(res);
}
}
{quote}
Edited description
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5281) GzipCodec fails second time it is
used in a process
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5281:
----------------------------------
Attachment: 5281-0.patch
GZt.java
When GzipCodec is specified, it's used only for the data, not the index (which always uses DefaultCodec). When the two match, there's no problem.
It's clear that the index is being read with GzipCodec after being written with DefaultCodec (so it expects a header, but doesn't find one). Still trying to determine how that's happening...
This didn't work with previous versions of Hadoop, right?
> GzipCodec fails second time it is used in a process
> ---------------------------------------------------
>
> Key: HADOOP-5281
> URL: https://issues.apache.org/jira/browse/HADOOP-5281
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: 0.19.0 / Linux / amd64
> Reporter: Ben Maurer
> Priority: Blocker
> Attachments: 5281-0.patch, GZt.java, GZt.java
>
>
> The attached code (GZt.java) raises:
> {noformat}
> java.io.IOException: incorrect header check
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
> at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221)
> at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:62)
> at java.io.DataInputStream.readByte(DataInputStream.java:248)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1853)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
> at org.apache.hadoop.io.MapFile$Reader.readIndex(MapFile.java:319)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:435)
> at org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:417)
> at org.apache.hadoop.io.MapFile$Reader.seek(MapFile.java:404)
> at org.apache.hadoop.io.MapFile$Reader.get(MapFile.java:523)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.