You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Paul Wilkinson (JIRA)" <ji...@apache.org> on 2012/07/10 14:44:34 UTC
[jira] [Created] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Paul Wilkinson created HADOOP-8582:
--------------------------------------
Summary: Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
Key: HADOOP-8582
URL: https://issues.apache.org/jira/browse/HADOOP-8582
Project: Hadoop Common
Issue Type: Improvement
Components: io
Environment: Centos 5.8, Java 6 Update 26
Reporter: Paul Wilkinson
Priority: Minor
At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
Exception in thread "main" java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
at test.SequenceReader.read(SequenceReader.java:23)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Paul Wilkinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Wilkinson updated HADOOP-8582:
-----------------------------------
Attachment: HADOOP-8582-1.diff
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Environment: Centos 5.8, Java 6 Update 26
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410567#comment-13410567 ]
Daryn Sharp commented on HADOOP-8582:
-------------------------------------
I think it would make more sense to avoid coupling {{SequenceFile}} against a specific codec. Shouldn't {{GzipCodec}} generate the exception? Perhaps in the ctor?
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413007#comment-13413007 ]
Harsh J commented on HADOOP-8582:
---------------------------------
Daryn,
Its sorta the latter. To be clearer, the reason is this, from HADOOP-538:
{quote}
Arun:
Context: gzip is just zlib algo + extra headers. java.util.zip.GZIP{Input|Output}Stream and hence existing GzipCodec won't work with SequenceFile due the fact that java.util.zip.GZIP{Input|Output}Streams will try to read/write gzip headers in the constructors which won't work in SequenceFiles since we typically read data from disk onto buffers, these buffers are empty on startup/after-reset and cause the java.util.zip.GZIP{Input|Output}Streams to fail.
{quote}
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410619#comment-13410619 ]
Harsh J commented on HADOOP-8582:
---------------------------------
Daryn,
There's a reason that check exists specifically in SequenceFile I think. See https://issues.apache.org/jira/browse/HADOOP-538?focusedCommentId=12444771&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12444771
In other places, GzipCodec works either ways I think (native or Java provided fallback), so we can't push back the thrower to it, as that'd break other apps.
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410877#comment-13410877 ]
Daryn Sharp commented on HADOOP-8582:
-------------------------------------
I skimmed the jira, but I'm not sure I groked it. It sounds like the problem is zlib vs gzip headers? Or that some of the header bytes are gobbled before the non-native gzip stream is created?
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Paul Wilkinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Wilkinson updated HADOOP-8582:
-----------------------------------
Attachment: (was: HADOOP-8582.diff)
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Environment: Centos 5.8, Java 6 Update 26
> Reporter: Paul Wilkinson
> Priority: Minor
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Paul Wilkinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413574#comment-13413574 ]
Paul Wilkinson commented on HADOOP-8582:
----------------------------------------
Harsh, it sounds like you are happy with the approach, so I'll add in those changes and resubmit later today - thanks.
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410312#comment-13410312 ]
Harsh J commented on HADOOP-8582:
---------------------------------
Hi Paul, thanks for filing this and the patch! I've run into this as well.
Patch looks good but can you also add in a test (you can selectively disable loading of native libs via configuration if need be), so we don't regress from this test, if possible?
Also, perhaps we can instead be more specific in the error message (saying "SequenceFile.Reader can't read Gzip compressed files without native-hadoop libraries" or so? Feel free to improve the Writer when you're at it too, if needed)
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Environment: Centos 5.8, Java 6 Update 26
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413489#comment-13413489 ]
Harsh J commented on HADOOP-8582:
---------------------------------
Daryn, are you good with the patch's general approach, given the above?
Paul, will you be sending an updated patch soon? If not, let me know, and I'm happy to tweak on your behalf and add in those changes.
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-8582:
----------------------------
Environment: (was: Centos 5.8, Java 6 Update 26)
Target Version/s: 2.0.1-alpha
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Paul Wilkinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Wilkinson updated HADOOP-8582:
-----------------------------------
Attachment: HADOOP-8582.diff
This should give a more helpful error message.
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Environment: Centos 5.8, Java 6 Update 26
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413774#comment-13413774 ]
Daryn Sharp commented on HADOOP-8582:
-------------------------------------
+1 Sure, I don't particularly like, but I guess it's inevitable due to the design...
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582-1.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Paul Wilkinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Wilkinson updated HADOOP-8582:
-----------------------------------
Description:
At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
{noformat}
Exception in thread "main" java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
at test.SequenceReader.read(SequenceReader.java:23)
{noformat}
was:
At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
Exception in thread "main" java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
at test.SequenceReader.read(SequenceReader.java:23)
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Environment: Centos 5.8, Java 6 Update 26
> Reporter: Paul Wilkinson
> Priority: Minor
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> {noformat}
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8582) Improve error reporting for
GZIP-compressed SequenceFiles with missing native libraries.
Posted by "Paul Wilkinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Wilkinson updated HADOOP-8582:
-----------------------------------
Affects Version/s: 2.0.0-alpha
> Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-8582
> URL: https://issues.apache.org/jira/browse/HADOOP-8582
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.0.0-alpha
> Environment: Centos 5.8, Java 6 Update 26
> Reporter: Paul Wilkinson
> Priority: Minor
> Attachments: HADOOP-8582.diff
>
>
> At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available.
> The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same:
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:67)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:95)
> at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:104)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173)
> at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1493)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1480)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at test.SequenceReader.read(SequenceReader.java:23)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira