You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Trupti Dhavle (JIRA)" <ji...@apache.org> on 2012/06/28 20:14:44 UTC
[jira] [Created] (HADOOP-8540) Compression with non-default codec's
fail on winhadoop
Trupti Dhavle created HADOOP-8540:
-------------------------------------
Summary: Compression with non-default codec's fail on winhadoop
Key: HADOOP-8540
URL: https://issues.apache.org/jira/browse/HADOOP-8540
Project: Hadoop Common
Issue Type: Bug
Reporter: Trupti Dhavle
I was trying to run the compression test case with GZipCodec. However the test fails with NPE. It is running fine with DefaultCodec.
With text input
c:\Workspace\winhadoopqe>c:\hdp\branch-1-win\bin\hadoop jar c:\hdp\branch-1-win\
build\hadoop-examples-1.1.0-SNAPSHOT.jar sort "-Dmapred.compress.map.output=true
" "-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
" "-Dmapred.output.compress=true" "-Dmapred.output.compression.type=NONE" "-Dm
apred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text Compression/texti
nput Compression/textoutput-123
Running on 1 nodes to sort from hdfs://localhost:8020/user/Administrator/Compres
sion/textinput into hdfs://localhost:8020/user/Administrator/Compression/textout
put-123 with 1 reduces.
Job started: Thu Jun 28 10:02:18 PDT 2012
12/06/28 10:02:18 INFO mapred.FileInputFormat: Total input paths to process : 1
12/06/28 10:02:18 INFO mapred.JobClient: Running job: job_201206271409_0045
12/06/28 10:02:19 INFO mapred.JobClient: map 0% reduce 0%
12/06/28 10:02:36 INFO mapred.JobClient: Task Id : attempt_201206271409_0045_m_0
00000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:102)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask
.java:1407)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1
298)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1103)
at org.apache.hadoop.mapred.Child.main(Child.java:265)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8540) Compression with non-default
codec's fail on Windows
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Mitic resolved HADOOP-8540.
--------------------------------
Resolution: Fixed
Support for zlib compression library was added with HADOOP-8564.
To enable zlib compression codecs on Windows:
- Download zlib sources and set ZLIB_HOME in the environment before building Hadoop on Windows (this step is needed for build to generate the corresponding stubs in hadoop.dll)
- Install zlib1.dll to system32 (with HADOOP-8907 you can also install it next to hadoop.dll)
> Compression with non-default codec's fail on Windows
> ----------------------------------------------------
>
> Key: HADOOP-8540
> URL: https://issues.apache.org/jira/browse/HADOOP-8540
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Trupti Dhavle
>
> I was trying to run the compression test case with GZipCodec. However the test fails with NPE. It is running fine with DefaultCodec.
> With text input
> c:\Workspace\winhadoopqe>c:\hdp\branch-1-win\bin\hadoop jar c:\hdp\branch-1-win\
> build\hadoop-examples-1.1.0-SNAPSHOT.jar sort "-Dmapred.compress.map.output=true
> " "-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
> " "-Dmapred.output.compress=true" "-Dmapred.output.compression.type=NONE" "-Dm
> apred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
> org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text Compression/texti
> nput Compression/textoutput-123
> Running on 1 nodes to sort from hdfs://localhost:8020/user/Administrator/Compres
> sion/textinput into hdfs://localhost:8020/user/Administrator/Compression/textout
> put-123 with 1 reduces.
> Job started: Thu Jun 28 10:02:18 PDT 2012
> 12/06/28 10:02:18 INFO mapred.FileInputFormat: Total input paths to process : 1
> 12/06/28 10:02:18 INFO mapred.JobClient: Running job: job_201206271409_0045
> 12/06/28 10:02:19 INFO mapred.JobClient: map 0% reduce 0%
> 12/06/28 10:02:36 INFO mapred.JobClient: Task Id : attempt_201206271409_0045_m_0
> 00000_0, Status : FAILED
> java.lang.NullPointerException
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:102)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask
> .java:1407)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1
> 298)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
> tion.java:1103)
> at org.apache.hadoop.mapred.Child.main(Child.java:265)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8540) Compression with non-default codec's
fail on Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated HADOOP-8540:
-------------------------------
Summary: Compression with non-default codec's fail on Windows (was: Compression with non-default codec's fail on winhadoop)
> Compression with non-default codec's fail on Windows
> ----------------------------------------------------
>
> Key: HADOOP-8540
> URL: https://issues.apache.org/jira/browse/HADOOP-8540
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Trupti Dhavle
>
> I was trying to run the compression test case with GZipCodec. However the test fails with NPE. It is running fine with DefaultCodec.
> With text input
> c:\Workspace\winhadoopqe>c:\hdp\branch-1-win\bin\hadoop jar c:\hdp\branch-1-win\
> build\hadoop-examples-1.1.0-SNAPSHOT.jar sort "-Dmapred.compress.map.output=true
> " "-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
> " "-Dmapred.output.compress=true" "-Dmapred.output.compression.type=NONE" "-Dm
> apred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
> org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text Compression/texti
> nput Compression/textoutput-123
> Running on 1 nodes to sort from hdfs://localhost:8020/user/Administrator/Compres
> sion/textinput into hdfs://localhost:8020/user/Administrator/Compression/textout
> put-123 with 1 reduces.
> Job started: Thu Jun 28 10:02:18 PDT 2012
> 12/06/28 10:02:18 INFO mapred.FileInputFormat: Total input paths to process : 1
> 12/06/28 10:02:18 INFO mapred.JobClient: Running job: job_201206271409_0045
> 12/06/28 10:02:19 INFO mapred.JobClient: map 0% reduce 0%
> 12/06/28 10:02:36 INFO mapred.JobClient: Task Id : attempt_201206271409_0045_m_0
> 00000_0, Status : FAILED
> java.lang.NullPointerException
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:102)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask
> .java:1407)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1
> 298)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
> tion.java:1103)
> at org.apache.hadoop.mapred.Child.main(Child.java:265)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira