You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Amir Hellershtein <am...@mellanox.com> on 2014/04/28 14:22:35 UTC
Disable Checksum
Hello,
I wish to disable checksum calculation & verification in Hadoop's framework,
I run the TestDfsIO application with the parameter dfs.checksum.type=NULL,
To my best understanding, this was supposed to turnoff check summing in the framework,
But I received the following error -
...
...
...
14/04/28 14:54:26.433 INFO hdfs.DFSOutputStream (1349) 96232.106, 96232.10464359 : Vanilla DFSOutputStream constructed successfully.
java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:184)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:172)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:148)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:137)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1810)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:99)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1240)
at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:750)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:649)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
14/04/28 14:54:26.581 INFO fs.FileSystem (585) 96232.249, 0.585955522 : Vanilla distributeFileSystem was closed.
Is seems that the framework does not support disabling the checksum by the above flag,
Does anybody knows why I get this error?
Is there another way to disable checksum?
Thanks
Amir
Re: Disable Checksum
Posted by Hu Liu <hl...@gopivotal.com>.
Hi Amir,
I'm also working on how to disable the checksum, it's known issue of apache
hadoop, you can find more details from the following apache JIRA :
https://issues.apache.org/jira/browse/HADOOP-9114
https://issues.apache.org/jira/browse/HDFS-5761
On Mon, Apr 28, 2014 at 8:22 PM, Amir Hellershtein <am...@mellanox.com>wrote:
> Hello,
>
>
>
> I wish to disable checksum calculation & verification in Hadoop's
> framework,
>
> I run the TestDfsIO application with the parameter dfs.checksum.type=NULL,
>
> To my best understanding, this was supposed to turnoff check summing in
> the framework,
>
> But I received the following error -
>
>
>
> ...
>
> ...
>
> ...
>
> 14/04/28 14:54:26.433 INFO hdfs.DFSOutputStream (1349) 96232.106,
> 96232.10464359 : Vanilla DFSOutputStream constructed successfully.
>
> java.lang.ArrayIndexOutOfBoundsException: 0
>
> at
> org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:184)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:172)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:148)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1810)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:99)
>
> at
> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1240)
>
> at
> org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
>
> at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:750)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:649)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
>
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
>
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
> 14/04/28 14:54:26.581 INFO fs.FileSystem (585) 96232.249, 0.585955522 :
> Vanilla distributeFileSystem was closed.
>
>
>
> Is seems that the framework does not support disabling the checksum by the
> above flag,
>
> Does anybody knows why I get this error?
>
> Is there another way to disable checksum?
>
>
>
> Thanks
>
>
>
> *Amir*
>
--
Best Regards,
Hu Liu
Re: Disable Checksum
Posted by Hu Liu <hl...@gopivotal.com>.
Hi Amir,
I'm also working on how to disable the checksum, it's known issue of apache
hadoop, you can find more details from the following apache JIRA :
https://issues.apache.org/jira/browse/HADOOP-9114
https://issues.apache.org/jira/browse/HDFS-5761
On Mon, Apr 28, 2014 at 8:22 PM, Amir Hellershtein <am...@mellanox.com>wrote:
> Hello,
>
>
>
> I wish to disable checksum calculation & verification in Hadoop's
> framework,
>
> I run the TestDfsIO application with the parameter dfs.checksum.type=NULL,
>
> To my best understanding, this was supposed to turnoff check summing in
> the framework,
>
> But I received the following error -
>
>
>
> ...
>
> ...
>
> ...
>
> 14/04/28 14:54:26.433 INFO hdfs.DFSOutputStream (1349) 96232.106,
> 96232.10464359 : Vanilla DFSOutputStream constructed successfully.
>
> java.lang.ArrayIndexOutOfBoundsException: 0
>
> at
> org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:184)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:172)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:148)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1810)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:99)
>
> at
> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1240)
>
> at
> org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
>
> at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:750)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:649)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
>
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
>
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
> 14/04/28 14:54:26.581 INFO fs.FileSystem (585) 96232.249, 0.585955522 :
> Vanilla distributeFileSystem was closed.
>
>
>
> Is seems that the framework does not support disabling the checksum by the
> above flag,
>
> Does anybody knows why I get this error?
>
> Is there another way to disable checksum?
>
>
>
> Thanks
>
>
>
> *Amir*
>
--
Best Regards,
Hu Liu
Re: Disable Checksum
Posted by Hu Liu <hl...@gopivotal.com>.
Hi Amir,
I'm also working on how to disable the checksum, it's known issue of apache
hadoop, you can find more details from the following apache JIRA :
https://issues.apache.org/jira/browse/HADOOP-9114
https://issues.apache.org/jira/browse/HDFS-5761
On Mon, Apr 28, 2014 at 8:22 PM, Amir Hellershtein <am...@mellanox.com>wrote:
> Hello,
>
>
>
> I wish to disable checksum calculation & verification in Hadoop's
> framework,
>
> I run the TestDfsIO application with the parameter dfs.checksum.type=NULL,
>
> To my best understanding, this was supposed to turnoff check summing in
> the framework,
>
> But I received the following error -
>
>
>
> ...
>
> ...
>
> ...
>
> 14/04/28 14:54:26.433 INFO hdfs.DFSOutputStream (1349) 96232.106,
> 96232.10464359 : Vanilla DFSOutputStream constructed successfully.
>
> java.lang.ArrayIndexOutOfBoundsException: 0
>
> at
> org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:184)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:172)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:148)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1810)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:99)
>
> at
> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1240)
>
> at
> org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
>
> at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:750)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:649)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
>
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
>
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
> 14/04/28 14:54:26.581 INFO fs.FileSystem (585) 96232.249, 0.585955522 :
> Vanilla distributeFileSystem was closed.
>
>
>
> Is seems that the framework does not support disabling the checksum by the
> above flag,
>
> Does anybody knows why I get this error?
>
> Is there another way to disable checksum?
>
>
>
> Thanks
>
>
>
> *Amir*
>
--
Best Regards,
Hu Liu
Re: Disable Checksum
Posted by Hu Liu <hl...@gopivotal.com>.
Hi Amir,
I'm also working on how to disable the checksum, it's known issue of apache
hadoop, you can find more details from the following apache JIRA :
https://issues.apache.org/jira/browse/HADOOP-9114
https://issues.apache.org/jira/browse/HDFS-5761
On Mon, Apr 28, 2014 at 8:22 PM, Amir Hellershtein <am...@mellanox.com>wrote:
> Hello,
>
>
>
> I wish to disable checksum calculation & verification in Hadoop's
> framework,
>
> I run the TestDfsIO application with the parameter dfs.checksum.type=NULL,
>
> To my best understanding, this was supposed to turnoff check summing in
> the framework,
>
> But I received the following error -
>
>
>
> ...
>
> ...
>
> ...
>
> 14/04/28 14:54:26.433 INFO hdfs.DFSOutputStream (1349) 96232.106,
> 96232.10464359 : Vanilla DFSOutputStream constructed successfully.
>
> java.lang.ArrayIndexOutOfBoundsException: 0
>
> at
> org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:184)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:172)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:148)
>
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:137)
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1810)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:99)
>
> at
> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1240)
>
> at
> org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
>
> at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:750)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:649)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
>
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
>
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:601)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
> 14/04/28 14:54:26.581 INFO fs.FileSystem (585) 96232.249, 0.585955522 :
> Vanilla distributeFileSystem was closed.
>
>
>
> Is seems that the framework does not support disabling the checksum by the
> above flag,
>
> Does anybody knows why I get this error?
>
> Is there another way to disable checksum?
>
>
>
> Thanks
>
>
>
> *Amir*
>
--
Best Regards,
Hu Liu