You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by sonali <so...@cybage.com> on 2010/05/27 16:22:02 UTC
Regading Benchmark test for Hadoop
Hi,
I am running Benchmarks for Testing performance of Hadoop cluster, I
need to know if there are any tests that are crucial or most widely used?
regards
Sonali Gavhane
"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment."
www.cybage.com
Regarding Benchmark test for Hadoop
Posted by sonali <so...@cybage.com>.
Hi,
> I am running Benchmarks for Testing performance of Hadoop cluster,
> i am getting following error while executing the
> *DistributedFSCheck** in hadoop-test.jar*
synatx i am using is : uhadoop@yogeshk-u:/usr/local/hadoop$ bin/hadoop
jar /usr/local/hadoop/hadoop-0.20.2-test.jar
DistributedFSCheck -write -nrFiles 1
-fileSize 10 -bufferSize 20
Error :
java.lang.StringIndexOutOfBoundsException: String index out of
range: -1
at
java.lang.String.substring(String.java:1937)
at
org.apache.hadoop.fs.DistributedFSCheck.analyzeResult(DistributedFSCheck.java:301)
at
org.apache.hadoop.fs.DistributedFSCheck.main(DistributedFSCheck.java:247)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:81)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Any help will be appreciated.
Thanks and regards,
Sonali Gavhane
"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment."
www.cybage.com
Re: Regading Benchmark test for Hadoop
Posted by Josh Patterson <jo...@cloudera.com>.
Hey Sonali,
A good start is Terrasort:
# generate 10m records
hadoop jar $HADOOP_HOME/hadoop-*-examples.jar teragen 10000000 input-dir
# run terasort
hadoop jar $HADOOP_HOME/hadoop-*-examples.jar terasort input-dir output-dir
Which will push your setup some. Its not a complete test, but its a good
first test!
Josh Patterson
Solutions Architect
Cloudera
On Thu, May 27, 2010 at 10:22 AM, sonali <so...@cybage.com> wrote:
> Hi,
> I am running Benchmarks for Testing performance of Hadoop cluster, I need
> to know if there are any tests that are crucial or most widely used?
>
> regards
> Sonali Gavhane
>
> "Legal Disclaimer: This electronic message and all contents contain
> information from Cybage Software Private Limited which may be privileged,
> confidential, or otherwise protected from disclosure. The information is
> intended to be for the addressee(s) only. If you are not an addressee, any
> disclosure, copy, distribution, or use of the contents of this message is
> strictly prohibited. If you have received this electronic message in error
> please notify the sender by reply e-mail to and destroy the original message
> and all copies. Cybage has taken every reasonable precaution to minimize the
> risk of malicious content in the mail, but is not liable for any damage you
> may sustain as a result of any malicious content in this e-mail. You should
> carry out your own malicious content checks before opening the e-mail or
> attachment." www.cybage.com
>
>
>
Regarding Benchmark test for Hadoop
Posted by sonali <so...@cybage.com>.
Hi,
> I am running Benchmarks for Testing performance of Hadoop cluster,
> i am getting following error while executing the *DFSCIOTest in
> hadoop-test.jar*
*** File /usr/local/hadoop/libhdfs/libhdfs.so.1 does not exist
any help will be appreciated.
> Thanks and regards,
> Sonali Gavhane
>
"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment."
www.cybage.com
Regarding Benchmark test for Hadoop
Posted by sonali <so...@cybage.com>.
Hi,
> I am running Benchmarks for Testing performance of Hadoop cluster,
> i am getting following error while executing the *DFSCIOTest in
> hadoop-test.jar*
*** File /usr/local/hadoop/libhdfs/libhdfs.so.1 does not exist
any help will be appreciated.
> Thanks and regards,
> Sonali Gavhane
>
"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment."
www.cybage.com
Re: Regading Benchmark test for Hadoop
Posted by Josh Patterson <jo...@cloudera.com>.
Hey Sonali,
A good start is Terrasort:
# generate 10m records
hadoop jar $HADOOP_HOME/hadoop-*-examples.jar teragen 10000000 input-dir
# run terasort
hadoop jar $HADOOP_HOME/hadoop-*-examples.jar terasort input-dir output-dir
Which will push your setup some. Its not a complete test, but its a good
first test!
Josh Patterson
Solutions Architect
Cloudera
On Thu, May 27, 2010 at 10:22 AM, sonali <so...@cybage.com> wrote:
> Hi,
> I am running Benchmarks for Testing performance of Hadoop cluster, I need
> to know if there are any tests that are crucial or most widely used?
>
> regards
> Sonali Gavhane
>
> "Legal Disclaimer: This electronic message and all contents contain
> information from Cybage Software Private Limited which may be privileged,
> confidential, or otherwise protected from disclosure. The information is
> intended to be for the addressee(s) only. If you are not an addressee, any
> disclosure, copy, distribution, or use of the contents of this message is
> strictly prohibited. If you have received this electronic message in error
> please notify the sender by reply e-mail to and destroy the original message
> and all copies. Cybage has taken every reasonable precaution to minimize the
> risk of malicious content in the mail, but is not liable for any damage you
> may sustain as a result of any malicious content in this e-mail. You should
> carry out your own malicious content checks before opening the e-mail or
> attachment." www.cybage.com
>
>
>
Regarding Benchmark test for Hadoop
Posted by sonali <so...@cybage.com>.
Hi,
> I am running Benchmarks for Testing performance of Hadoop cluster,
> i am getting following error while executing the
> *DistributedFSCheck** in hadoop-test.jar*
synatx i am using is : uhadoop@yogeshk-u:/usr/local/hadoop$ bin/hadoop
jar /usr/local/hadoop/hadoop-0.20.2-test.jar
DistributedFSCheck -write -nrFiles 1
-fileSize 10 -bufferSize 20
Error :
java.lang.StringIndexOutOfBoundsException: String index out of
range: -1
at
java.lang.String.substring(String.java:1937)
at
org.apache.hadoop.fs.DistributedFSCheck.analyzeResult(DistributedFSCheck.java:301)
at
org.apache.hadoop.fs.DistributedFSCheck.main(DistributedFSCheck.java:247)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:81)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Any help will be appreciated.
Thanks and regards,
Sonali Gavhane
"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment."
www.cybage.com