You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by llpind <so...@hotmail.com> on 2009/06/09 18:25:12 UTC
PerformanceEvaluation test
Hey all,
getting started with map/reduce jobs. Figured I'd try the
PerformanceEvaluation test program first.
After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh, I issue
the following command:
hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
sequentialWrite 4
and get the following output:
09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/06/09 09:20:56 INFO mapred.JobClient: Running job: job_200906090918_0001
09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
at
org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
at
org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
at
org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
What am I doing wrong...?
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
cambridgemike wrote:
>
>
> -tried moving hbase-0.19.2.jar to the hadoop/lib folder of all the slave
> machines.
>
>
Hmm thats weird. Moving the hbase jars solved my issue. go to the job
tracker UI, and look at what machine is throwing the exception, and make
sure you have hbase jars in the hadoop/lib on all boxes (jobtracker as
well). restart Hadoop
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23975204.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by cambridgemike <sa...@gmail.com>.
Hi, I'm having almost the exact same problem (this rowcounter jar is one I
compiled myself)
./bin/hadoop jar rowcounter.jar org.myorg.RowCounter /user/myUser/output/
TABLE_NAME
09/06/10 19:44:19 INFO mapred.TableInputFormatBase: split:
0->domain:,18226263
09/06/10 19:44:19 INFO mapred.TableInputFormatBase: split:
1->domain:18226263,
09/06/10 19:44:19 INFO mapred.JobClient: Running job: job_200906101852_0004
09/06/10 19:44:20 INFO mapred.JobClient: map 0% reduce 0%
09/06/10 19:44:24 INFO mapred.JobClient: Task Id :
attempt_200906101852_0004_m_000000_0, Status : FAILED
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
After reading through this forum I:
-fiddled with my HADOOP_CLASSPATH in hadoop/conf/hadoop-env.sh
-tried restarting hbase and hadoop, and finally,
-tried moving hbase-0.19.2.jar to the hadoop/lib folder of all the slave
machines.
Still not working though, what exactly was required to get this to work?
Cheers,
Mike
llpind wrote:
>
> Hey Andy,
>
> Yeah I eventually did that instead. It just bothers me why it doesn't
> work otherwise. Anyway, thanks to all
>
>
> Andrew Purtell-2 wrote:
>>
>> For what it's worth, I deploy HBase jars into the Hadoop library
>> directory so I don't have to deal with this. Way back I had classpath
>> problems with 'hadoop jar', and not just with HBase. That may not be a
>> satisfactory answer, but it works for me.
>>
>> - Andy
>>
>>
>
>
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23973184.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
Hey Andy,
Yeah I eventually did that instead. It just bothers me why it doesn't work
otherwise. Anyway, thanks to all
Andrew Purtell-2 wrote:
>
> For what it's worth, I deploy HBase jars into the Hadoop library directory
> so I don't have to deal with this. Way back I had classpath problems with
> 'hadoop jar', and not just with HBase. That may not be a satisfactory
> answer, but it works for me.
>
> - Andy
>
>
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23951088.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by Andrew Purtell <ap...@apache.org>.
For what it's worth, I deploy HBase jars into the Hadoop library directory so I don't have to deal with this. Way back I had classpath problems with 'hadoop jar', and not just with HBase. That may not be a satisfactory answer, but it works for me.
- Andy
________________________________
From: llpind <so...@hotmail.com>
To: hbase-user@hadoop.apache.org
Sent: Tuesday, June 9, 2009 11:23:46 AM
Subject: Re: PerformanceEvaluation test
I made an executable jar with all the jars in a lib folder with each listed
in the MANIFEST file under Class-Path. Looks like this:
Manifest-Version: 1.0
Class-Path: lib/ojdbc14.jar lib/commons-cli-2.0-SNAPSHOT.jar lib/comm
ons-httpclient-3.0.1.jar lib/commons-loggin-1.0.4.jar lib/commons-l
ogging-api-1.0.4.jar lib/commons-math-1.1.jar lib/jetty-5.1.4.jar
lib/jruby-complete-1.1.6.jar lib/junit-3.8.1.jar lib/libthrift-2008
0411p1.jar lib/log4j-1.2.13.jar lib/lucene-core-2.2.0.jar lib/serv
let-api.jar lib/xmlenc-0.52.jar lib/zookeeper-3.0.1.jar lib lib/had
oop-0.19.1-core.jar lib/hadoop-0.19.1-test.jar lib/hbase-0.19.3.jar
lib/hbase-0.19.3-test.jar
Main-Class: com.techsys.hbase.test.ColumnCountMapReduce
then i ran the executable from like this:
hadoop19/bin/hadoop jar dev/columncounter/columncounter.jar inputTableName
Errors:
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23948665.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
I made an executable job with all the jars in a lib folder with each listed
in the MANIFEST file under Class-Path. Looks like this:
Manifest-Version: 1.0
Class-Path: lib/ojdbc14.jar lib/commons-cli-2.0-SNAPSHOT.jar lib/comm
ons-httpclient-3.0.1.jar lib/commons-loggin-1.0.4.jar lib/commons-l
ogging-api-1.0.4.jar lib/commons-math-1.1.jar lib/jetty-5.1.4.jar
lib/jruby-complete-1.1.6.jar lib/junit-3.8.1.jar lib/libthrift-2008
0411p1.jar lib/log4j-1.2.13.jar lib/lucene-core-2.2.0.jar lib/serv
let-api.jar lib/xmlenc-0.52.jar lib/zookeeper-3.0.1.jar lib lib/had
oop-0.19.1-core.jar lib/hadoop-0.19.1-test.jar lib/hbase-0.19.3.jar
lib/hbase-0.19.3-test.jar
Main-Class: com.techsys.hbase.test.ColumnCountMapReduce
then i ran the executable from like this:
hadoop19/bin/hadoop jar dev/columncounter/columncounter.jar inputTableName
Errors:
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23948665.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
I'm really at a lost here. I even changed it to the full path:
export
HADOOP_CLASSPATH=/home/hadoop/hbase193/hbase-0.19.3.jar:/home/hadoop/hbase193/hbase-0.19.3-test.jar:/sharedfs/hbase-config/conf
export
HADOOP_CLASSPATH=/home/hadoop/hbase193/hbase-0.19.3.jar:/home/hadoop/hbase193/hbase-0.19.3-test.jar:/home/hadoop/hbase193/conf
restarted cluster and still same problem.
stack-3 wrote:
>
> That looks fine but there is something in your setup that is off. For
> example, HBASE_HOME is being expanded in the below properly on all hosts?
> You are ssh forwarding this environment variable? (if you are using this
> variable name).
> St.Ack
>
> On Tue, Jun 9, 2009 at 10:47 AM, llpind <so...@hotmail.com> wrote:
>
>>
>> Yeah that was my first thought. rowcounter and PerformanceEvaluation
>> work.
>> here is my HADOOP_CLASSPATH:
>> ===================================
>> export
>>
>> HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:/sharedfs/hbase-config/conf
>> ===================================
>> slave boxes reference a shared conf:
>> export
>>
>> HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:$HBASE_HOME/conf
>> ===================================
>> where HBASE_HOME=/home/hadoop/hbase193. this is the same path on all
>> boxes.
>>
>> I run my .class path from the default package like so:
>>
>> hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
>>
>>
>>
>>
>> stack-3 wrote:
>> >
>> > Thats a CLASSPATH issue. For sure the hbase jars are out on all nodes
>> of
>> > your cluster? For sure the paths are correct?
>> > St.Ack
>> >
>> > On Tue, Jun 9, 2009 at 10:28 AM, llpind <so...@hotmail.com> wrote:
>> >
>> >>
>> >> Yep. Thanks, I didn't think to look there :). I do have another
>> issue
>> >> when
>> >> running my own map/reduce. I've got the jar files in HADOOP_CLASSPATH
>> as
>> >> i
>> >> mentioned above, but when i run the job below is what i get. The
>> >> ClassDefNotFoundError happens on TableSplit multiple times.
>> >> ================================================
>> >>
>> >> hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
>> >> 09/06/09 10:26:08 WARN mapred.JobClient: No job jar file set. User
>> >> classes
>> >> may not be found. See JobConf(Class) or JobConf#setJar(String).
>> >> 09/06/09 10:26:11 INFO mapred.JobClient: Running job:
>> >> job_200906090918_0005
>> >> 09/06/09 10:26:12 INFO mapred.JobClient: map 0% reduce 0%
>> >> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
>> >> attempt_200906090918_0005_m_000000_0, Status : FAILED
>> >> java.io.IOException: Split class
>> >> org.apache.hadoop.hbase.mapred.TableSplit
>> >> not found
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
>> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> >> Caused by: java.lang.ClassNotFoundException:
>> >> org.apache.hadoop.hbase.mapred.TableSplit
>> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >> at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> >> at
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> >> at java.lang.Class.forName0(Native Method)
>> >> at java.lang.Class.forName(Class.java:247)
>> >> at
>> >>
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
>> >> ... 1 more
>> >>
>> >> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
>> >> attempt_200906090918_0005_m_000004_0, Status : FAILED
>> >> java.io.IOException: Split class
>> >> org.apache.hadoop.hbase.mapred.TableSplit
>> >> not found
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
>> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> >> Caused by: java.lang.ClassNotFoundException:
>> >> org.apache.hadoop.hbase.mapred.TableSplit
>> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >> at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> >> at
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> >> at java.lang.Class.forName0(Native Method)
>> >> at java.lang.Class.forName(Class.java:247)
>> >> at
>> >>
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
>> >> ... 1 more
>> >>
>> >>
>> >> stack-3 wrote:
>> >> >
>> >> > It looks pretty obvious what your issue is.
>> >> >
>> >> > My guess is that your child process needs its heap size bumped up.
>> >> >
>> >> > Look for the child parameters configuration in your
>> hadoop-default.xml
>> >> (or
>> >> > your mapred-default.xml if 0.20.x hadoop).
>> >> >
>> >> > St.Ack
>> >> >
>> >> > On Tue, Jun 9, 2009 at 10:05 AM, llpind <so...@hotmail.com>
>> wrote:
>> >> >
>> >> >>
>> >> >> Hmm...jobtracker UI has the following:
>> >> >>
>> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> >> >> contact
>> >> >> region server 192.168.0.195:60020 for region
>> TestTable,,1244499094604,
>> >> >> row
>> >> >> '0000010477', but failed after 10 attempts.
>> >> >> Exceptions:
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >>
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
>> >> >> at
>> >> >>
>> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
>> >> >> at
>> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
>> >> >> at
>> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
>> >> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>> >> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> >> >>
>> >> >> =================================================
>> >> >>
>> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> >> >> contact
>> >> >> region server 192.168.0.195:60020 for region
>> TestTable,,1244499094604,
>> >> >> row
>> >> >> '0002107629', but failed after 10 attempts.
>> >> >> Exceptions:
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >>
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
>> >> >> at
>> >> >>
>> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
>> >> >> at
>> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
>> >> >> at
>> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
>> >> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>> >> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> stack-3 wrote:
>> >> >> >
>> >> >> > Look at why the job failed in the jobtracker UI -- usually on
>> port
>> >> >> 50030.
>> >> >> > Looks like your job launched fine. You have the conf dir on the
>> >> >> > HADOOP_CLASSPATH so the MR job can find hbase?
>> >> >> > St.Ack
>> >> >> >
>> >> >> > On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com>
>> >> wrote:
>> >> >> >
>> >> >> >>
>> >> >> >> Hey all,
>> >> >> >>
>> >> >> >> getting started with map/reduce jobs. Figured I'd try the
>> >> >> >> PerformanceEvaluation test program first.
>> >> >> >>
>> >> >> >> After adding HBase jar files to HADOOP_CLASSPATH in
>> hadoop-env.sh,
>> >> I
>> >> >> >> issue
>> >> >> >> the following command:
>> >> >> >>
>> >> >> >> hadoop19/bin/hadoop
>> org.apache.hadoop.hbase.PerformanceEvaluation
>> >> >> >> sequentialWrite 4
>> >> >> >>
>> >> >> >> and get the following output:
>> >> >> >>
>> >> >> >> 09/06/09 09:20:54 WARN mapred.JobClient: Use
>> GenericOptionsParser
>> >> for
>> >> >> >> parsing the arguments. Applications should implement Tool for
>> the
>> >> >> same.
>> >> >> >> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths
>> to
>> >> >> >> process
>> >> >> >> : 1
>> >> >> >> 09/06/09 09:20:56 INFO mapred.JobClient: Running job:
>> >> >> >> job_200906090918_0001
>> >> >> >> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
>> >> >> >> java.io.IOException: Job failed!
>> >> >> >> at
>> >> >> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>> >> >> >> at
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
>> >> >> >> at
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
>> >> >> >> at
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
>> >> >> >> at
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
>> >> >> >> at
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
>> >> >> >>
>> >> >> >>
>> >> >> >> What am I doing wrong...?
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> View this message in context:
>> >> >> >>
>> >> >>
>> >>
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
>> >> >> >> Sent from the HBase User mailing list archive at Nabble.com.
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html
>> >> >> Sent from the HBase User mailing list archive at Nabble.com.
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947629.html
>> >> Sent from the HBase User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947980.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23948256.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by stack <st...@duboce.net>.
That looks fine but there is something in your setup that is off. For
example, HBASE_HOME is being expanded in the below properly on all hosts?
You are ssh forwarding this environment variable? (if you are using this
variable name).
St.Ack
On Tue, Jun 9, 2009 at 10:47 AM, llpind <so...@hotmail.com> wrote:
>
> Yeah that was my first thought. rowcounter and PerformanceEvaluation work.
> here is my HADOOP_CLASSPATH:
> ===================================
> export
>
> HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:/sharedfs/hbase-config/conf
> ===================================
> slave boxes reference a shared conf:
> export
>
> HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:$HBASE_HOME/conf
> ===================================
> where HBASE_HOME=/home/hadoop/hbase193. this is the same path on all
> boxes.
>
> I run my .class path from the default package like so:
>
> hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
>
>
>
>
> stack-3 wrote:
> >
> > Thats a CLASSPATH issue. For sure the hbase jars are out on all nodes of
> > your cluster? For sure the paths are correct?
> > St.Ack
> >
> > On Tue, Jun 9, 2009 at 10:28 AM, llpind <so...@hotmail.com> wrote:
> >
> >>
> >> Yep. Thanks, I didn't think to look there :). I do have another issue
> >> when
> >> running my own map/reduce. I've got the jar files in HADOOP_CLASSPATH
> as
> >> i
> >> mentioned above, but when i run the job below is what i get. The
> >> ClassDefNotFoundError happens on TableSplit multiple times.
> >> ================================================
> >>
> >> hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
> >> 09/06/09 10:26:08 WARN mapred.JobClient: No job jar file set. User
> >> classes
> >> may not be found. See JobConf(Class) or JobConf#setJar(String).
> >> 09/06/09 10:26:11 INFO mapred.JobClient: Running job:
> >> job_200906090918_0005
> >> 09/06/09 10:26:12 INFO mapred.JobClient: map 0% reduce 0%
> >> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
> >> attempt_200906090918_0005_m_000000_0, Status : FAILED
> >> java.io.IOException: Split class
> >> org.apache.hadoop.hbase.mapred.TableSplit
> >> not found
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.mapred.TableSplit
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:247)
> >> at
> >>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
> >> ... 1 more
> >>
> >> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
> >> attempt_200906090918_0005_m_000004_0, Status : FAILED
> >> java.io.IOException: Split class
> >> org.apache.hadoop.hbase.mapred.TableSplit
> >> not found
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.mapred.TableSplit
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:247)
> >> at
> >>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
> >> ... 1 more
> >>
> >>
> >> stack-3 wrote:
> >> >
> >> > It looks pretty obvious what your issue is.
> >> >
> >> > My guess is that your child process needs its heap size bumped up.
> >> >
> >> > Look for the child parameters configuration in your hadoop-default.xml
> >> (or
> >> > your mapred-default.xml if 0.20.x hadoop).
> >> >
> >> > St.Ack
> >> >
> >> > On Tue, Jun 9, 2009 at 10:05 AM, llpind <so...@hotmail.com>
> wrote:
> >> >
> >> >>
> >> >> Hmm...jobtracker UI has the following:
> >> >>
> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> >> >> contact
> >> >> region server 192.168.0.195:60020 for region
> TestTable,,1244499094604,
> >> >> row
> >> >> '0000010477', but failed after 10 attempts.
> >> >> Exceptions:
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >>
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
> >> >> at
> >> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
> >> >> at
> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
> >> >> at
> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
> >> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> >> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> >> >>
> >> >> =================================================
> >> >>
> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> >> >> contact
> >> >> region server 192.168.0.195:60020 for region
> TestTable,,1244499094604,
> >> >> row
> >> >> '0002107629', but failed after 10 attempts.
> >> >> Exceptions:
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >>
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
> >> >> at
> >> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
> >> >> at
> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
> >> >> at
> >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
> >> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> >> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> stack-3 wrote:
> >> >> >
> >> >> > Look at why the job failed in the jobtracker UI -- usually on port
> >> >> 50030.
> >> >> > Looks like your job launched fine. You have the conf dir on the
> >> >> > HADOOP_CLASSPATH so the MR job can find hbase?
> >> >> > St.Ack
> >> >> >
> >> >> > On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com>
> >> wrote:
> >> >> >
> >> >> >>
> >> >> >> Hey all,
> >> >> >>
> >> >> >> getting started with map/reduce jobs. Figured I'd try the
> >> >> >> PerformanceEvaluation test program first.
> >> >> >>
> >> >> >> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh,
> >> I
> >> >> >> issue
> >> >> >> the following command:
> >> >> >>
> >> >> >> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
> >> >> >> sequentialWrite 4
> >> >> >>
> >> >> >> and get the following output:
> >> >> >>
> >> >> >> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser
> >> for
> >> >> >> parsing the arguments. Applications should implement Tool for the
> >> >> same.
> >> >> >> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths
> to
> >> >> >> process
> >> >> >> : 1
> >> >> >> 09/06/09 09:20:56 INFO mapred.JobClient: Running job:
> >> >> >> job_200906090918_0001
> >> >> >> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
> >> >> >> java.io.IOException: Job failed!
> >> >> >> at
> >> >> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
> >> >> >>
> >> >> >>
> >> >> >> What am I doing wrong...?
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> View this message in context:
> >> >> >>
> >> >>
> >>
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
> >> >> >> Sent from the HBase User mailing list archive at Nabble.com.
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html
> >> >> Sent from the HBase User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947629.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947980.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
Yeah that was my first thought. rowcounter and PerformanceEvaluation work.
here is my HADOOP_CLASSPATH:
===================================
export
HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:/sharedfs/hbase-config/conf
===================================
slave boxes reference a shared conf:
export
HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:$HBASE_HOME/conf
===================================
where HBASE_HOME=/home/hadoop/hbase193. this is the same path on all boxes.
I run my .class path from the default package like so:
hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
stack-3 wrote:
>
> Thats a CLASSPATH issue. For sure the hbase jars are out on all nodes of
> your cluster? For sure the paths are correct?
> St.Ack
>
> On Tue, Jun 9, 2009 at 10:28 AM, llpind <so...@hotmail.com> wrote:
>
>>
>> Yep. Thanks, I didn't think to look there :). I do have another issue
>> when
>> running my own map/reduce. I've got the jar files in HADOOP_CLASSPATH as
>> i
>> mentioned above, but when i run the job below is what i get. The
>> ClassDefNotFoundError happens on TableSplit multiple times.
>> ================================================
>>
>> hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
>> 09/06/09 10:26:08 WARN mapred.JobClient: No job jar file set. User
>> classes
>> may not be found. See JobConf(Class) or JobConf#setJar(String).
>> 09/06/09 10:26:11 INFO mapred.JobClient: Running job:
>> job_200906090918_0005
>> 09/06/09 10:26:12 INFO mapred.JobClient: map 0% reduce 0%
>> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
>> attempt_200906090918_0005_m_000000_0, Status : FAILED
>> java.io.IOException: Split class
>> org.apache.hadoop.hbase.mapred.TableSplit
>> not found
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
>> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.mapred.TableSplit
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:247)
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
>> ... 1 more
>>
>> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
>> attempt_200906090918_0005_m_000004_0, Status : FAILED
>> java.io.IOException: Split class
>> org.apache.hadoop.hbase.mapred.TableSplit
>> not found
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
>> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.mapred.TableSplit
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:247)
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
>> ... 1 more
>>
>>
>> stack-3 wrote:
>> >
>> > It looks pretty obvious what your issue is.
>> >
>> > My guess is that your child process needs its heap size bumped up.
>> >
>> > Look for the child parameters configuration in your hadoop-default.xml
>> (or
>> > your mapred-default.xml if 0.20.x hadoop).
>> >
>> > St.Ack
>> >
>> > On Tue, Jun 9, 2009 at 10:05 AM, llpind <so...@hotmail.com> wrote:
>> >
>> >>
>> >> Hmm...jobtracker UI has the following:
>> >>
>> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> >> contact
>> >> region server 192.168.0.195:60020 for region TestTable,,1244499094604,
>> >> row
>> >> '0000010477', but failed after 10 attempts.
>> >> Exceptions:
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >>
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
>> >> at
>> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
>> >> at
>> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
>> >> at
>> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
>> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> >>
>> >> =================================================
>> >>
>> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> >> contact
>> >> region server 192.168.0.195:60020 for region TestTable,,1244499094604,
>> >> row
>> >> '0002107629', but failed after 10 attempts.
>> >> Exceptions:
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >>
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
>> >> at
>> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
>> >> at
>> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
>> >> at
>> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
>> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>> >>
>> >>
>> >>
>> >>
>> >> stack-3 wrote:
>> >> >
>> >> > Look at why the job failed in the jobtracker UI -- usually on port
>> >> 50030.
>> >> > Looks like your job launched fine. You have the conf dir on the
>> >> > HADOOP_CLASSPATH so the MR job can find hbase?
>> >> > St.Ack
>> >> >
>> >> > On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com>
>> wrote:
>> >> >
>> >> >>
>> >> >> Hey all,
>> >> >>
>> >> >> getting started with map/reduce jobs. Figured I'd try the
>> >> >> PerformanceEvaluation test program first.
>> >> >>
>> >> >> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh,
>> I
>> >> >> issue
>> >> >> the following command:
>> >> >>
>> >> >> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
>> >> >> sequentialWrite 4
>> >> >>
>> >> >> and get the following output:
>> >> >>
>> >> >> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser
>> for
>> >> >> parsing the arguments. Applications should implement Tool for the
>> >> same.
>> >> >> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths to
>> >> >> process
>> >> >> : 1
>> >> >> 09/06/09 09:20:56 INFO mapred.JobClient: Running job:
>> >> >> job_200906090918_0001
>> >> >> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
>> >> >> java.io.IOException: Job failed!
>> >> >> at
>> >> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
>> >> >>
>> >> >>
>> >> >> What am I doing wrong...?
>> >> >>
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
>> >> >> Sent from the HBase User mailing list archive at Nabble.com.
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html
>> >> Sent from the HBase User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947629.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947980.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by stack <st...@duboce.net>.
Thats a CLASSPATH issue. For sure the hbase jars are out on all nodes of
your cluster? For sure the paths are correct?
St.Ack
On Tue, Jun 9, 2009 at 10:28 AM, llpind <so...@hotmail.com> wrote:
>
> Yep. Thanks, I didn't think to look there :). I do have another issue
> when
> running my own map/reduce. I've got the jar files in HADOOP_CLASSPATH as i
> mentioned above, but when i run the job below is what i get. The
> ClassDefNotFoundError happens on TableSplit multiple times.
> ================================================
>
> hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
> 09/06/09 10:26:08 WARN mapred.JobClient: No job jar file set. User classes
> may not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/09 10:26:11 INFO mapred.JobClient: Running job: job_200906090918_0005
> 09/06/09 10:26:12 INFO mapred.JobClient: map 0% reduce 0%
> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
> attempt_200906090918_0005_m_000000_0, Status : FAILED
> java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
> not found
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapred.TableSplit
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
> ... 1 more
>
> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
> attempt_200906090918_0005_m_000004_0, Status : FAILED
> java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
> not found
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapred.TableSplit
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
> ... 1 more
>
>
> stack-3 wrote:
> >
> > It looks pretty obvious what your issue is.
> >
> > My guess is that your child process needs its heap size bumped up.
> >
> > Look for the child parameters configuration in your hadoop-default.xml
> (or
> > your mapred-default.xml if 0.20.x hadoop).
> >
> > St.Ack
> >
> > On Tue, Jun 9, 2009 at 10:05 AM, llpind <so...@hotmail.com> wrote:
> >
> >>
> >> Hmm...jobtracker UI has the following:
> >>
> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> >> contact
> >> region server 192.168.0.195:60020 for region TestTable,,1244499094604,
> >> row
> >> '0000010477', but failed after 10 attempts.
> >> Exceptions:
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >>
> >> at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
> >> at
> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
> >> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
> >> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> >>
> >> =================================================
> >>
> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> >> contact
> >> region server 192.168.0.195:60020 for region TestTable,,1244499094604,
> >> row
> >> '0002107629', but failed after 10 attempts.
> >> Exceptions:
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >> java.lang.OutOfMemoryError: Java heap space
> >>
> >> at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
> >> at
> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
> >> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
> >> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> >>
> >>
> >>
> >>
> >> stack-3 wrote:
> >> >
> >> > Look at why the job failed in the jobtracker UI -- usually on port
> >> 50030.
> >> > Looks like your job launched fine. You have the conf dir on the
> >> > HADOOP_CLASSPATH so the MR job can find hbase?
> >> > St.Ack
> >> >
> >> > On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com>
> wrote:
> >> >
> >> >>
> >> >> Hey all,
> >> >>
> >> >> getting started with map/reduce jobs. Figured I'd try the
> >> >> PerformanceEvaluation test program first.
> >> >>
> >> >> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh, I
> >> >> issue
> >> >> the following command:
> >> >>
> >> >> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
> >> >> sequentialWrite 4
> >> >>
> >> >> and get the following output:
> >> >>
> >> >> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser for
> >> >> parsing the arguments. Applications should implement Tool for the
> >> same.
> >> >> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths to
> >> >> process
> >> >> : 1
> >> >> 09/06/09 09:20:56 INFO mapred.JobClient: Running job:
> >> >> job_200906090918_0001
> >> >> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
> >> >> java.io.IOException: Job failed!
> >> >> at
> >> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
> >> >>
> >> >>
> >> >> What am I doing wrong...?
> >> >>
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
> >> >> Sent from the HBase User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947629.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
Yep. Thanks, I didn't think to look there :). I do have another issue when
running my own map/reduce. I've got the jar files in HADOOP_CLASSPATH as i
mentioned above, but when i run the job below is what i get. The
ClassDefNotFoundError happens on TableSplit multiple times.
================================================
hadoop19/bin/hadoop ColumnCountMapReduce inputTableName
09/06/09 10:26:08 WARN mapred.JobClient: No job jar file set. User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
09/06/09 10:26:11 INFO mapred.JobClient: Running job: job_200906090918_0005
09/06/09 10:26:12 INFO mapred.JobClient: map 0% reduce 0%
09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
attempt_200906090918_0005_m_000000_0, Status : FAILED
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
09/06/09 10:26:21 INFO mapred.JobClient: Task Id :
attempt_200906090918_0005_m_000004_0, Status : FAILED
java.io.IOException: Split class org.apache.hadoop.hbase.mapred.TableSplit
not found
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.mapred.TableSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
... 1 more
stack-3 wrote:
>
> It looks pretty obvious what your issue is.
>
> My guess is that your child process needs its heap size bumped up.
>
> Look for the child parameters configuration in your hadoop-default.xml (or
> your mapred-default.xml if 0.20.x hadoop).
>
> St.Ack
>
> On Tue, Jun 9, 2009 at 10:05 AM, llpind <so...@hotmail.com> wrote:
>
>>
>> Hmm...jobtracker UI has the following:
>>
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> contact
>> region server 192.168.0.195:60020 for region TestTable,,1244499094604,
>> row
>> '0000010477', but failed after 10 attempts.
>> Exceptions:
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>>
>> at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
>> at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
>> at
>> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
>> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
>> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>>
>> =================================================
>>
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> contact
>> region server 192.168.0.195:60020 for region TestTable,,1244499094604,
>> row
>> '0002107629', but failed after 10 attempts.
>> Exceptions:
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>>
>> at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
>> at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
>> at
>> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
>> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
>> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>>
>>
>>
>>
>> stack-3 wrote:
>> >
>> > Look at why the job failed in the jobtracker UI -- usually on port
>> 50030.
>> > Looks like your job launched fine. You have the conf dir on the
>> > HADOOP_CLASSPATH so the MR job can find hbase?
>> > St.Ack
>> >
>> > On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com> wrote:
>> >
>> >>
>> >> Hey all,
>> >>
>> >> getting started with map/reduce jobs. Figured I'd try the
>> >> PerformanceEvaluation test program first.
>> >>
>> >> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh, I
>> >> issue
>> >> the following command:
>> >>
>> >> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
>> >> sequentialWrite 4
>> >>
>> >> and get the following output:
>> >>
>> >> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser for
>> >> parsing the arguments. Applications should implement Tool for the
>> same.
>> >> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths to
>> >> process
>> >> : 1
>> >> 09/06/09 09:20:56 INFO mapred.JobClient: Running job:
>> >> job_200906090918_0001
>> >> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
>> >> java.io.IOException: Job failed!
>> >> at
>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
>> >>
>> >>
>> >> What am I doing wrong...?
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
>> >> Sent from the HBase User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947629.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by stack <st...@duboce.net>.
It looks pretty obvious what your issue is.
My guess is that your child process needs its heap size bumped up.
Look for the child parameters configuration in your hadoop-default.xml (or
your mapred-default.xml if 0.20.x hadoop).
St.Ack
On Tue, Jun 9, 2009 at 10:05 AM, llpind <so...@hotmail.com> wrote:
>
> Hmm...jobtracker UI has the following:
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
> region server 192.168.0.195:60020 for region TestTable,,1244499094604, row
> '0000010477', but failed after 10 attempts.
> Exceptions:
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
>
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>
> =================================================
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
> region server 192.168.0.195:60020 for region TestTable,,1244499094604, row
> '0002107629', but failed after 10 attempts.
> Exceptions:
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
>
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
>
>
>
>
> stack-3 wrote:
> >
> > Look at why the job failed in the jobtracker UI -- usually on port 50030.
> > Looks like your job launched fine. You have the conf dir on the
> > HADOOP_CLASSPATH so the MR job can find hbase?
> > St.Ack
> >
> > On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com> wrote:
> >
> >>
> >> Hey all,
> >>
> >> getting started with map/reduce jobs. Figured I'd try the
> >> PerformanceEvaluation test program first.
> >>
> >> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh, I
> >> issue
> >> the following command:
> >>
> >> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
> >> sequentialWrite 4
> >>
> >> and get the following output:
> >>
> >> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser for
> >> parsing the arguments. Applications should implement Tool for the same.
> >> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths to
> >> process
> >> : 1
> >> 09/06/09 09:20:56 INFO mapred.JobClient: Running job:
> >> job_200906090918_0001
> >> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
> >> java.io.IOException: Job failed!
> >> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
> >>
> >>
> >> What am I doing wrong...?
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
Hmm...jobtracker UI has the following:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 192.168.0.195:60020 for region TestTable,,1244499094604, row
'0000010477', but failed after 10 attempts.
Exceptions:
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
at
org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
at
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
=================================================
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 192.168.0.195:60020 for region TestTable,,1244499094604, row
'0002107629', but failed after 10 attempts.
Exceptions:
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321)
at
org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515)
at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583)
at
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
stack-3 wrote:
>
> Look at why the job failed in the jobtracker UI -- usually on port 50030.
> Looks like your job launched fine. You have the conf dir on the
> HADOOP_CLASSPATH so the MR job can find hbase?
> St.Ack
>
> On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com> wrote:
>
>>
>> Hey all,
>>
>> getting started with map/reduce jobs. Figured I'd try the
>> PerformanceEvaluation test program first.
>>
>> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh, I
>> issue
>> the following command:
>>
>> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
>> sequentialWrite 4
>>
>> and get the following output:
>>
>> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths to
>> process
>> : 1
>> 09/06/09 09:20:56 INFO mapred.JobClient: Running job:
>> job_200906090918_0001
>> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
>> java.io.IOException: Job failed!
>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
>> at
>>
>> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
>>
>>
>> What am I doing wrong...?
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by llpind <so...@hotmail.com>.
stack-3 wrote:
>
> You have the conf dir on the
> HADOOP_CLASSPATH so the MR job can find hbase?
> St.Ack
>
>
Yeah. I have conf in my HADOOP_CLASSPATH on all boxes in the cluster. I
have a shared directory for all Slave boxes which holds the hadoop config.
each box has hbase 0.19.3
--
View this message in context: http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947269.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: PerformanceEvaluation test
Posted by stack <st...@duboce.net>.
Look at why the job failed in the jobtracker UI -- usually on port 50030.
Looks like your job launched fine. You have the conf dir on the
HADOOP_CLASSPATH so the MR job can find hbase?
St.Ack
On Tue, Jun 9, 2009 at 9:25 AM, llpind <so...@hotmail.com> wrote:
>
> Hey all,
>
> getting started with map/reduce jobs. Figured I'd try the
> PerformanceEvaluation test program first.
>
> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh, I issue
> the following command:
>
> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation
> sequentialWrite 4
>
> and get the following output:
>
> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 09/06/09 09:20:56 INFO mapred.JobClient: Running job: job_200906090918_0001
> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0%
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748)
> at
>
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768)
>
>
> What am I doing wrong...?
>
>
> --
> View this message in context:
> http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>