You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Matthew John <tm...@gmail.com> on 2010/12/22 17:19:32 UTC

hdfs with raid

Hi all,

Got to know about a hdfs with raid implementation from the following
documentation :
http://wiki.apache.org/hadoop/HDFS-RAID

In the documentation, it says u can find the hadoop-*-raid.jar file which
has got the libraries to run the raid-hdfs.
Where to get this file ? Searched a lot , but could not get my hands on it
..

Thanks,
Matthew

Re: -libjars option

Posted by Milind Bhandarkar <mb...@linkedin.com>.
Your main class needs to call GenericOptionsParser.

- milind

On Jan 10, 2011, at 7:32 PM, C.V.Krishnakumar Iyer wrote:

> Hi,
> 
> Could anyone please guide me as to how to use the -libjars option in HDFS? 
> 
> I have added the necessary jar file (the hbase jar - to be precise)  to the classpath of the node where I am starting the job. 
> 
> The following is the format that i am invoking: 
> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by Commas)> <Arguments to our main class>
> 
> bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a
> 
> Despite this,  I find that I get the java.lang.ClassNotFoundException error! :(
> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
> 	at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
> 	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
> 
> The strange thing is that there is another MR job I have  that runs perfectly with the libjars option! Could anybody tell me what I am doing wrong? One more thing - not sure if it is relevant : I am using the new Hadoop MapReduce API.
> 
> Thanks in advance!
> 
> Regards,
> Krishnakumar.
> 
> 
> 
> 
> 

---
Milind Bhandarkar
mbhandarkar@linkedin.com

-libjars option

Posted by "C.V.Krishnakumar Iyer" <f2...@gmail.com>.
Hi,

Could anyone please guide me as to how to use the -libjars option in HDFS? 

I have added the necessary jar file (the hbase jar - to be precise)  to the classpath of the node where I am starting the job. 

The following is the format that i am invoking: 
bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by Commas)> <Arguments to our main class>

bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a

Despite this,  I find that I get the java.lang.ClassNotFoundException error! :(
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
	at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

 The strange thing is that there is another MR job I have  that runs perfectly with the libjars option! Could anybody tell me what I am doing wrong? One more thing - not sure if it is relevant : I am using the new Hadoop MapReduce API.

Thanks in advance!

Regards,
Krishnakumar.

Re: hdfs with raid

Posted by Harsh J <qw...@gmail.com>.
As of 0.21.0, the RAID-HDFS feature resides inside mapreduce's
contrib. You may find the jar inside a 0.21.0 tar's
mapred/contrib/raid folder.

-- 
Harsh J
www.harshj.com