You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Saptarshi Guha <sa...@gmail.com> on 2008/12/19 00:38:04 UTC

Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)

Hello,
I intend to start a mapreduce job from another java app, using
ToolRunner.run method. This works fine on a local job. However when
distributed i get java.lang.NoClassDefFoundError:
org/rosuda/REngine/Rserve/RserveException
	at java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
	at java.lang.Class.getConstructor0(Class.java:2699)
	at java.lang.Class.getDeclaredConstructor(Class.java:1985)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74)
	at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:402)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)

Now, i have a CLASSPATH entry in my .bashrc which contains the
location of Rserve.jar (which contains the above classes), yet I still
get the above error.
Do I have to somehow emulate RunJar.main or is there a simpler way out?
Thank you
Saptarshi

-- 
Saptarshi Guha - saptarshi.guha@gmail.com

hadoop 1.18 version install on single node with two disks producing createBlockOutputStream java.io.IOException - help needed

Posted by "Akturan, Cagdas" <ca...@intel.com>.
Hi all,

I am new to this list so please forgive me if this is not the right way or format to ask for help.
I installed hadoop version hadoop-0.18.1 with single node and single disk and everything was working fine.
When I added one more disk I started getting the errors copied below.
I tried balancerRM_br18.patch and also tried the same on version hadoop-0.18.2 with no luck.
Does anyone had similar issue and can help me?

        Job started: Tue Dec 23 13:20:43 PST 2008
08/12/23 13:20:43 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream
08/12/23 13:20:43 INFO dfs.DFSClient: Abandoning block blk_3043170186642040141_1002
08/12/23 13:20:49 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream
08/12/23 13:20:49 INFO dfs.DFSClient: Abandoning block blk_-2352529149307511754_1003

Best regards,
Cagdas Akturan

Re: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)

Posted by Jason Venner <ja...@attributor.com>.
Yes this will work. You will need to configure the class path to include 
that directory.
The Tasktracker's really only have the classpath as setup by 
conf/hadoop-env.sh, and the Tasktracker$Child's have that classpath + 
the unpacked distributed cache directory.

Saptarshi Guha wrote:
> Hello,
> What if each machine has the identical directory structure and
> Rserve.jar is present at the same place(i.e the entry in the classpath
> is identical across machines). The will the latter suggestion work?
>
> On Thu, Dec 18, 2008 at 9:02 PM, Zheng Shao <zs...@facebook.com> wrote:
>   
>> You either need to add the your Rserve.jar to -libjars, or put Rserve.jar in a cluster-accessible NFS mount (and add it to CLASSPATH/HADOOP_CLASSPATH).
>>
>> Zheng
>> -----Original Message-----
>> From: Saptarshi Guha [mailto:saptarshi.guha@gmail.com]
>> Sent: Thursday, December 18, 2008 3:38 PM
>> To: core-user@hadoop.apache.org
>> Subject: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)
>>
>> Hello,
>> I intend to start a mapreduce job from another java app, using
>> ToolRunner.run method. This works fine on a local job. However when
>> distributed i get java.lang.NoClassDefFoundError:
>> org/rosuda/REngine/Rserve/RserveException
>>        at java.lang.Class.getDeclaredConstructors0(Native Method)
>>        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
>>        at java.lang.Class.getConstructor0(Class.java:2699)
>>        at java.lang.Class.getDeclaredConstructor(Class.java:1985)
>>        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74)
>>        at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:402)
>>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
>>        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
>>
>> Now, i have a CLASSPATH entry in my .bashrc which contains the
>> location of Rserve.jar (which contains the above classes), yet I still
>> get the above error.
>> Do I have to somehow emulate RunJar.main or is there a simpler way out?
>> Thank you
>> Saptarshi
>>
>> --
>> Saptarshi Guha - saptarshi.guha@gmail.com
>>
>>     
>
>
>
>

Re: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)

Posted by Saptarshi Guha <sa...@gmail.com>.
Hello,
What if each machine has the identical directory structure and
Rserve.jar is present at the same place(i.e the entry in the classpath
is identical across machines). The will the latter suggestion work?

On Thu, Dec 18, 2008 at 9:02 PM, Zheng Shao <zs...@facebook.com> wrote:
> You either need to add the your Rserve.jar to -libjars, or put Rserve.jar in a cluster-accessible NFS mount (and add it to CLASSPATH/HADOOP_CLASSPATH).
>
> Zheng
> -----Original Message-----
> From: Saptarshi Guha [mailto:saptarshi.guha@gmail.com]
> Sent: Thursday, December 18, 2008 3:38 PM
> To: core-user@hadoop.apache.org
> Subject: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)
>
> Hello,
> I intend to start a mapreduce job from another java app, using
> ToolRunner.run method. This works fine on a local job. However when
> distributed i get java.lang.NoClassDefFoundError:
> org/rosuda/REngine/Rserve/RserveException
>        at java.lang.Class.getDeclaredConstructors0(Native Method)
>        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
>        at java.lang.Class.getConstructor0(Class.java:2699)
>        at java.lang.Class.getDeclaredConstructor(Class.java:1985)
>        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74)
>        at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:402)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
>        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
>
> Now, i have a CLASSPATH entry in my .bashrc which contains the
> location of Rserve.jar (which contains the above classes), yet I still
> get the above error.
> Do I have to somehow emulate RunJar.main or is there a simpler way out?
> Thank you
> Saptarshi
>
> --
> Saptarshi Guha - saptarshi.guha@gmail.com
>



-- 
Saptarshi Guha - saptarshi.guha@gmail.com

RE: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)

Posted by Zheng Shao <zs...@facebook.com>.
You either need to add the your Rserve.jar to -libjars, or put Rserve.jar in a cluster-accessible NFS mount (and add it to CLASSPATH/HADOOP_CLASSPATH).  

Zheng
-----Original Message-----
From: Saptarshi Guha [mailto:saptarshi.guha@gmail.com] 
Sent: Thursday, December 18, 2008 3:38 PM
To: core-user@hadoop.apache.org
Subject: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)

Hello,
I intend to start a mapreduce job from another java app, using
ToolRunner.run method. This works fine on a local job. However when
distributed i get java.lang.NoClassDefFoundError:
org/rosuda/REngine/Rserve/RserveException
	at java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
	at java.lang.Class.getConstructor0(Class.java:2699)
	at java.lang.Class.getDeclaredConstructor(Class.java:1985)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74)
	at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:402)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)

Now, i have a CLASSPATH entry in my .bashrc which contains the
location of Rserve.jar (which contains the above classes), yet I still
get the above error.
Do I have to somehow emulate RunJar.main or is there a simpler way out?
Thank you
Saptarshi

-- 
Saptarshi Guha - saptarshi.guha@gmail.com