You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Francis.Hu" <fr...@reachjunction.com> on 2013/07/10 10:33:46 UTC
cannot submit a job via java client in hadoop- 2.0.5-alpha
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Actually ,I have mapreduce.framework.name configured in mapred-site.xml, see
below:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework set to Hadoop YARN.</description>
</property>
发件人: hadoop hive [mailto:hadoophive@gmail.com]
发送时间: Wednesday, July 10, 2013 18:39
收件人: user@hadoop.apache.org
主题: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Actually ,I have mapreduce.framework.name configured in mapred-site.xml, see
below:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework set to Hadoop YARN.</description>
</property>
发件人: hadoop hive [mailto:hadoophive@gmail.com]
发送时间: Wednesday, July 10, 2013 18:39
收件人: user@hadoop.apache.org
主题: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Actually ,I have mapreduce.framework.name configured in mapred-site.xml, see
below:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework set to Hadoop YARN.</description>
</property>
发件人: hadoop hive [mailto:hadoophive@gmail.com]
发送时间: Wednesday, July 10, 2013 18:39
收件人: user@hadoop.apache.org
主题: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Actually ,I have mapreduce.framework.name configured in mapred-site.xml, see
below:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework set to Hadoop YARN.</description>
</property>
发件人: hadoop hive [mailto:hadoophive@gmail.com]
发送时间: Wednesday, July 10, 2013 18:39
收件人: user@hadoop.apache.org
主题: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by hadoop hive <ha...@gmail.com>.
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by hadoop hive <ha...@gmail.com>.
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by hadoop hive <ha...@gmail.com>.
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Hi, Devaraj k and Azuryy Yu
Thanks both of you.
I just get it resolved. The problem is that below highlighted jar is not
included in my java client side so that when the Job is initializing, it can
not find the class YarnClientProtocolProvider to do further initialization.
Then it causes the exception in Cluster.initialize() is thrown out.
hadoop-mapreduce-client-jobclient-2.0.5-alpha.jar
Thanks,
Francis.Hu
发件人: Devaraj k [mailto:devaraj.k@huawei.com]
发送时间: Wednesday, July 10, 2013 19:11
收件人: user@hadoop.apache.org
主题: RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Hi Francis,
Could you check whether those configuration files are
getting loaded or not, There could be a chance that these configuration
files are not getting loaded into configuration object due to some invalid
path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties
‘yarn.resourcemanager.address’ & ‘mapreduce.framework.name’ values and
check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Hi, Devaraj k and Azuryy Yu
Thanks both of you.
I just get it resolved. The problem is that below highlighted jar is not
included in my java client side so that when the Job is initializing, it can
not find the class YarnClientProtocolProvider to do further initialization.
Then it causes the exception in Cluster.initialize() is thrown out.
hadoop-mapreduce-client-jobclient-2.0.5-alpha.jar
Thanks,
Francis.Hu
发件人: Devaraj k [mailto:devaraj.k@huawei.com]
发送时间: Wednesday, July 10, 2013 19:11
收件人: user@hadoop.apache.org
主题: RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Hi Francis,
Could you check whether those configuration files are
getting loaded or not, There could be a chance that these configuration
files are not getting loaded into configuration object due to some invalid
path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties
‘yarn.resourcemanager.address’ & ‘mapreduce.framework.name’ values and
check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Hi, Devaraj k and Azuryy Yu
Thanks both of you.
I just get it resolved. The problem is that below highlighted jar is not
included in my java client side so that when the Job is initializing, it can
not find the class YarnClientProtocolProvider to do further initialization.
Then it causes the exception in Cluster.initialize() is thrown out.
hadoop-mapreduce-client-jobclient-2.0.5-alpha.jar
Thanks,
Francis.Hu
发件人: Devaraj k [mailto:devaraj.k@huawei.com]
发送时间: Wednesday, July 10, 2013 19:11
收件人: user@hadoop.apache.org
主题: RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Hi Francis,
Could you check whether those configuration files are
getting loaded or not, There could be a chance that these configuration
files are not getting loaded into configuration object due to some invalid
path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties
‘yarn.resourcemanager.address’ & ‘mapreduce.framework.name’ values and
check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Hi, Devaraj k and Azuryy Yu
Thanks both of you.
I just get it resolved. The problem is that below highlighted jar is not
included in my java client side so that when the Job is initializing, it can
not find the class YarnClientProtocolProvider to do further initialization.
Then it causes the exception in Cluster.initialize() is thrown out.
hadoop-mapreduce-client-jobclient-2.0.5-alpha.jar
Thanks,
Francis.Hu
发件人: Devaraj k [mailto:devaraj.k@huawei.com]
发送时间: Wednesday, July 10, 2013 19:11
收件人: user@hadoop.apache.org
主题: RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Hi Francis,
Could you check whether those configuration files are
getting loaded or not, There could be a chance that these configuration
files are not getting loaded into configuration object due to some invalid
path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties
‘yarn.resourcemanager.address’ & ‘mapreduce.framework.name’ values and
check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>
wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
Hi Francis,
Could you check whether those configuration files are getting loaded or not, There could be a chance that these configuration files are not getting loaded into configuration object due to some invalid path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties 'yarn.resourcemanager.address' & 'mapreduce.framework.name' values and check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
Hi Francis,
Could you check whether those configuration files are getting loaded or not, There could be a chance that these configuration files are not getting loaded into configuration object due to some invalid path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties 'yarn.resourcemanager.address' & 'mapreduce.framework.name' values and check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
Hi Francis,
Could you check whether those configuration files are getting loaded or not, There could be a chance that these configuration files are not getting loaded into configuration object due to some invalid path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties 'yarn.resourcemanager.address' & 'mapreduce.framework.name' values and check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
'yarn.nodemanager.address' is not required to submit the Job, it will be required only in NM side.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
'yarn.nodemanager.address' is not required to submit the Job, it will be required only in NM side.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
'yarn.nodemanager.address' is not required to submit the Job, it will be required only in NM side.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
'yarn.nodemanager.address' is not required to submit the Job, it will be required only in NM side.
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Devaraj k <de...@huawei.com>.
Hi Francis,
Could you check whether those configuration files are getting loaded or not, There could be a chance that these configuration files are not getting loaded into configuration object due to some invalid path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties 'yarn.resourcemanager.address' & 'mapreduce.framework.name' values and check they are coming as per the config files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 10 July 2013 16:22
To: user@hadoop.apache.org
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Azuryy Yu <az...@gmail.com>.
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Azuryy Yu <az...@gmail.com>.
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Azuryy Yu <az...@gmail.com>.
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by hadoop hive <ha...@gmail.com>.
Here its showing like you are not using mapreduce.framework.name as yarn,
please resend it we are unable to see the configuration
On Wed, Jul 10, 2013 at 1:33 AM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
Posted by Azuryy Yu <az...@gmail.com>.
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Hi,All****
>
> ** **
>
> I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
> Manager and all data nodes started and can access web ui of Resource
> Manager. ****
>
> I wrote a java client to submit a job as TestJob class below. But the job
> is never submitted successfully. It throws out exception all the time. ***
> *
>
> My configurations are attached. Can anyone help me? Thanks.****
>
> ** **
>
> ---------my-java client****
>
> *public* *class* TestJob {****
>
> ****
>
> *public* *void* execute() {****
>
> ** **
>
> Configuration conf1 = *new* Configuration();****
>
> conf1.addResource("resources/core-site.xml");****
>
> conf1.addResource("resources/hdfs-site.xml");****
>
> conf1.addResource("resources/yarn-site.xml");****
>
> conf1.addResource("resources/mapred-site.xml");****
>
> JobConf conf = *new* JobConf(conf1);****
>
> ****
>
> conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");****
>
> conf.setJobName("Test");****
>
> ** **
>
> conf.setInputFormat(TextInputFormat.*class*);****
>
> conf.setOutputFormat(TextOutputFormat.*class*);****
>
> ** **
>
> conf.setOutputKeyClass(Text.*class*);****
>
> conf.setOutputValueClass(IntWritable.*class*);****
>
> ** **
>
> conf.setMapperClass(DisplayRequestMapper.*class*);****
>
> conf.setReducerClass(DisplayRequestReducer.*class*);****
>
> ** **
>
> FileInputFormat.*setInputPaths*(conf,*new* Path(
> "/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));****
>
> FileOutputFormat.*setOutputPath*(conf, *new* Path(
> "/home/francis/hadoop-jobs/result/"));****
>
> ** **
>
> *try* {****
>
> JobClient client = *new* JobClient(conf);****
>
> RunningJob job = client.submitJob(conf);****
>
> job.waitForCompletion();****
>
> } *catch* (IOException e) {****
>
> e.printStackTrace();****
>
> }****
>
> }****
>
> }****
>
> ** **
>
> ----------Exception ****
>
> ** **
>
> jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check
> your configuration for mapreduce.framework.name and the correspond server
> addresses.****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)****
>
> jvm 1 | at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)****
>
> jvm 1 | at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)****
>
> jvm 1 | at
> com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)****
>
> ** **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>