You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Carmen Manzulli <ca...@gmail.com> on 2015/05/25 12:36:06 UTC
Fwd: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Hi,
I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which submits
a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is configured
using single node setting up in /usr/local folder. This job works if i
start it from command line but, when i try to execute the follow code from
netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster. Please
check your configuration for mapreduce.framework.name and the correspond
server addresses."
Configuration config = new Configuration();
config.addResource(new
Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
config.addResource(new
Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
config.addResource(new
Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
config.addResource(new
Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
config.set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
config.set("yarn.resourcemanager.address","master:8032");
config.set("mapreduce.framework.name","yarn");
config.set("fs.defaultFS","hdfs://master:9000");
//input query parameter as string
config.set(ArqReducerConstants.MY_QUERY,args[0]);
Job job = Job.getInstance(config);
job.setJarByClass(QueryServlet.class);//input number of lines for
mapper parameter as int, used to reduce number of splits and run map()
method quicklyString N=args[4];int n=Integer.parseInt(N);
job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
job.setMapperClass(MapperDoesNothing.class);
job.setMapOutputKeyClass(NullWritable.class);
job.setMapOutputValueClass(TripleWritable.class);
job.setReducerClass(QueryReducer.class);
job.setInputFormatClass( BlockedNTriplesInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);String
in="hdfs://master:9000"+args[1];String
out="hdfs://master:9000"+args[2];//input and output paths
parametersFileInputFormat.setInputPaths(job, new
Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
job.waitForCompletion(true);
//where args is just a name of a String array with some input parameters..
There are no problems with hadoop or in the .xml files; there are no
problems with permissions, because i've disable its with:
dfs.permission.enables=false in hdfs-site.xml; there are no problem with
permissions also for my hadoop folder because i've used chmod -R 777;
So...what does my project miss to obtain the goal? I need help...
i think there is something missing when i set the Configuration Object, or
problem could be due to something about jar in the classpath but in this
case i don't know how to insert all hadoop jar using maven...
Thanks in advance, Carmen.
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by Carmen Manzulli <ca...@gmail.com>.
Thanks you were right!!!! but loading *-site.xml files in hdfs for example
in a folder /configurationFiles/ created by: hdfs dfs -mkdir
/configurationFiles and writing
config.addResource(new
Path("hdfs://master:9000/configurationFiles/core-site.xml"));
i get the same error... this is the complete error:
java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:415)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
serv.QueryServlet.processRequest(QueryServlet.java:97)
<---In line 97 I've job.waitForCompletion(true);
serv.QueryServlet.doPost(QueryServlet.java:133)
javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:393)
Any Idea?
2015-05-28 3:10 GMT+02:00 杨浩 <ya...@gmail.com>:
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>>
>> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
> do you have the resource in hdfs?
>
> can you try the API
>
>> config.addResource(InputStream in)
>
>
> 2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
>>
>> Hi,
>>
>> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which
>> submits a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is
>> configured using single node setting up in /usr/local folder. This job
>> works if i start it from command line but, when i try to execute the follow
>> code from netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster.
>> Please check your configuration for mapreduce.framework.name and the
>> correspond server addresses."
>>
>> Configuration config = new Configuration();
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>>
>> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
>> config.set("yarn.resourcemanager.address","master:8032");
>> config.set("mapreduce.framework.name","yarn");
>> config.set("fs.defaultFS","hdfs://master:9000");
>> //input query parameter as string
>> config.set(ArqReducerConstants.MY_QUERY,args[0]);
>> Job job = Job.getInstance(config);
>> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
>> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>>
>> job.setMapperClass(MapperDoesNothing.class);
>> job.setMapOutputKeyClass(NullWritable.class);
>> job.setMapOutputValueClass(TripleWritable.class);
>>
>> job.setReducerClass(QueryReducer.class);
>>
>> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>>
>> job.waitForCompletion(true);
>>
>> //where args is just a name of a String array with some input
>> parameters..
>>
>> There are no problems with hadoop or in the .xml files; there are no
>> problems with permissions, because i've disable its with:
>> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
>> permissions also for my hadoop folder because i've used chmod -R 777;
>>
>> So...what does my project miss to obtain the goal? I need help...
>>
>> i think there is something missing when i set the Configuration Object,
>> or problem could be due to something about jar in the classpath but in this
>> case i don't know how to insert all hadoop jar using maven...
>>
>> Thanks in advance, Carmen.
>>
>>
>
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by Carmen Manzulli <ca...@gmail.com>.
Thanks you were right!!!! but loading *-site.xml files in hdfs for example
in a folder /configurationFiles/ created by: hdfs dfs -mkdir
/configurationFiles and writing
config.addResource(new
Path("hdfs://master:9000/configurationFiles/core-site.xml"));
i get the same error... this is the complete error:
java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:415)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
serv.QueryServlet.processRequest(QueryServlet.java:97)
<---In line 97 I've job.waitForCompletion(true);
serv.QueryServlet.doPost(QueryServlet.java:133)
javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:393)
Any Idea?
2015-05-28 3:10 GMT+02:00 杨浩 <ya...@gmail.com>:
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>>
>> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
> do you have the resource in hdfs?
>
> can you try the API
>
>> config.addResource(InputStream in)
>
>
> 2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
>>
>> Hi,
>>
>> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which
>> submits a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is
>> configured using single node setting up in /usr/local folder. This job
>> works if i start it from command line but, when i try to execute the follow
>> code from netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster.
>> Please check your configuration for mapreduce.framework.name and the
>> correspond server addresses."
>>
>> Configuration config = new Configuration();
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>>
>> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
>> config.set("yarn.resourcemanager.address","master:8032");
>> config.set("mapreduce.framework.name","yarn");
>> config.set("fs.defaultFS","hdfs://master:9000");
>> //input query parameter as string
>> config.set(ArqReducerConstants.MY_QUERY,args[0]);
>> Job job = Job.getInstance(config);
>> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
>> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>>
>> job.setMapperClass(MapperDoesNothing.class);
>> job.setMapOutputKeyClass(NullWritable.class);
>> job.setMapOutputValueClass(TripleWritable.class);
>>
>> job.setReducerClass(QueryReducer.class);
>>
>> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>>
>> job.waitForCompletion(true);
>>
>> //where args is just a name of a String array with some input
>> parameters..
>>
>> There are no problems with hadoop or in the .xml files; there are no
>> problems with permissions, because i've disable its with:
>> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
>> permissions also for my hadoop folder because i've used chmod -R 777;
>>
>> So...what does my project miss to obtain the goal? I need help...
>>
>> i think there is something missing when i set the Configuration Object,
>> or problem could be due to something about jar in the classpath but in this
>> case i don't know how to insert all hadoop jar using maven...
>>
>> Thanks in advance, Carmen.
>>
>>
>
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by Carmen Manzulli <ca...@gmail.com>.
Thanks you were right!!!! but loading *-site.xml files in hdfs for example
in a folder /configurationFiles/ created by: hdfs dfs -mkdir
/configurationFiles and writing
config.addResource(new
Path("hdfs://master:9000/configurationFiles/core-site.xml"));
i get the same error... this is the complete error:
java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:415)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
serv.QueryServlet.processRequest(QueryServlet.java:97)
<---In line 97 I've job.waitForCompletion(true);
serv.QueryServlet.doPost(QueryServlet.java:133)
javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:393)
Any Idea?
2015-05-28 3:10 GMT+02:00 杨浩 <ya...@gmail.com>:
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>>
>> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
> do you have the resource in hdfs?
>
> can you try the API
>
>> config.addResource(InputStream in)
>
>
> 2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
>>
>> Hi,
>>
>> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which
>> submits a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is
>> configured using single node setting up in /usr/local folder. This job
>> works if i start it from command line but, when i try to execute the follow
>> code from netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster.
>> Please check your configuration for mapreduce.framework.name and the
>> correspond server addresses."
>>
>> Configuration config = new Configuration();
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>>
>> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
>> config.set("yarn.resourcemanager.address","master:8032");
>> config.set("mapreduce.framework.name","yarn");
>> config.set("fs.defaultFS","hdfs://master:9000");
>> //input query parameter as string
>> config.set(ArqReducerConstants.MY_QUERY,args[0]);
>> Job job = Job.getInstance(config);
>> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
>> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>>
>> job.setMapperClass(MapperDoesNothing.class);
>> job.setMapOutputKeyClass(NullWritable.class);
>> job.setMapOutputValueClass(TripleWritable.class);
>>
>> job.setReducerClass(QueryReducer.class);
>>
>> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>>
>> job.waitForCompletion(true);
>>
>> //where args is just a name of a String array with some input
>> parameters..
>>
>> There are no problems with hadoop or in the .xml files; there are no
>> problems with permissions, because i've disable its with:
>> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
>> permissions also for my hadoop folder because i've used chmod -R 777;
>>
>> So...what does my project miss to obtain the goal? I need help...
>>
>> i think there is something missing when i set the Configuration Object,
>> or problem could be due to something about jar in the classpath but in this
>> case i don't know how to insert all hadoop jar using maven...
>>
>> Thanks in advance, Carmen.
>>
>>
>
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by Carmen Manzulli <ca...@gmail.com>.
Thanks you were right!!!! but loading *-site.xml files in hdfs for example
in a folder /configurationFiles/ created by: hdfs dfs -mkdir
/configurationFiles and writing
config.addResource(new
Path("hdfs://master:9000/configurationFiles/core-site.xml"));
i get the same error... this is the complete error:
java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:415)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
serv.QueryServlet.processRequest(QueryServlet.java:97)
<---In line 97 I've job.waitForCompletion(true);
serv.QueryServlet.doPost(QueryServlet.java:133)
javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:393)
Any Idea?
2015-05-28 3:10 GMT+02:00 杨浩 <ya...@gmail.com>:
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>>
>> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
> do you have the resource in hdfs?
>
> can you try the API
>
>> config.addResource(InputStream in)
>
>
> 2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
>>
>> Hi,
>>
>> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which
>> submits a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is
>> configured using single node setting up in /usr/local folder. This job
>> works if i start it from command line but, when i try to execute the follow
>> code from netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster.
>> Please check your configuration for mapreduce.framework.name and the
>> correspond server addresses."
>>
>> Configuration config = new Configuration();
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>>
>> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
>> config.set("yarn.resourcemanager.address","master:8032");
>> config.set("mapreduce.framework.name","yarn");
>> config.set("fs.defaultFS","hdfs://master:9000");
>> //input query parameter as string
>> config.set(ArqReducerConstants.MY_QUERY,args[0]);
>> Job job = Job.getInstance(config);
>> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
>> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>>
>> job.setMapperClass(MapperDoesNothing.class);
>> job.setMapOutputKeyClass(NullWritable.class);
>> job.setMapOutputValueClass(TripleWritable.class);
>>
>> job.setReducerClass(QueryReducer.class);
>>
>> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>>
>> job.waitForCompletion(true);
>>
>> //where args is just a name of a String array with some input
>> parameters..
>>
>> There are no problems with hadoop or in the .xml files; there are no
>> problems with permissions, because i've disable its with:
>> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
>> permissions also for my hadoop folder because i've used chmod -R 777;
>>
>> So...what does my project miss to obtain the goal? I need help...
>>
>> i think there is something missing when i set the Configuration Object,
>> or problem could be due to something about jar in the classpath but in this
>> case i don't know how to insert all hadoop jar using maven...
>>
>> Thanks in advance, Carmen.
>>
>>
>
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by 杨浩 <ya...@gmail.com>.
>
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>
> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
do you have the resource in hdfs?
can you try the API
> config.addResource(InputStream in)
2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
> Hi,
>
> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which submits
> a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is configured
> using single node setting up in /usr/local folder. This job works if i
> start it from command line but, when i try to execute the follow code from
> netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster. Please
> check your configuration for mapreduce.framework.name and the correspond
> server addresses."
>
> Configuration config = new Configuration();
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>
> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
> config.set("yarn.resourcemanager.address","master:8032");
> config.set("mapreduce.framework.name","yarn");
> config.set("fs.defaultFS","hdfs://master:9000");
> //input query parameter as string
> config.set(ArqReducerConstants.MY_QUERY,args[0]);
> Job job = Job.getInstance(config);
> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>
> job.setMapperClass(MapperDoesNothing.class);
> job.setMapOutputKeyClass(NullWritable.class);
> job.setMapOutputValueClass(TripleWritable.class);
>
> job.setReducerClass(QueryReducer.class);
>
> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>
> job.waitForCompletion(true);
>
> //where args is just a name of a String array with some input parameters..
>
> There are no problems with hadoop or in the .xml files; there are no
> problems with permissions, because i've disable its with:
> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
> permissions also for my hadoop folder because i've used chmod -R 777;
>
> So...what does my project miss to obtain the goal? I need help...
>
> i think there is something missing when i set the Configuration Object, or
> problem could be due to something about jar in the classpath but in this
> case i don't know how to insert all hadoop jar using maven...
>
> Thanks in advance, Carmen.
>
>
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by 杨浩 <ya...@gmail.com>.
>
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>
> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
do you have the resource in hdfs?
can you try the API
> config.addResource(InputStream in)
2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
> Hi,
>
> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which submits
> a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is configured
> using single node setting up in /usr/local folder. This job works if i
> start it from command line but, when i try to execute the follow code from
> netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster. Please
> check your configuration for mapreduce.framework.name and the correspond
> server addresses."
>
> Configuration config = new Configuration();
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>
> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
> config.set("yarn.resourcemanager.address","master:8032");
> config.set("mapreduce.framework.name","yarn");
> config.set("fs.defaultFS","hdfs://master:9000");
> //input query parameter as string
> config.set(ArqReducerConstants.MY_QUERY,args[0]);
> Job job = Job.getInstance(config);
> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>
> job.setMapperClass(MapperDoesNothing.class);
> job.setMapOutputKeyClass(NullWritable.class);
> job.setMapOutputValueClass(TripleWritable.class);
>
> job.setReducerClass(QueryReducer.class);
>
> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>
> job.waitForCompletion(true);
>
> //where args is just a name of a String array with some input parameters..
>
> There are no problems with hadoop or in the .xml files; there are no
> problems with permissions, because i've disable its with:
> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
> permissions also for my hadoop folder because i've used chmod -R 777;
>
> So...what does my project miss to obtain the goal? I need help...
>
> i think there is something missing when i set the Configuration Object, or
> problem could be due to something about jar in the classpath but in this
> case i don't know how to insert all hadoop jar using maven...
>
> Thanks in advance, Carmen.
>
>
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by 杨浩 <ya...@gmail.com>.
>
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>
> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
do you have the resource in hdfs?
can you try the API
> config.addResource(InputStream in)
2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
> Hi,
>
> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which submits
> a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is configured
> using single node setting up in /usr/local folder. This job works if i
> start it from command line but, when i try to execute the follow code from
> netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster. Please
> check your configuration for mapreduce.framework.name and the correspond
> server addresses."
>
> Configuration config = new Configuration();
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>
> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
> config.set("yarn.resourcemanager.address","master:8032");
> config.set("mapreduce.framework.name","yarn");
> config.set("fs.defaultFS","hdfs://master:9000");
> //input query parameter as string
> config.set(ArqReducerConstants.MY_QUERY,args[0]);
> Job job = Job.getInstance(config);
> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>
> job.setMapperClass(MapperDoesNothing.class);
> job.setMapOutputKeyClass(NullWritable.class);
> job.setMapOutputValueClass(TripleWritable.class);
>
> job.setReducerClass(QueryReducer.class);
>
> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>
> job.waitForCompletion(true);
>
> //where args is just a name of a String array with some input parameters..
>
> There are no problems with hadoop or in the .xml files; there are no
> problems with permissions, because i've disable its with:
> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
> permissions also for my hadoop folder because i've used chmod -R 777;
>
> So...what does my project miss to obtain the goal? I need help...
>
> i think there is something missing when i set the Configuration Object, or
> problem could be due to something about jar in the classpath but in this
> case i don't know how to insert all hadoop jar using maven...
>
> Thanks in advance, Carmen.
>
>
Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Posted by 杨浩 <ya...@gmail.com>.
>
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>
> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
do you have the resource in hdfs?
can you try the API
> config.addResource(InputStream in)
2015-05-25 18:36 GMT+08:00 Carmen Manzulli <ca...@gmail.com>:
>
> Hi,
>
> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which submits
> a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is configured
> using single node setting up in /usr/local folder. This job works if i
> start it from command line but, when i try to execute the follow code from
> netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster. Please
> check your configuration for mapreduce.framework.name and the correspond
> server addresses."
>
> Configuration config = new Configuration();
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>
> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
> config.set("yarn.resourcemanager.address","master:8032");
> config.set("mapreduce.framework.name","yarn");
> config.set("fs.defaultFS","hdfs://master:9000");
> //input query parameter as string
> config.set(ArqReducerConstants.MY_QUERY,args[0]);
> Job job = Job.getInstance(config);
> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>
> job.setMapperClass(MapperDoesNothing.class);
> job.setMapOutputKeyClass(NullWritable.class);
> job.setMapOutputValueClass(TripleWritable.class);
>
> job.setReducerClass(QueryReducer.class);
>
> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job, new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>
> job.waitForCompletion(true);
>
> //where args is just a name of a String array with some input parameters..
>
> There are no problems with hadoop or in the .xml files; there are no
> problems with permissions, because i've disable its with:
> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
> permissions also for my hadoop folder because i've used chmod -R 777;
>
> So...what does my project miss to obtain the goal? I need help...
>
> i think there is something missing when i set the Configuration Object, or
> problem could be due to something about jar in the classpath but in this
> case i don't know how to insert all hadoop jar using maven...
>
> Thanks in advance, Carmen.
>
>