You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Cornish, Duane C." <Du...@jhuapl.edu> on 2013/02/08 18:25:04 UTC
pragmattically invoking map reduce job from a .war deployed in
tomcat
I have an accumulo instance running on a small 4 node cluster. The namenode is running tomcat. I am using a restful service to allow clients to access accumulo and kick off map reduce jobs.
Previously, I had been creating runnable jars of my map reduce jobs. The restful service would make a command line call to run the jar. i.e. "java -jar mapreducejob.java -arg1 -arg2 etc."
I am now trying to switch these jobs to be called programmatically. I am aware that I need to pass a comma separated list of jars into the ToolRunner.run call. i.e.
arg[0] = "-libjar";
arg[1] = "accumulo-core-1.4.1,jar2" etc.
arg[2] = "myArg1";
arg[3] = "myArg2";
etc.
When I run my job, I get an error in in my map tasks in the jobtracker Admin webpage:
java.lang.RuntimeException: java.lang.ClassNotFoundException: <package.Name>.<functionNameOfMyMapper>
I don't know why the system can't find the mapper function. It is in the same package as my Job file and they should all be contained in the .war file that is deployed in tomcat.
Has anyone had success calling a map reduce job programmatically from within tomcat?
Thanks,
Duane
RE: pragmattically invoking map reduce job from a .war deployed in tomcat
Posted by Bo...@l-3com.com.
I jar up my map/reduce classes and all their dependencies to deploy to
the lib/ext directory. Those classes are not runnable; dependent on
ToolRunner.
From: Cornish, Duane C. [mailto:Duane.Cornish@jhuapl.edu]
Sent: Monday, February 11, 2013 09:14
To: user@accumulo.apache.org
Subject: RE: pragmattically invoking map reduce job from a .war deployed
in tomcat
Thanks, Bob. I'll try this out as well. When you say you deploy your
job jars, are you just referring to a jar of the entire project
(including the map reduce jobs), or do you just jar up the specific
packages that contain the map reduce jobs, mappers, and reducers? Also,
am I correct in assuming that you're not making runnable jars?
Thanks,
Duane
From: Bob.Thorman@l-3com.com [mailto:Bob.Thorman@l-3com.com]
Sent: Friday, February 08, 2013 2:26 PM
To: user@accumulo.apache.org
Subject: RE: pragmattically invoking map reduce job from a .war deployed
in tomcat
I do this same thing, but during deployment of my war I also deploy my
job jars and their dependencies to $ACCUMULO_HOME\lib\ext to make sure
they are available to the ensemble.
From: Cornish, Duane C. [mailto:Duane.Cornish@jhuapl.edu]
Sent: Friday, February 08, 2013 11:25
To: user@accumulo.apache.org
Subject: pragmattically invoking map reduce job from a .war deployed in
tomcat
I have an accumulo instance running on a small 4 node cluster. The
namenode is running tomcat. I am using a restful service to allow
clients to access accumulo and kick off map reduce jobs.
Previously, I had been creating runnable jars of my map reduce jobs.
The restful service would make a command line call to run the jar. i.e.
"java -jar mapreducejob.java -arg1 -arg2 etc."
I am now trying to switch these jobs to be called programmatically. I
am aware that I need to pass a comma separated list of jars into the
ToolRunner.run call. i.e.
arg[0] = "-libjar";
arg[1] = "accumulo-core-1.4.1,jar2" etc.
arg[2] = "myArg1";
arg[3] = "myArg2";
etc.
When I run my job, I get an error in in my map tasks in the jobtracker
Admin webpage:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
<package.Name>.<functionNameOfMyMapper>
I don't know why the system can't find the mapper function. It is in
the same package as my Job file and they should all be contained in the
.war file that is deployed in tomcat.
Has anyone had success calling a map reduce job programmatically from
within tomcat?
Thanks,
Duane
RE: pragmattically invoking map reduce job from a .war deployed in
tomcat
Posted by "Cornish, Duane C." <Du...@jhuapl.edu>.
Thanks, Bob. I'll try this out as well. When you say you deploy your job jars, are you just referring to a jar of the entire project (including the map reduce jobs), or do you just jar up the specific packages that contain the map reduce jobs, mappers, and reducers? Also, am I correct in assuming that you're not making runnable jars?
Thanks,
Duane
From: Bob.Thorman@l-3com.com [mailto:Bob.Thorman@l-3com.com]
Sent: Friday, February 08, 2013 2:26 PM
To: user@accumulo.apache.org
Subject: RE: pragmattically invoking map reduce job from a .war deployed in tomcat
I do this same thing, but during deployment of my war I also deploy my job jars and their dependencies to $ACCUMULO_HOME\lib\ext to make sure they are available to the ensemble.
From: Cornish, Duane C. [mailto:Duane.Cornish@jhuapl.edu]
Sent: Friday, February 08, 2013 11:25
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: pragmattically invoking map reduce job from a .war deployed in tomcat
I have an accumulo instance running on a small 4 node cluster. The namenode is running tomcat. I am using a restful service to allow clients to access accumulo and kick off map reduce jobs.
Previously, I had been creating runnable jars of my map reduce jobs. The restful service would make a command line call to run the jar. i.e. "java -jar mapreducejob.java -arg1 -arg2 etc."
I am now trying to switch these jobs to be called programmatically. I am aware that I need to pass a comma separated list of jars into the ToolRunner.run call. i.e.
arg[0] = "-libjar";
arg[1] = "accumulo-core-1.4.1,jar2" etc.
arg[2] = "myArg1";
arg[3] = "myArg2";
etc.
When I run my job, I get an error in in my map tasks in the jobtracker Admin webpage:
java.lang.RuntimeException: java.lang.ClassNotFoundException: <package.Name>.<functionNameOfMyMapper>
I don't know why the system can't find the mapper function. It is in the same package as my Job file and they should all be contained in the .war file that is deployed in tomcat.
Has anyone had success calling a map reduce job programmatically from within tomcat?
Thanks,
Duane
RE: pragmattically invoking map reduce job from a .war deployed in tomcat
Posted by Bo...@l-3com.com.
I do this same thing, but during deployment of my war I also deploy my
job jars and their dependencies to $ACCUMULO_HOME\lib\ext to make sure
they are available to the ensemble.
From: Cornish, Duane C. [mailto:Duane.Cornish@jhuapl.edu]
Sent: Friday, February 08, 2013 11:25
To: user@accumulo.apache.org
Subject: pragmattically invoking map reduce job from a .war deployed in
tomcat
I have an accumulo instance running on a small 4 node cluster. The
namenode is running tomcat. I am using a restful service to allow
clients to access accumulo and kick off map reduce jobs.
Previously, I had been creating runnable jars of my map reduce jobs.
The restful service would make a command line call to run the jar. i.e.
"java -jar mapreducejob.java -arg1 -arg2 etc."
I am now trying to switch these jobs to be called programmatically. I
am aware that I need to pass a comma separated list of jars into the
ToolRunner.run call. i.e.
arg[0] = "-libjar";
arg[1] = "accumulo-core-1.4.1,jar2" etc.
arg[2] = "myArg1";
arg[3] = "myArg2";
etc.
When I run my job, I get an error in in my map tasks in the jobtracker
Admin webpage:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
<package.Name>.<functionNameOfMyMapper>
I don't know why the system can't find the mapper function. It is in
the same package as my Job file and they should all be contained in the
.war file that is deployed in tomcat.
Has anyone had success calling a map reduce job programmatically from
within tomcat?
Thanks,
Duane
RE: pragmattically invoking map reduce job from a .war deployed in
tomcat
Posted by "Cornish, Duane C." <Du...@jhuapl.edu>.
Thanks for the quick response, Billie! You are correct it should be -libjars. That was just a typo in my email. You mentioned that I'm supposed to have my job jar and other dependencies in the lib folder of the webapp. I'll give that a shot!
Thanks,
Duane
From: Billie Rinaldi [mailto:billie@apache.org]
Sent: Friday, February 08, 2013 12:46 PM
To: user@accumulo.apache.org
Subject: Re: pragmattically invoking map reduce job from a .war deployed in tomcat
On Fri, Feb 8, 2013 at 12:25 PM, Cornish, Duane C. <Du...@jhuapl.edu>> wrote:
I have an accumulo instance running on a small 4 node cluster. The namenode is running tomcat. I am using a restful service to allow clients to access accumulo and kick off map reduce jobs.
Previously, I had been creating runnable jars of my map reduce jobs. The restful service would make a command line call to run the jar. i.e. "java -jar mapreducejob.java -arg1 -arg2 etc."
I am now trying to switch these jobs to be called programmatically. I am aware that I need to pass a comma separated list of jars into the ToolRunner.run call. i.e.
arg[0] = "-libjar";
This should be -libjars. However, that is probably not the issue because the job classes themselves are not passed in via libjars. Searching around, I saw some indication that you are supposed to have your job jar and other dependencies in the lib folder for your webapp. Are you doing that?
Billie
arg[1] = "accumulo-core-1.4.1,jar2" etc.
arg[2] = "myArg1";
arg[3] = "myArg2";
etc.
When I run my job, I get an error in in my map tasks in the jobtracker Admin webpage:
java.lang.RuntimeException: java.lang.ClassNotFoundException: <package.Name>.<functionNameOfMyMapper>
I don't know why the system can't find the mapper function. It is in the same package as my Job file and they should all be contained in the .war file that is deployed in tomcat.
Has anyone had success calling a map reduce job programmatically from within tomcat?
Thanks,
Duane
Re: pragmattically invoking map reduce job from a .war deployed in tomcat
Posted by Billie Rinaldi <bi...@apache.org>.
On Fri, Feb 8, 2013 at 12:25 PM, Cornish, Duane C. <Duane.Cornish@jhuapl.edu
> wrote:
> I have an accumulo instance running on a small 4 node cluster. The
> namenode is running tomcat. I am using a restful service to allow clients
> to access accumulo and kick off map reduce jobs.****
>
> ** **
>
> Previously, I had been creating runnable jars of my map reduce jobs. The
> restful service would make a command line call to run the jar. i.e. “java
> –jar mapreducejob.java –arg1 –arg2 etc.” ****
>
> ** **
>
> I am now trying to switch these jobs to be called programmatically. I am
> aware that I need to pass a comma separated list of jars into the
> ToolRunner.run call. i.e. ****
>
> ** **
>
> arg[0] = “-libjar”;
>
This should be -libjars. However, that is probably not the issue because
the job classes themselves are not passed in via libjars. Searching
around, I saw some indication that you are supposed to have your job jar
and other dependencies in the lib folder for your webapp. Are you doing
that?
Billie
> ****
>
> arg[1] = “accumulo-core-1.4.1,jar2” etc.****
>
> arg[2] = “myArg1”;****
>
> arg[3] = “myArg2”;****
>
> etc.****
>
> ** **
>
> When I run my job, I get an error in in my map tasks in the jobtracker
> Admin webpage:****
>
> ** **
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException: <package.Name>.<functionNameOfMyMapper> ****
>
> ** **
>
> I don’t know why the system can’t find the mapper function. It is in the
> same package as my Job file and they should all be contained in the .war
> file that is deployed in tomcat. ****
>
> ** **
>
> Has anyone had success calling a map reduce job programmatically from
> within tomcat?****
>
> ** **
>
> Thanks,****
>
> Duane****
>