You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Istabrak Abdul-Fatah <if...@gmail.com> on 2015/10/07 16:28:33 UTC
Invoking a MapRed job via Runtime class
Greetings to all,
Is it possible to invoke a MapReduce job from another java class using the
runtime command? If not, what are the alternatives?
Here is a sample code snippet:
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
-Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
} catch (IOException e) {
e.printStackTrace();
}
Regards,
Ista
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
Well seems more like not a Hadoop related query,
Possible solutions what i can think of is
1. Code refactoring such that some methods are exposed in your MR driver
class and call the method directly.
2. Not sure whats the error in your code but possible reason would be
process's error and out put needs to be read using Stream gobbler.
P.S. you can have "*job.waitForCompletion(true);" *itself
+ Naga
On Fri, Oct 9, 2015 at 12:36 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
>
> I have tried having the client driver as the mapreduce job itself and as
> you know it works with no issues.
> The intention is to have a java driver that can trigger these mapreduce
> jobs.
> This driver may run inside an app server or something similar (on the
> Hadoop cluster admin node) and respond to remote client calls to trigger
> such jobs since these remote clients are not supposed to have any Hadoop
> jars /config files.
>
> So my question is: Would it be possible to invoke such a mapreduce job
> from a java driver that has no Mapreduce code other than the invocation
> statement?
>
> BR
>
> Ista
>
> On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
> naganarasimha.gr@gmail.com> wrote:
>
>> Hi Ista,
>> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
>> main method if your intention is to just run the MR job and not wait for
>> the result. May be it would be good(helpful) for you to go through the java
>> docs for Job.
>> Whats the intention of running this job ? Testing of the MR or the
>> client driver ?
>>
>> + Naga
>>
>> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Hi Naga,
>>> Thx for the reply.
>>> Here is a high level description of what I am trying to achieve (if
>>> possible):
>>>
>>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>>> being implemented, compiled and the respective generated jars are there as
>>> well.
>>>
>>> In the same node, I have a java application running and am trying to
>>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>>> job" ).
>>>
>>> The code snippets are provided below.
>>> Currently, it is not working and I am wondering if I have missed
>>> something or have incorrect impl.
>>>
>>> How would the code changes if the calling application would like to wait
>>> for the mapreduce job to complete?
>>>
>>> Thx
>>>
>>> Ista
>>>
>>>
>>> #################################################
>>>
>>> The java application code snippet
>>> ===========================
>>>
>>> public class javaDriver2 {
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> System.out.println("caught exception" + e.toString());
>>> }
>>> catch (SecurityException se) {
>>> se.printStackTrace();
>>> System.out.println("caught exception" + se.toString());
>>> }
>>>
>>> }
>>>
>>> ///////////////////////////////////
>>>
>>> The Mapreduce job code:
>>> =====================
>>>
>>> public static void main(String[] args) throws Exception {
>>> Configuration conf = new Configuration();
>>> int res = ToolRunner.run(conf, new AvgEight(), args);
>>> System.exit(res);
>>> }
>>>
>>> @Override
>>> public int run(String[] args) throws Exception {
>>>
>>> // When implementing tool
>>> Configuration conf = this.getConf();
>>>
>>> // Create job
>>> Job job = Job.getInstance(conf, "AvgEight");
>>> job.setJarByClass(AvgEight.class);
>>>
>>> // Setup MapReduce job
>>> // Do not specify the number of Reducers
>>> job.setMapperClass(Map2.class);
>>> job.setReducerClass(Reduce2.class);
>>>
>>> job.setOutputKeyClass(Text.class);
>>>
>>> //job.setOutputValueClass(IntWritable.class);
>>> //job.setOutputValueClass(ArrayWritable.class);
>>> job.setOutputValueClass(LongArrayWritable.class);
>>>
>>> job.setInputFormatClass(TextInputFormat.class);
>>> job.setOutputFormatClass(TextOutputFormat.class);
>>>
>>> FileInputFormat.addInputPath(job, new Path(args[0]));
>>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>>
>>> job.submit();
>>> return 1;
>>> }
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>>> garlanaganarasimha@huawei.com> wrote:
>>>
>>>> Hi Ista,
>>>> IIUC you just want it to be submitted and not wait till it finish and
>>>> print the stats ?
>>>> Basically sample code which you might be referring will be creating the
>>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>>> configurations calling
>>>> *job.waitForCompletion(true);* Instead you can just call
>>>> *job.submit();*
>>>>
>>>> Regards,
>>>> Naga
>>>> ------------------------------
>>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>>> *Sent:* Wednesday, October 07, 2015 19:58
>>>> *To:* user@hadoop.apache.org
>>>> *Subject:* Invoking a MapRed job via Runtime class
>>>>
>>>> Greetings to all,
>>>> Is it possible to invoke a MapReduce job from another java class using
>>>> the runtime command? If not, what are the alternatives?
>>>>
>>>> Here is a sample code snippet:
>>>>
>>>> public static void main(String[] args) {
>>>> Runtime rt = Runtime.getRuntime();
>>>> try {
>>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>>> } catch (IOException e) {
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> Regards,
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
Well seems more like not a Hadoop related query,
Possible solutions what i can think of is
1. Code refactoring such that some methods are exposed in your MR driver
class and call the method directly.
2. Not sure whats the error in your code but possible reason would be
process's error and out put needs to be read using Stream gobbler.
P.S. you can have "*job.waitForCompletion(true);" *itself
+ Naga
On Fri, Oct 9, 2015 at 12:36 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
>
> I have tried having the client driver as the mapreduce job itself and as
> you know it works with no issues.
> The intention is to have a java driver that can trigger these mapreduce
> jobs.
> This driver may run inside an app server or something similar (on the
> Hadoop cluster admin node) and respond to remote client calls to trigger
> such jobs since these remote clients are not supposed to have any Hadoop
> jars /config files.
>
> So my question is: Would it be possible to invoke such a mapreduce job
> from a java driver that has no Mapreduce code other than the invocation
> statement?
>
> BR
>
> Ista
>
> On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
> naganarasimha.gr@gmail.com> wrote:
>
>> Hi Ista,
>> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
>> main method if your intention is to just run the MR job and not wait for
>> the result. May be it would be good(helpful) for you to go through the java
>> docs for Job.
>> Whats the intention of running this job ? Testing of the MR or the
>> client driver ?
>>
>> + Naga
>>
>> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Hi Naga,
>>> Thx for the reply.
>>> Here is a high level description of what I am trying to achieve (if
>>> possible):
>>>
>>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>>> being implemented, compiled and the respective generated jars are there as
>>> well.
>>>
>>> In the same node, I have a java application running and am trying to
>>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>>> job" ).
>>>
>>> The code snippets are provided below.
>>> Currently, it is not working and I am wondering if I have missed
>>> something or have incorrect impl.
>>>
>>> How would the code changes if the calling application would like to wait
>>> for the mapreduce job to complete?
>>>
>>> Thx
>>>
>>> Ista
>>>
>>>
>>> #################################################
>>>
>>> The java application code snippet
>>> ===========================
>>>
>>> public class javaDriver2 {
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> System.out.println("caught exception" + e.toString());
>>> }
>>> catch (SecurityException se) {
>>> se.printStackTrace();
>>> System.out.println("caught exception" + se.toString());
>>> }
>>>
>>> }
>>>
>>> ///////////////////////////////////
>>>
>>> The Mapreduce job code:
>>> =====================
>>>
>>> public static void main(String[] args) throws Exception {
>>> Configuration conf = new Configuration();
>>> int res = ToolRunner.run(conf, new AvgEight(), args);
>>> System.exit(res);
>>> }
>>>
>>> @Override
>>> public int run(String[] args) throws Exception {
>>>
>>> // When implementing tool
>>> Configuration conf = this.getConf();
>>>
>>> // Create job
>>> Job job = Job.getInstance(conf, "AvgEight");
>>> job.setJarByClass(AvgEight.class);
>>>
>>> // Setup MapReduce job
>>> // Do not specify the number of Reducers
>>> job.setMapperClass(Map2.class);
>>> job.setReducerClass(Reduce2.class);
>>>
>>> job.setOutputKeyClass(Text.class);
>>>
>>> //job.setOutputValueClass(IntWritable.class);
>>> //job.setOutputValueClass(ArrayWritable.class);
>>> job.setOutputValueClass(LongArrayWritable.class);
>>>
>>> job.setInputFormatClass(TextInputFormat.class);
>>> job.setOutputFormatClass(TextOutputFormat.class);
>>>
>>> FileInputFormat.addInputPath(job, new Path(args[0]));
>>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>>
>>> job.submit();
>>> return 1;
>>> }
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>>> garlanaganarasimha@huawei.com> wrote:
>>>
>>>> Hi Ista,
>>>> IIUC you just want it to be submitted and not wait till it finish and
>>>> print the stats ?
>>>> Basically sample code which you might be referring will be creating the
>>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>>> configurations calling
>>>> *job.waitForCompletion(true);* Instead you can just call
>>>> *job.submit();*
>>>>
>>>> Regards,
>>>> Naga
>>>> ------------------------------
>>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>>> *Sent:* Wednesday, October 07, 2015 19:58
>>>> *To:* user@hadoop.apache.org
>>>> *Subject:* Invoking a MapRed job via Runtime class
>>>>
>>>> Greetings to all,
>>>> Is it possible to invoke a MapReduce job from another java class using
>>>> the runtime command? If not, what are the alternatives?
>>>>
>>>> Here is a sample code snippet:
>>>>
>>>> public static void main(String[] args) {
>>>> Runtime rt = Runtime.getRuntime();
>>>> try {
>>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>>> } catch (IOException e) {
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> Regards,
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
Well seems more like not a Hadoop related query,
Possible solutions what i can think of is
1. Code refactoring such that some methods are exposed in your MR driver
class and call the method directly.
2. Not sure whats the error in your code but possible reason would be
process's error and out put needs to be read using Stream gobbler.
P.S. you can have "*job.waitForCompletion(true);" *itself
+ Naga
On Fri, Oct 9, 2015 at 12:36 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
>
> I have tried having the client driver as the mapreduce job itself and as
> you know it works with no issues.
> The intention is to have a java driver that can trigger these mapreduce
> jobs.
> This driver may run inside an app server or something similar (on the
> Hadoop cluster admin node) and respond to remote client calls to trigger
> such jobs since these remote clients are not supposed to have any Hadoop
> jars /config files.
>
> So my question is: Would it be possible to invoke such a mapreduce job
> from a java driver that has no Mapreduce code other than the invocation
> statement?
>
> BR
>
> Ista
>
> On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
> naganarasimha.gr@gmail.com> wrote:
>
>> Hi Ista,
>> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
>> main method if your intention is to just run the MR job and not wait for
>> the result. May be it would be good(helpful) for you to go through the java
>> docs for Job.
>> Whats the intention of running this job ? Testing of the MR or the
>> client driver ?
>>
>> + Naga
>>
>> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Hi Naga,
>>> Thx for the reply.
>>> Here is a high level description of what I am trying to achieve (if
>>> possible):
>>>
>>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>>> being implemented, compiled and the respective generated jars are there as
>>> well.
>>>
>>> In the same node, I have a java application running and am trying to
>>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>>> job" ).
>>>
>>> The code snippets are provided below.
>>> Currently, it is not working and I am wondering if I have missed
>>> something or have incorrect impl.
>>>
>>> How would the code changes if the calling application would like to wait
>>> for the mapreduce job to complete?
>>>
>>> Thx
>>>
>>> Ista
>>>
>>>
>>> #################################################
>>>
>>> The java application code snippet
>>> ===========================
>>>
>>> public class javaDriver2 {
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> System.out.println("caught exception" + e.toString());
>>> }
>>> catch (SecurityException se) {
>>> se.printStackTrace();
>>> System.out.println("caught exception" + se.toString());
>>> }
>>>
>>> }
>>>
>>> ///////////////////////////////////
>>>
>>> The Mapreduce job code:
>>> =====================
>>>
>>> public static void main(String[] args) throws Exception {
>>> Configuration conf = new Configuration();
>>> int res = ToolRunner.run(conf, new AvgEight(), args);
>>> System.exit(res);
>>> }
>>>
>>> @Override
>>> public int run(String[] args) throws Exception {
>>>
>>> // When implementing tool
>>> Configuration conf = this.getConf();
>>>
>>> // Create job
>>> Job job = Job.getInstance(conf, "AvgEight");
>>> job.setJarByClass(AvgEight.class);
>>>
>>> // Setup MapReduce job
>>> // Do not specify the number of Reducers
>>> job.setMapperClass(Map2.class);
>>> job.setReducerClass(Reduce2.class);
>>>
>>> job.setOutputKeyClass(Text.class);
>>>
>>> //job.setOutputValueClass(IntWritable.class);
>>> //job.setOutputValueClass(ArrayWritable.class);
>>> job.setOutputValueClass(LongArrayWritable.class);
>>>
>>> job.setInputFormatClass(TextInputFormat.class);
>>> job.setOutputFormatClass(TextOutputFormat.class);
>>>
>>> FileInputFormat.addInputPath(job, new Path(args[0]));
>>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>>
>>> job.submit();
>>> return 1;
>>> }
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>>> garlanaganarasimha@huawei.com> wrote:
>>>
>>>> Hi Ista,
>>>> IIUC you just want it to be submitted and not wait till it finish and
>>>> print the stats ?
>>>> Basically sample code which you might be referring will be creating the
>>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>>> configurations calling
>>>> *job.waitForCompletion(true);* Instead you can just call
>>>> *job.submit();*
>>>>
>>>> Regards,
>>>> Naga
>>>> ------------------------------
>>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>>> *Sent:* Wednesday, October 07, 2015 19:58
>>>> *To:* user@hadoop.apache.org
>>>> *Subject:* Invoking a MapRed job via Runtime class
>>>>
>>>> Greetings to all,
>>>> Is it possible to invoke a MapReduce job from another java class using
>>>> the runtime command? If not, what are the alternatives?
>>>>
>>>> Here is a sample code snippet:
>>>>
>>>> public static void main(String[] args) {
>>>> Runtime rt = Runtime.getRuntime();
>>>> try {
>>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>>> } catch (IOException e) {
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> Regards,
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
Well seems more like not a Hadoop related query,
Possible solutions what i can think of is
1. Code refactoring such that some methods are exposed in your MR driver
class and call the method directly.
2. Not sure whats the error in your code but possible reason would be
process's error and out put needs to be read using Stream gobbler.
P.S. you can have "*job.waitForCompletion(true);" *itself
+ Naga
On Fri, Oct 9, 2015 at 12:36 AM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
>
> I have tried having the client driver as the mapreduce job itself and as
> you know it works with no issues.
> The intention is to have a java driver that can trigger these mapreduce
> jobs.
> This driver may run inside an app server or something similar (on the
> Hadoop cluster admin node) and respond to remote client calls to trigger
> such jobs since these remote clients are not supposed to have any Hadoop
> jars /config files.
>
> So my question is: Would it be possible to invoke such a mapreduce job
> from a java driver that has no Mapreduce code other than the invocation
> statement?
>
> BR
>
> Ista
>
> On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
> naganarasimha.gr@gmail.com> wrote:
>
>> Hi Ista,
>> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
>> main method if your intention is to just run the MR job and not wait for
>> the result. May be it would be good(helpful) for you to go through the java
>> docs for Job.
>> Whats the intention of running this job ? Testing of the MR or the
>> client driver ?
>>
>> + Naga
>>
>> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
>> wrote:
>>
>>> Hi Naga,
>>> Thx for the reply.
>>> Here is a high level description of what I am trying to achieve (if
>>> possible):
>>>
>>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>>> being implemented, compiled and the respective generated jars are there as
>>> well.
>>>
>>> In the same node, I have a java application running and am trying to
>>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>>> job" ).
>>>
>>> The code snippets are provided below.
>>> Currently, it is not working and I am wondering if I have missed
>>> something or have incorrect impl.
>>>
>>> How would the code changes if the calling application would like to wait
>>> for the mapreduce job to complete?
>>>
>>> Thx
>>>
>>> Ista
>>>
>>>
>>> #################################################
>>>
>>> The java application code snippet
>>> ===========================
>>>
>>> public class javaDriver2 {
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> System.out.println("caught exception" + e.toString());
>>> }
>>> catch (SecurityException se) {
>>> se.printStackTrace();
>>> System.out.println("caught exception" + se.toString());
>>> }
>>>
>>> }
>>>
>>> ///////////////////////////////////
>>>
>>> The Mapreduce job code:
>>> =====================
>>>
>>> public static void main(String[] args) throws Exception {
>>> Configuration conf = new Configuration();
>>> int res = ToolRunner.run(conf, new AvgEight(), args);
>>> System.exit(res);
>>> }
>>>
>>> @Override
>>> public int run(String[] args) throws Exception {
>>>
>>> // When implementing tool
>>> Configuration conf = this.getConf();
>>>
>>> // Create job
>>> Job job = Job.getInstance(conf, "AvgEight");
>>> job.setJarByClass(AvgEight.class);
>>>
>>> // Setup MapReduce job
>>> // Do not specify the number of Reducers
>>> job.setMapperClass(Map2.class);
>>> job.setReducerClass(Reduce2.class);
>>>
>>> job.setOutputKeyClass(Text.class);
>>>
>>> //job.setOutputValueClass(IntWritable.class);
>>> //job.setOutputValueClass(ArrayWritable.class);
>>> job.setOutputValueClass(LongArrayWritable.class);
>>>
>>> job.setInputFormatClass(TextInputFormat.class);
>>> job.setOutputFormatClass(TextOutputFormat.class);
>>>
>>> FileInputFormat.addInputPath(job, new Path(args[0]));
>>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>>
>>> job.submit();
>>> return 1;
>>> }
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>>> garlanaganarasimha@huawei.com> wrote:
>>>
>>>> Hi Ista,
>>>> IIUC you just want it to be submitted and not wait till it finish and
>>>> print the stats ?
>>>> Basically sample code which you might be referring will be creating the
>>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>>> configurations calling
>>>> *job.waitForCompletion(true);* Instead you can just call
>>>> *job.submit();*
>>>>
>>>> Regards,
>>>> Naga
>>>> ------------------------------
>>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>>> *Sent:* Wednesday, October 07, 2015 19:58
>>>> *To:* user@hadoop.apache.org
>>>> *Subject:* Invoking a MapRed job via Runtime class
>>>>
>>>> Greetings to all,
>>>> Is it possible to invoke a MapReduce job from another java class using
>>>> the runtime command? If not, what are the alternatives?
>>>>
>>>> Here is a sample code snippet:
>>>>
>>>> public static void main(String[] args) {
>>>> Runtime rt = Runtime.getRuntime();
>>>> try {
>>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>>> } catch (IOException e) {
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> Regards,
>>>>
>>>> Ista
>>>>
>>>
>>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
I have tried having the client driver as the mapreduce job itself and as
you know it works with no issues.
The intention is to have a java driver that can trigger these mapreduce
jobs.
This driver may run inside an app server or something similar (on the
Hadoop cluster admin node) and respond to remote client calls to trigger
such jobs since these remote clients are not supposed to have any Hadoop
jars /config files.
So my question is: Would it be possible to invoke such a mapreduce job from
a java driver that has no Mapreduce code other than the invocation
statement?
BR
Ista
On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
naganarasimha.gr@gmail.com> wrote:
> Hi Ista,
> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
> main method if your intention is to just run the MR job and not wait for
> the result. May be it would be good(helpful) for you to go through the java
> docs for Job.
> Whats the intention of running this job ? Testing of the MR or the client
> driver ?
>
> + Naga
>
> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the reply.
>> Here is a high level description of what I am trying to achieve (if
>> possible):
>>
>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>> being implemented, compiled and the respective generated jars are there as
>> well.
>>
>> In the same node, I have a java application running and am trying to
>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>> job" ).
>>
>> The code snippets are provided below.
>> Currently, it is not working and I am wondering if I have missed
>> something or have incorrect impl.
>>
>> How would the code changes if the calling application would like to wait
>> for the mapreduce job to complete?
>>
>> Thx
>>
>> Ista
>>
>>
>> #################################################
>>
>> The java application code snippet
>> ===========================
>>
>> public class javaDriver2 {
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> System.out.println("caught exception" + e.toString());
>> }
>> catch (SecurityException se) {
>> se.printStackTrace();
>> System.out.println("caught exception" + se.toString());
>> }
>>
>> }
>>
>> ///////////////////////////////////
>>
>> The Mapreduce job code:
>> =====================
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> int res = ToolRunner.run(conf, new AvgEight(), args);
>> System.exit(res);
>> }
>>
>> @Override
>> public int run(String[] args) throws Exception {
>>
>> // When implementing tool
>> Configuration conf = this.getConf();
>>
>> // Create job
>> Job job = Job.getInstance(conf, "AvgEight");
>> job.setJarByClass(AvgEight.class);
>>
>> // Setup MapReduce job
>> // Do not specify the number of Reducers
>> job.setMapperClass(Map2.class);
>> job.setReducerClass(Reduce2.class);
>>
>> job.setOutputKeyClass(Text.class);
>>
>> //job.setOutputValueClass(IntWritable.class);
>> //job.setOutputValueClass(ArrayWritable.class);
>> job.setOutputValueClass(LongArrayWritable.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);
>>
>> FileInputFormat.addInputPath(job, new Path(args[0]));
>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>
>> job.submit();
>> return 1;
>> }
>>
>>
>>
>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>> IIUC you just want it to be submitted and not wait till it finish and
>>> print the stats ?
>>> Basically sample code which you might be referring will be creating the
>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>> configurations calling
>>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>>
>>> Regards,
>>> Naga
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Wednesday, October 07, 2015 19:58
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Invoking a MapRed job via Runtime class
>>>
>>> Greetings to all,
>>> Is it possible to invoke a MapReduce job from another java class using
>>> the runtime command? If not, what are the alternatives?
>>>
>>> Here is a sample code snippet:
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> }
>>>
>>> Regards,
>>>
>>> Ista
>>>
>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
I have tried having the client driver as the mapreduce job itself and as
you know it works with no issues.
The intention is to have a java driver that can trigger these mapreduce
jobs.
This driver may run inside an app server or something similar (on the
Hadoop cluster admin node) and respond to remote client calls to trigger
such jobs since these remote clients are not supposed to have any Hadoop
jars /config files.
So my question is: Would it be possible to invoke such a mapreduce job from
a java driver that has no Mapreduce code other than the invocation
statement?
BR
Ista
On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
naganarasimha.gr@gmail.com> wrote:
> Hi Ista,
> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
> main method if your intention is to just run the MR job and not wait for
> the result. May be it would be good(helpful) for you to go through the java
> docs for Job.
> Whats the intention of running this job ? Testing of the MR or the client
> driver ?
>
> + Naga
>
> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the reply.
>> Here is a high level description of what I am trying to achieve (if
>> possible):
>>
>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>> being implemented, compiled and the respective generated jars are there as
>> well.
>>
>> In the same node, I have a java application running and am trying to
>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>> job" ).
>>
>> The code snippets are provided below.
>> Currently, it is not working and I am wondering if I have missed
>> something or have incorrect impl.
>>
>> How would the code changes if the calling application would like to wait
>> for the mapreduce job to complete?
>>
>> Thx
>>
>> Ista
>>
>>
>> #################################################
>>
>> The java application code snippet
>> ===========================
>>
>> public class javaDriver2 {
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> System.out.println("caught exception" + e.toString());
>> }
>> catch (SecurityException se) {
>> se.printStackTrace();
>> System.out.println("caught exception" + se.toString());
>> }
>>
>> }
>>
>> ///////////////////////////////////
>>
>> The Mapreduce job code:
>> =====================
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> int res = ToolRunner.run(conf, new AvgEight(), args);
>> System.exit(res);
>> }
>>
>> @Override
>> public int run(String[] args) throws Exception {
>>
>> // When implementing tool
>> Configuration conf = this.getConf();
>>
>> // Create job
>> Job job = Job.getInstance(conf, "AvgEight");
>> job.setJarByClass(AvgEight.class);
>>
>> // Setup MapReduce job
>> // Do not specify the number of Reducers
>> job.setMapperClass(Map2.class);
>> job.setReducerClass(Reduce2.class);
>>
>> job.setOutputKeyClass(Text.class);
>>
>> //job.setOutputValueClass(IntWritable.class);
>> //job.setOutputValueClass(ArrayWritable.class);
>> job.setOutputValueClass(LongArrayWritable.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);
>>
>> FileInputFormat.addInputPath(job, new Path(args[0]));
>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>
>> job.submit();
>> return 1;
>> }
>>
>>
>>
>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>> IIUC you just want it to be submitted and not wait till it finish and
>>> print the stats ?
>>> Basically sample code which you might be referring will be creating the
>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>> configurations calling
>>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>>
>>> Regards,
>>> Naga
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Wednesday, October 07, 2015 19:58
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Invoking a MapRed job via Runtime class
>>>
>>> Greetings to all,
>>> Is it possible to invoke a MapReduce job from another java class using
>>> the runtime command? If not, what are the alternatives?
>>>
>>> Here is a sample code snippet:
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> }
>>>
>>> Regards,
>>>
>>> Ista
>>>
>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
I have tried having the client driver as the mapreduce job itself and as
you know it works with no issues.
The intention is to have a java driver that can trigger these mapreduce
jobs.
This driver may run inside an app server or something similar (on the
Hadoop cluster admin node) and respond to remote client calls to trigger
such jobs since these remote clients are not supposed to have any Hadoop
jars /config files.
So my question is: Would it be possible to invoke such a mapreduce job from
a java driver that has no Mapreduce code other than the invocation
statement?
BR
Ista
On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
naganarasimha.gr@gmail.com> wrote:
> Hi Ista,
> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
> main method if your intention is to just run the MR job and not wait for
> the result. May be it would be good(helpful) for you to go through the java
> docs for Job.
> Whats the intention of running this job ? Testing of the MR or the client
> driver ?
>
> + Naga
>
> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the reply.
>> Here is a high level description of what I am trying to achieve (if
>> possible):
>>
>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>> being implemented, compiled and the respective generated jars are there as
>> well.
>>
>> In the same node, I have a java application running and am trying to
>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>> job" ).
>>
>> The code snippets are provided below.
>> Currently, it is not working and I am wondering if I have missed
>> something or have incorrect impl.
>>
>> How would the code changes if the calling application would like to wait
>> for the mapreduce job to complete?
>>
>> Thx
>>
>> Ista
>>
>>
>> #################################################
>>
>> The java application code snippet
>> ===========================
>>
>> public class javaDriver2 {
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> System.out.println("caught exception" + e.toString());
>> }
>> catch (SecurityException se) {
>> se.printStackTrace();
>> System.out.println("caught exception" + se.toString());
>> }
>>
>> }
>>
>> ///////////////////////////////////
>>
>> The Mapreduce job code:
>> =====================
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> int res = ToolRunner.run(conf, new AvgEight(), args);
>> System.exit(res);
>> }
>>
>> @Override
>> public int run(String[] args) throws Exception {
>>
>> // When implementing tool
>> Configuration conf = this.getConf();
>>
>> // Create job
>> Job job = Job.getInstance(conf, "AvgEight");
>> job.setJarByClass(AvgEight.class);
>>
>> // Setup MapReduce job
>> // Do not specify the number of Reducers
>> job.setMapperClass(Map2.class);
>> job.setReducerClass(Reduce2.class);
>>
>> job.setOutputKeyClass(Text.class);
>>
>> //job.setOutputValueClass(IntWritable.class);
>> //job.setOutputValueClass(ArrayWritable.class);
>> job.setOutputValueClass(LongArrayWritable.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);
>>
>> FileInputFormat.addInputPath(job, new Path(args[0]));
>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>
>> job.submit();
>> return 1;
>> }
>>
>>
>>
>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>> IIUC you just want it to be submitted and not wait till it finish and
>>> print the stats ?
>>> Basically sample code which you might be referring will be creating the
>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>> configurations calling
>>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>>
>>> Regards,
>>> Naga
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Wednesday, October 07, 2015 19:58
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Invoking a MapRed job via Runtime class
>>>
>>> Greetings to all,
>>> Is it possible to invoke a MapReduce job from another java class using
>>> the runtime command? If not, what are the alternatives?
>>>
>>> Here is a sample code snippet:
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> }
>>>
>>> Regards,
>>>
>>> Ista
>>>
>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
I have tried having the client driver as the mapreduce job itself and as
you know it works with no issues.
The intention is to have a java driver that can trigger these mapreduce
jobs.
This driver may run inside an app server or something similar (on the
Hadoop cluster admin node) and respond to remote client calls to trigger
such jobs since these remote clients are not supposed to have any Hadoop
jars /config files.
So my question is: Would it be possible to invoke such a mapreduce job from
a java driver that has no Mapreduce code other than the invocation
statement?
BR
Ista
On Thu, Oct 8, 2015 at 1:19 PM, Naganarasimha Garla <
naganarasimha.gr@gmail.com> wrote:
> Hi Ista,
> What i meant was to have your "mapReduce" job's run code in your "javaDriver2"'s
> main method if your intention is to just run the MR job and not wait for
> the result. May be it would be good(helpful) for you to go through the java
> docs for Job.
> Whats the intention of running this job ? Testing of the MR or the client
> driver ?
>
> + Naga
>
> On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
> wrote:
>
>> Hi Naga,
>> Thx for the reply.
>> Here is a high level description of what I am trying to achieve (if
>> possible):
>>
>> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
>> being implemented, compiled and the respective generated jars are there as
>> well.
>>
>> In the same node, I have a java application running and am trying to
>> start one of the existing mapreduce jobs (as you called it "to sbumit the
>> job" ).
>>
>> The code snippets are provided below.
>> Currently, it is not working and I am wondering if I have missed
>> something or have incorrect impl.
>>
>> How would the code changes if the calling application would like to wait
>> for the mapreduce job to complete?
>>
>> Thx
>>
>> Ista
>>
>>
>> #################################################
>>
>> The java application code snippet
>> ===========================
>>
>> public class javaDriver2 {
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
>> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> System.out.println("caught exception" + e.toString());
>> }
>> catch (SecurityException se) {
>> se.printStackTrace();
>> System.out.println("caught exception" + se.toString());
>> }
>>
>> }
>>
>> ///////////////////////////////////
>>
>> The Mapreduce job code:
>> =====================
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration();
>> int res = ToolRunner.run(conf, new AvgEight(), args);
>> System.exit(res);
>> }
>>
>> @Override
>> public int run(String[] args) throws Exception {
>>
>> // When implementing tool
>> Configuration conf = this.getConf();
>>
>> // Create job
>> Job job = Job.getInstance(conf, "AvgEight");
>> job.setJarByClass(AvgEight.class);
>>
>> // Setup MapReduce job
>> // Do not specify the number of Reducers
>> job.setMapperClass(Map2.class);
>> job.setReducerClass(Reduce2.class);
>>
>> job.setOutputKeyClass(Text.class);
>>
>> //job.setOutputValueClass(IntWritable.class);
>> //job.setOutputValueClass(ArrayWritable.class);
>> job.setOutputValueClass(LongArrayWritable.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);
>>
>> FileInputFormat.addInputPath(job, new Path(args[0]));
>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>
>> job.submit();
>> return 1;
>> }
>>
>>
>>
>> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
>> garlanaganarasimha@huawei.com> wrote:
>>
>>> Hi Ista,
>>> IIUC you just want it to be submitted and not wait till it finish and
>>> print the stats ?
>>> Basically sample code which you might be referring will be creating the
>>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>>> configurations calling
>>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>>
>>> Regards,
>>> Naga
>>> ------------------------------
>>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>>> *Sent:* Wednesday, October 07, 2015 19:58
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Invoking a MapRed job via Runtime class
>>>
>>> Greetings to all,
>>> Is it possible to invoke a MapReduce job from another java class using
>>> the runtime command? If not, what are the alternatives?
>>>
>>> Here is a sample code snippet:
>>>
>>> public static void main(String[] args) {
>>> Runtime rt = Runtime.getRuntime();
>>> try {
>>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>>> } catch (IOException e) {
>>> e.printStackTrace();
>>> }
>>>
>>> Regards,
>>>
>>> Ista
>>>
>>
>>
>
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
What i meant was to have your "mapReduce" job's run code in your
"javaDriver2"'s
main method if your intention is to just run the MR job and not wait for
the result. May be it would be good(helpful) for you to go through the java
docs for Job.
Whats the intention of running this job ? Testing of the MR or the client
driver ?
+ Naga
On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the reply.
> Here is a high level description of what I am trying to achieve (if
> possible):
>
> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
> being implemented, compiled and the respective generated jars are there as
> well.
>
> In the same node, I have a java application running and am trying to start
> one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
>
> The code snippets are provided below.
> Currently, it is not working and I am wondering if I have missed something
> or have incorrect impl.
>
> How would the code changes if the calling application would like to wait
> for the mapreduce job to complete?
>
> Thx
>
> Ista
>
>
> #################################################
>
> The java application code snippet
> ===========================
>
> public class javaDriver2 {
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> System.out.println("caught exception" + e.toString());
> }
> catch (SecurityException se) {
> se.printStackTrace();
> System.out.println("caught exception" + se.toString());
> }
>
> }
>
> ///////////////////////////////////
>
> The Mapreduce job code:
> =====================
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> int res = ToolRunner.run(conf, new AvgEight(), args);
> System.exit(res);
> }
>
> @Override
> public int run(String[] args) throws Exception {
>
> // When implementing tool
> Configuration conf = this.getConf();
>
> // Create job
> Job job = Job.getInstance(conf, "AvgEight");
> job.setJarByClass(AvgEight.class);
>
> // Setup MapReduce job
> // Do not specify the number of Reducers
> job.setMapperClass(Map2.class);
> job.setReducerClass(Reduce2.class);
>
> job.setOutputKeyClass(Text.class);
>
> //job.setOutputValueClass(IntWritable.class);
> //job.setOutputValueClass(ArrayWritable.class);
> job.setOutputValueClass(LongArrayWritable.class);
>
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
>
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> job.submit();
> return 1;
> }
>
>
>
> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>> IIUC you just want it to be submitted and not wait till it finish and
>> print the stats ?
>> Basically sample code which you might be referring will be creating the
>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>> configurations calling
>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>
>> Regards,
>> Naga
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Wednesday, October 07, 2015 19:58
>> *To:* user@hadoop.apache.org
>> *Subject:* Invoking a MapRed job via Runtime class
>>
>> Greetings to all,
>> Is it possible to invoke a MapReduce job from another java class using
>> the runtime command? If not, what are the alternatives?
>>
>> Here is a sample code snippet:
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> }
>>
>> Regards,
>>
>> Ista
>>
>
>
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
What i meant was to have your "mapReduce" job's run code in your
"javaDriver2"'s
main method if your intention is to just run the MR job and not wait for
the result. May be it would be good(helpful) for you to go through the java
docs for Job.
Whats the intention of running this job ? Testing of the MR or the client
driver ?
+ Naga
On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the reply.
> Here is a high level description of what I am trying to achieve (if
> possible):
>
> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
> being implemented, compiled and the respective generated jars are there as
> well.
>
> In the same node, I have a java application running and am trying to start
> one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
>
> The code snippets are provided below.
> Currently, it is not working and I am wondering if I have missed something
> or have incorrect impl.
>
> How would the code changes if the calling application would like to wait
> for the mapreduce job to complete?
>
> Thx
>
> Ista
>
>
> #################################################
>
> The java application code snippet
> ===========================
>
> public class javaDriver2 {
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> System.out.println("caught exception" + e.toString());
> }
> catch (SecurityException se) {
> se.printStackTrace();
> System.out.println("caught exception" + se.toString());
> }
>
> }
>
> ///////////////////////////////////
>
> The Mapreduce job code:
> =====================
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> int res = ToolRunner.run(conf, new AvgEight(), args);
> System.exit(res);
> }
>
> @Override
> public int run(String[] args) throws Exception {
>
> // When implementing tool
> Configuration conf = this.getConf();
>
> // Create job
> Job job = Job.getInstance(conf, "AvgEight");
> job.setJarByClass(AvgEight.class);
>
> // Setup MapReduce job
> // Do not specify the number of Reducers
> job.setMapperClass(Map2.class);
> job.setReducerClass(Reduce2.class);
>
> job.setOutputKeyClass(Text.class);
>
> //job.setOutputValueClass(IntWritable.class);
> //job.setOutputValueClass(ArrayWritable.class);
> job.setOutputValueClass(LongArrayWritable.class);
>
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
>
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> job.submit();
> return 1;
> }
>
>
>
> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>> IIUC you just want it to be submitted and not wait till it finish and
>> print the stats ?
>> Basically sample code which you might be referring will be creating the
>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>> configurations calling
>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>
>> Regards,
>> Naga
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Wednesday, October 07, 2015 19:58
>> *To:* user@hadoop.apache.org
>> *Subject:* Invoking a MapRed job via Runtime class
>>
>> Greetings to all,
>> Is it possible to invoke a MapReduce job from another java class using
>> the runtime command? If not, what are the alternatives?
>>
>> Here is a sample code snippet:
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> }
>>
>> Regards,
>>
>> Ista
>>
>
>
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
What i meant was to have your "mapReduce" job's run code in your
"javaDriver2"'s
main method if your intention is to just run the MR job and not wait for
the result. May be it would be good(helpful) for you to go through the java
docs for Job.
Whats the intention of running this job ? Testing of the MR or the client
driver ?
+ Naga
On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the reply.
> Here is a high level description of what I am trying to achieve (if
> possible):
>
> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
> being implemented, compiled and the respective generated jars are there as
> well.
>
> In the same node, I have a java application running and am trying to start
> one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
>
> The code snippets are provided below.
> Currently, it is not working and I am wondering if I have missed something
> or have incorrect impl.
>
> How would the code changes if the calling application would like to wait
> for the mapreduce job to complete?
>
> Thx
>
> Ista
>
>
> #################################################
>
> The java application code snippet
> ===========================
>
> public class javaDriver2 {
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> System.out.println("caught exception" + e.toString());
> }
> catch (SecurityException se) {
> se.printStackTrace();
> System.out.println("caught exception" + se.toString());
> }
>
> }
>
> ///////////////////////////////////
>
> The Mapreduce job code:
> =====================
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> int res = ToolRunner.run(conf, new AvgEight(), args);
> System.exit(res);
> }
>
> @Override
> public int run(String[] args) throws Exception {
>
> // When implementing tool
> Configuration conf = this.getConf();
>
> // Create job
> Job job = Job.getInstance(conf, "AvgEight");
> job.setJarByClass(AvgEight.class);
>
> // Setup MapReduce job
> // Do not specify the number of Reducers
> job.setMapperClass(Map2.class);
> job.setReducerClass(Reduce2.class);
>
> job.setOutputKeyClass(Text.class);
>
> //job.setOutputValueClass(IntWritable.class);
> //job.setOutputValueClass(ArrayWritable.class);
> job.setOutputValueClass(LongArrayWritable.class);
>
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
>
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> job.submit();
> return 1;
> }
>
>
>
> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>> IIUC you just want it to be submitted and not wait till it finish and
>> print the stats ?
>> Basically sample code which you might be referring will be creating the
>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>> configurations calling
>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>
>> Regards,
>> Naga
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Wednesday, October 07, 2015 19:58
>> *To:* user@hadoop.apache.org
>> *Subject:* Invoking a MapRed job via Runtime class
>>
>> Greetings to all,
>> Is it possible to invoke a MapReduce job from another java class using
>> the runtime command? If not, what are the alternatives?
>>
>> Here is a sample code snippet:
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> }
>>
>> Regards,
>>
>> Ista
>>
>
>
Re: Invoking a MapRed job via Runtime class
Posted by Naganarasimha Garla <na...@gmail.com>.
Hi Ista,
What i meant was to have your "mapReduce" job's run code in your
"javaDriver2"'s
main method if your intention is to just run the MR job and not wait for
the result. May be it would be good(helpful) for you to go through the java
docs for Job.
Whats the intention of running this job ? Testing of the MR or the client
driver ?
+ Naga
On Thu, Oct 8, 2015 at 7:40 PM, Istabrak Abdul-Fatah <if...@gmail.com>
wrote:
> Hi Naga,
> Thx for the reply.
> Here is a high level description of what I am trying to achieve (if
> possible):
>
> On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
> being implemented, compiled and the respective generated jars are there as
> well.
>
> In the same node, I have a java application running and am trying to start
> one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
>
> The code snippets are provided below.
> Currently, it is not working and I am wondering if I have missed something
> or have incorrect impl.
>
> How would the code changes if the calling application would like to wait
> for the mapreduce job to complete?
>
> Thx
>
> Ista
>
>
> #################################################
>
> The java application code snippet
> ===========================
>
> public class javaDriver2 {
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
> -Dtest=\"9999\" /user/yarn/input/sample.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> System.out.println("caught exception" + e.toString());
> }
> catch (SecurityException se) {
> se.printStackTrace();
> System.out.println("caught exception" + se.toString());
> }
>
> }
>
> ///////////////////////////////////
>
> The Mapreduce job code:
> =====================
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> int res = ToolRunner.run(conf, new AvgEight(), args);
> System.exit(res);
> }
>
> @Override
> public int run(String[] args) throws Exception {
>
> // When implementing tool
> Configuration conf = this.getConf();
>
> // Create job
> Job job = Job.getInstance(conf, "AvgEight");
> job.setJarByClass(AvgEight.class);
>
> // Setup MapReduce job
> // Do not specify the number of Reducers
> job.setMapperClass(Map2.class);
> job.setReducerClass(Reduce2.class);
>
> job.setOutputKeyClass(Text.class);
>
> //job.setOutputValueClass(IntWritable.class);
> //job.setOutputValueClass(ArrayWritable.class);
> job.setOutputValueClass(LongArrayWritable.class);
>
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
>
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> job.submit();
> return 1;
> }
>
>
>
> On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Ista,
>> IIUC you just want it to be submitted and not wait till it finish and
>> print the stats ?
>> Basically sample code which you might be referring will be creating the
>> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
>> configurations calling
>> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>>
>> Regards,
>> Naga
>> ------------------------------
>> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
>> *Sent:* Wednesday, October 07, 2015 19:58
>> *To:* user@hadoop.apache.org
>> *Subject:* Invoking a MapRed job via Runtime class
>>
>> Greetings to all,
>> Is it possible to invoke a MapReduce job from another java class using
>> the runtime command? If not, what are the alternatives?
>>
>> Here is a sample code snippet:
>>
>> public static void main(String[] args) {
>> Runtime rt = Runtime.getRuntime();
>> try {
>> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
>> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
>> } catch (IOException e) {
>> e.printStackTrace();
>> }
>>
>> Regards,
>>
>> Ista
>>
>
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the reply.
Here is a high level description of what I am trying to achieve (if
possible):
On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
being implemented, compiled and the respective generated jars are there as
well.
In the same node, I have a java application running and am trying to start
one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
The code snippets are provided below.
Currently, it is not working and I am wondering if I have missed something
or have incorrect impl.
How would the code changes if the calling application would like to wait
for the mapreduce job to complete?
Thx
Ista
#################################################
The java application code snippet
===========================
public class javaDriver2 {
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
-Dtest=\"9999\" /user/yarn/input/sample.csv output");
} catch (IOException e) {
e.printStackTrace();
System.out.println("caught exception" + e.toString());
}
catch (SecurityException se) {
se.printStackTrace();
System.out.println("caught exception" + se.toString());
}
}
///////////////////////////////////
The Mapreduce job code:
=====================
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
int res = ToolRunner.run(conf, new AvgEight(), args);
System.exit(res);
}
@Override
public int run(String[] args) throws Exception {
// When implementing tool
Configuration conf = this.getConf();
// Create job
Job job = Job.getInstance(conf, "AvgEight");
job.setJarByClass(AvgEight.class);
// Setup MapReduce job
// Do not specify the number of Reducers
job.setMapperClass(Map2.class);
job.setReducerClass(Reduce2.class);
job.setOutputKeyClass(Text.class);
//job.setOutputValueClass(IntWritable.class);
//job.setOutputValueClass(ArrayWritable.class);
job.setOutputValueClass(LongArrayWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.submit();
return 1;
}
On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
> IIUC you just want it to be submitted and not wait till it finish and
> print the stats ?
> Basically sample code which you might be referring will be creating the
> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
> configurations calling
> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>
> Regards,
> Naga
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Wednesday, October 07, 2015 19:58
> *To:* user@hadoop.apache.org
> *Subject:* Invoking a MapRed job via Runtime class
>
> Greetings to all,
> Is it possible to invoke a MapReduce job from another java class using the
> runtime command? If not, what are the alternatives?
>
> Here is a sample code snippet:
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> }
>
> Regards,
>
> Ista
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the reply.
Here is a high level description of what I am trying to achieve (if
possible):
On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
being implemented, compiled and the respective generated jars are there as
well.
In the same node, I have a java application running and am trying to start
one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
The code snippets are provided below.
Currently, it is not working and I am wondering if I have missed something
or have incorrect impl.
How would the code changes if the calling application would like to wait
for the mapreduce job to complete?
Thx
Ista
#################################################
The java application code snippet
===========================
public class javaDriver2 {
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
-Dtest=\"9999\" /user/yarn/input/sample.csv output");
} catch (IOException e) {
e.printStackTrace();
System.out.println("caught exception" + e.toString());
}
catch (SecurityException se) {
se.printStackTrace();
System.out.println("caught exception" + se.toString());
}
}
///////////////////////////////////
The Mapreduce job code:
=====================
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
int res = ToolRunner.run(conf, new AvgEight(), args);
System.exit(res);
}
@Override
public int run(String[] args) throws Exception {
// When implementing tool
Configuration conf = this.getConf();
// Create job
Job job = Job.getInstance(conf, "AvgEight");
job.setJarByClass(AvgEight.class);
// Setup MapReduce job
// Do not specify the number of Reducers
job.setMapperClass(Map2.class);
job.setReducerClass(Reduce2.class);
job.setOutputKeyClass(Text.class);
//job.setOutputValueClass(IntWritable.class);
//job.setOutputValueClass(ArrayWritable.class);
job.setOutputValueClass(LongArrayWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.submit();
return 1;
}
On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
> IIUC you just want it to be submitted and not wait till it finish and
> print the stats ?
> Basically sample code which you might be referring will be creating the
> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
> configurations calling
> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>
> Regards,
> Naga
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Wednesday, October 07, 2015 19:58
> *To:* user@hadoop.apache.org
> *Subject:* Invoking a MapRed job via Runtime class
>
> Greetings to all,
> Is it possible to invoke a MapReduce job from another java class using the
> runtime command? If not, what are the alternatives?
>
> Here is a sample code snippet:
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> }
>
> Regards,
>
> Ista
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the reply.
Here is a high level description of what I am trying to achieve (if
possible):
On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
being implemented, compiled and the respective generated jars are there as
well.
In the same node, I have a java application running and am trying to start
one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
The code snippets are provided below.
Currently, it is not working and I am wondering if I have missed something
or have incorrect impl.
How would the code changes if the calling application would like to wait
for the mapreduce job to complete?
Thx
Ista
#################################################
The java application code snippet
===========================
public class javaDriver2 {
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
-Dtest=\"9999\" /user/yarn/input/sample.csv output");
} catch (IOException e) {
e.printStackTrace();
System.out.println("caught exception" + e.toString());
}
catch (SecurityException se) {
se.printStackTrace();
System.out.println("caught exception" + se.toString());
}
}
///////////////////////////////////
The Mapreduce job code:
=====================
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
int res = ToolRunner.run(conf, new AvgEight(), args);
System.exit(res);
}
@Override
public int run(String[] args) throws Exception {
// When implementing tool
Configuration conf = this.getConf();
// Create job
Job job = Job.getInstance(conf, "AvgEight");
job.setJarByClass(AvgEight.class);
// Setup MapReduce job
// Do not specify the number of Reducers
job.setMapperClass(Map2.class);
job.setReducerClass(Reduce2.class);
job.setOutputKeyClass(Text.class);
//job.setOutputValueClass(IntWritable.class);
//job.setOutputValueClass(ArrayWritable.class);
job.setOutputValueClass(LongArrayWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.submit();
return 1;
}
On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
> IIUC you just want it to be submitted and not wait till it finish and
> print the stats ?
> Basically sample code which you might be referring will be creating the
> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
> configurations calling
> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>
> Regards,
> Naga
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Wednesday, October 07, 2015 19:58
> *To:* user@hadoop.apache.org
> *Subject:* Invoking a MapRed job via Runtime class
>
> Greetings to all,
> Is it possible to invoke a MapReduce job from another java class using the
> runtime command? If not, what are the alternatives?
>
> Here is a sample code snippet:
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> }
>
> Regards,
>
> Ista
>
Re: Invoking a MapRed job via Runtime class
Posted by Istabrak Abdul-Fatah <if...@gmail.com>.
Hi Naga,
Thx for the reply.
Here is a high level description of what I am trying to achieve (if
possible):
On node A, I have the Hadoop/Yarn v2.7 running with the mapreduce jobs
being implemented, compiled and the respective generated jars are there as
well.
In the same node, I have a java application running and am trying to start
one of the existing mapreduce jobs (as you called it "to sbumit the job" ).
The code snippets are provided below.
Currently, it is not working and I am wondering if I have missed something
or have incorrect impl.
How would the code changes if the calling application would like to wait
for the mapreduce job to complete?
Thx
Ista
#################################################
The java application code snippet
===========================
public class javaDriver2 {
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p2 = rt.exec("yarn jar /opt/yarn/my_examples/AvgEight.jar
-Dtest=\"9999\" /user/yarn/input/sample.csv output");
} catch (IOException e) {
e.printStackTrace();
System.out.println("caught exception" + e.toString());
}
catch (SecurityException se) {
se.printStackTrace();
System.out.println("caught exception" + se.toString());
}
}
///////////////////////////////////
The Mapreduce job code:
=====================
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
int res = ToolRunner.run(conf, new AvgEight(), args);
System.exit(res);
}
@Override
public int run(String[] args) throws Exception {
// When implementing tool
Configuration conf = this.getConf();
// Create job
Job job = Job.getInstance(conf, "AvgEight");
job.setJarByClass(AvgEight.class);
// Setup MapReduce job
// Do not specify the number of Reducers
job.setMapperClass(Map2.class);
job.setReducerClass(Reduce2.class);
job.setOutputKeyClass(Text.class);
//job.setOutputValueClass(IntWritable.class);
//job.setOutputValueClass(ArrayWritable.class);
job.setOutputValueClass(LongArrayWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.submit();
return 1;
}
On Wed, Oct 7, 2015 at 11:22 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Ista,
> IIUC you just want it to be submitted and not wait till it finish and
> print the stats ?
> Basically sample code which you might be referring will be creating the
> instance of "org.apache.hadoop.mapreduce.Job" and setting with the req
> configurations calling
> *job.waitForCompletion(true);* Instead you can just call *job.submit();*
>
> Regards,
> Naga
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Wednesday, October 07, 2015 19:58
> *To:* user@hadoop.apache.org
> *Subject:* Invoking a MapRed job via Runtime class
>
> Greetings to all,
> Is it possible to invoke a MapReduce job from another java class using the
> runtime command? If not, what are the alternatives?
>
> Here is a sample code snippet:
>
> public static void main(String[] args) {
> Runtime rt = Runtime.getRuntime();
> try {
> Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar
> -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
> } catch (IOException e) {
> e.printStackTrace();
> }
>
> Regards,
>
> Ista
>
RE: Invoking a MapRed job via Runtime class
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
IIUC you just want it to be submitted and not wait till it finish and print the stats ?
Basically sample code which you might be referring will be creating the instance of "org.apache.hadoop.mapreduce.Job" and setting with the req configurations calling
job.waitForCompletion(true); Instead you can just call job.submit();
Regards,
Naga
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Wednesday, October 07, 2015 19:58
To: user@hadoop.apache.org
Subject: Invoking a MapRed job via Runtime class
Greetings to all,
Is it possible to invoke a MapReduce job from another java class using the runtime command? If not, what are the alternatives?
Here is a sample code snippet:
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
} catch (IOException e) {
e.printStackTrace();
}
Regards,
Ista
RE: Invoking a MapRed job via Runtime class
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
IIUC you just want it to be submitted and not wait till it finish and print the stats ?
Basically sample code which you might be referring will be creating the instance of "org.apache.hadoop.mapreduce.Job" and setting with the req configurations calling
job.waitForCompletion(true); Instead you can just call job.submit();
Regards,
Naga
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Wednesday, October 07, 2015 19:58
To: user@hadoop.apache.org
Subject: Invoking a MapRed job via Runtime class
Greetings to all,
Is it possible to invoke a MapReduce job from another java class using the runtime command? If not, what are the alternatives?
Here is a sample code snippet:
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
} catch (IOException e) {
e.printStackTrace();
}
Regards,
Ista
RE: Invoking a MapRed job via Runtime class
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
IIUC you just want it to be submitted and not wait till it finish and print the stats ?
Basically sample code which you might be referring will be creating the instance of "org.apache.hadoop.mapreduce.Job" and setting with the req configurations calling
job.waitForCompletion(true); Instead you can just call job.submit();
Regards,
Naga
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Wednesday, October 07, 2015 19:58
To: user@hadoop.apache.org
Subject: Invoking a MapRed job via Runtime class
Greetings to all,
Is it possible to invoke a MapReduce job from another java class using the runtime command? If not, what are the alternatives?
Here is a sample code snippet:
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
} catch (IOException e) {
e.printStackTrace();
}
Regards,
Ista
RE: Invoking a MapRed job via Runtime class
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Ista,
IIUC you just want it to be submitted and not wait till it finish and print the stats ?
Basically sample code which you might be referring will be creating the instance of "org.apache.hadoop.mapreduce.Job" and setting with the req configurations calling
job.waitForCompletion(true); Instead you can just call job.submit();
Regards,
Naga
________________________________
From: Istabrak Abdul-Fatah [ifatah@gmail.com]
Sent: Wednesday, October 07, 2015 19:58
To: user@hadoop.apache.org
Subject: Invoking a MapRed job via Runtime class
Greetings to all,
Is it possible to invoke a MapReduce job from another java class using the runtime command? If not, what are the alternatives?
Here is a sample code snippet:
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
try {
Process p = rt.exec("yarn jar /opt/yarn/my_examples/AvgSeven.jar -Dtest=\"9999\" /user/yarn/input/samplefile.csv output");
} catch (IOException e) {
e.printStackTrace();
}
Regards,
Ista