How to run or submit MapReduce Job to Hadoop in my own program?

["Felix.徐" <yg...@gmail.com>]

Hi,all..How can I run a MR job though my own program instead of using
console to submit a job to a real Hadoop env?
I write code like this, this program works fine but i don't think it ran in
my Hadoop env,since nothing was produced in hadoop logs folder.
public int run(String[] args) throws Exception {

                Configuration config = new Configuration();
                config.set("fs.default.name", "hdfs://localhost:54310");
                config.set("mapred.job.tracker","localhost:54311");
                JobConf conf = new JobConf(config,CustomMR.class);
                conf.setJobName("custom aggregation");
                conf.setMapOutputKeyClass(Text.class);
                conf.setMapOutputValueClass(IntWritable.class);

            conf.setOutputFormat(HiveIgnoreKeyTextOutputFormat.class);

            conf.setMapperClass(MapClass.class);

            conf.setReducerClass(Reduce.class);
            TextInputFormat.setInputPaths(conf,args[0]);
            HiveIgnoreKeyTextOutputFormat.setOutputPath(conf, new
Path(args[1]));

            JobClient.runJob(conf);
                //conf
                return 0;
        }
public static void main(String[] args) throws Exception {
            args = new String[]{"/user/hive/warehouse/ptest","/testout"};
            int res = ToolRunner.run(new Configuration(), new CustomMR(),
args);
            System.exit(res);
}
Could some one give me an example?Thanks!

RE: How to run or submit MapReduce Job to Hadoop in my own program?

[pr...@nokia.com]

Hi there,
I think you got the run(String[] args) method right but in the main method you are not calling your run method but ToolRunner.run. You need to invoke your method in order to point to localhost:54310 otherwise it will read those properties from the default hadoop conf.

Praveen
________________________________
From: ext Felix.徐 [ygnhzeus@gmail.com]
Sent: Tuesday, May 17, 2011 6:47 AM
To: mapreduce-user@hadoop.apache.org
Subject: How to run or submit MapReduce Job to Hadoop in my own program?

Hi,all..How can I run a MR job though my own program instead of using console to submit a job to a real Hadoop env?
I write code like this, this program works fine but i don't think it ran in my Hadoop env,since nothing was produced in hadoop logs folder.
public int run(String[] args) throws Exception {

                Configuration config = new Configuration();
                config.set("fs.default.name<http://fs.default.name>", "hdfs://localhost:54310");
                config.set("mapred.job.tracker","localhost:54311");
                JobConf conf = new JobConf(config,CustomMR.class);
                conf.setJobName("custom aggregation");
                conf.setMapOutputKeyClass(Text.class);
                conf.setMapOutputValueClass(IntWritable.class);

            conf.setOutputFormat(HiveIgnoreKeyTextOutputFormat.class);

            conf.setMapperClass(MapClass.class);

            conf.setReducerClass(Reduce.class);
            TextInputFormat.setInputPaths(conf,args[0]);
            HiveIgnoreKeyTextOutputFormat.setOutputPath(conf, new Path(args[1]));

            JobClient.runJob(conf);
                //conf
                return 0;
        }
public static void main(String[] args) throws Exception {
            args = new String[]{"/user/hive/warehouse/ptest","/testout"};
            int res = ToolRunner.run(new Configuration(), new CustomMR(), args);
            System.exit(res);
}
Could some one give me an example?Thanks!