答复: Passing information from one job to the next in a JobControl

[jerry ye <zq...@trafficcast.com.cn>]

Hi Saptarshi:

 

Please refer the following example code, I wish it can help you.

 

    JobConf grepJob = new JobConf(getConf(), Grep.class);

    

    try {

      

      grepJob.setJobName("search");

 

      FileInputFormat.setInputPaths(grepJob, args[0]);

      …………………………………

      FileOutputFormat.setOutputPath(grepJob, tempDir);

      ……………………………….

      JobClient.runJob(grepJob);

 

      JobConf sortJob = new JobConf(Grep.class);

      sortJob.setJobName("sort");

      FileInputFormat.setInputPaths(sortJob, tempDir);

      ……………………………….

      FileOutputFormat.setOutputPath(sortJob, new Path(args[1]));

      ……………………………………..

      JobClient.runJob(sortJob);

 

--Jerry

 

-----邮件原件-----
发件人: Saptarshi Guha [mailto:saptarshi.guha@gmail.com] 
发送时间: 2008年11月11日 12:06
收件人: core-user@hadoop.apache.org
主题: Passing information from one job to the next in a JobControl

 

Hello,

I am using JobControl to run a sequence of jobs(Job_1,Job_2,..Job_n)

on after the other. Each job returns some information

e.g

key1 value1,value2

key2 value1,value2

 

and so on. This can be found in the outdir passed to the jar file.

Is there a way for Job_1 to return some data (which can be passed onto

the Job_2), without my main program having to read the information

from the file in the HDFS?

I could use things like Linda Spaces, however does MapReduce have a

framework for this?

 

Thanks

Saptarshi

-- 

Saptarshi Guha - saptarshi.guha@gmail.com

Re: 答复: Passing information from one job to the next in a JobControl

[Saptarshi Guha <sa...@gmail.com>]

Hi Jerry,
	This actually makes a lot of sense. Hadn't seen it in this light.
	Thank you
	Saptarshi

On Nov 12, 2008, at 3:07 AM, jerry ye wrote:

> Hi Saptarshi:
>
> Please refer the following example code, I wish it can help you.
>
>     JobConf grepJob = new JobConf(getConf(), Grep.class);
>
>     try {
>
>       grepJob.setJobName("search");
>
>       FileInputFormat.setInputPaths(grepJob, args[0]);
>       …………………………………
>       FileOutputFormat.setOutputPath(grepJob, tempDir);
>       ……………………………….
>       JobClient.runJob(grepJob);
>
>       JobConf sortJob = new JobConf(Grep.class);
>       sortJob.setJobName("sort");
>       FileInputFormat.setInputPaths(sortJob, tempDir);
>       ……………………………….
>       FileOutputFormat.setOutputPath(sortJob, new Path(args[1]));
>       ……………………………………..
>       JobClient.runJob(sortJob);
>
> --Jerry
>
> -----邮件原件-----
> 发件人: Saptarshi Guha [mailto:saptarshi.guha@gmail.com]
> 发送时间: 2008年11月11日 12:06
> 收件人: core-user@hadoop.apache.org
> 主题: Passing information from one job to the next in a JobControl
>
> Hello,
> I am using JobControl to run a sequence of jobs(Job_1,Job_2,..Job_n)
> on after the other. Each job returns some information
> e.g
> key1 value1,value2
> key2 value1,value2
>
> and so on. This can be found in the outdir passed to the jar file.
> Is there a way for Job_1 to return some data (which can be passed onto
> the Job_2), without my main program having to read the information
> from the file in the HDFS?
> I could use things like Linda Spaces, however does MapReduce have a
> framework for this?
>
> Thanks
> Saptarshi
> --
> Saptarshi Guha - saptarshi.guha@gmail.com

Saptarshi Guha | saptarshi.guha@gmail.com | http://www.stat.purdue.edu/~sguha
Intel CPUs are not defective, they just act that way.
		-- Henry Spencer