how pig generate jobs

[Gang Luo <lg...@yahoo.com.cn>]

Hi All,
so, after Pig generates the DAG with each node representing
a MepReduce job, it will sort it in topological order. All the
MapReduce jobs, in the form of a sequence, will be packed into a jar
file and submitted to Hadoop for execution. In order to solve the
dependencies among jobs, there will be wait() inserted between two
depend jobs. The question is, if two job are peer without any
dependency and they are next to each other in the topological sequence,
is there also a wait() between them?

For example, job1 is the
father and has two children job2 and job3. The topological sequence is
job1--job2--job3, even job3 doesn't depend on job2. Can job3 run before
job 2 finishes? Comparisons with HIve on this are welcome.

Thanks,
-Gang


      

Re: how pig generate jobs

[Gang Luo <lg...@yahoo.com.cn>]

Thanks Richard. 
So, Pig will generate more than on jar files and submit them in order. In that case, there will be two jars(batches). The first one contains only job1 and the second contains job2 and job3. If so, the wait() is actually controlled by pig, meaning only after the previous batch/jar finishes will Pig submit the next batch/jar. Please correct me if there is something wrong. 

Thanks,
-Gang




----- 原始邮件 ----
发件人： Richard Ding <rd...@yahoo-inc.com>
收件人： pig-user@hadoop.apache.org
发送日期： 2010/4/19 (周一) 1:14:05 下午
主   题： RE: how pig generate jobs

Actually in this case, Pig sends both job2 and job3 for execution
through Hadoop JobControl at the same time. 

Pig sends jobs for execution in batches: first all the "roots" of the
DAG (i.e., all jobs that don't depend on other jobs in the DAG), after
that Pig removes these jobs from the DAG and sends in the next "root"
batch of the remaining DAG, and so on.  

Thanks,
-Richard
-----Original Message-----
From: Gang Luo [mailto:lgpublic@yahoo.com.cn] 
Sent: Saturday, April 17, 2010 2:31 PM
To: pig-user@hadoop.apache.org
Subject: how pig generate jobs

Hi All,
so, after Pig generates the DAG with each node representing
a MepReduce job, it will sort it in topological order. All the
MapReduce jobs, in the form of a sequence, will be packed into a jar
file and submitted to Hadoop for execution. In order to solve the
dependencies among jobs, there will be wait() inserted between two
depend jobs. The question is, if two job are peer without any
dependency and they are next to each other in the topological sequence,
is there also a wait() between them?

For example, job1 is the
father and has two children job2 and job3. The topological sequence is
job1--job2--job3, even job3 doesn't depend on job2. Can job3 run before
job 2 finishes? Comparisons with HIve on this are welcome.

Thanks,
-Gang


      

RE: how pig generate jobs

[Richard Ding <rd...@yahoo-inc.com>]

Actually in this case, Pig sends both job2 and job3 for execution
through Hadoop JobControl at the same time. 

Pig sends jobs for execution in batches: first all the "roots" of the
DAG (i.e., all jobs that don't depend on other jobs in the DAG), after
that Pig removes these jobs from the DAG and sends in the next "root"
batch of the remaining DAG, and so on.  

Thanks,
-Richard
-----Original Message-----
From: Gang Luo [mailto:lgpublic@yahoo.com.cn] 
Sent: Saturday, April 17, 2010 2:31 PM
To: pig-user@hadoop.apache.org
Subject: how pig generate jobs

Hi All,
so, after Pig generates the DAG with each node representing
a MepReduce job, it will sort it in topological order. All the
MapReduce jobs, in the form of a sequence, will be packed into a jar
file and submitted to Hadoop for execution. In order to solve the
dependencies among jobs, there will be wait() inserted between two
depend jobs. The question is, if two job are peer without any
dependency and they are next to each other in the topological sequence,
is there also a wait() between them?

For example, job1 is the
father and has two children job2 and job3. The topological sequence is
job1--job2--job3, even job3 doesn't depend on job2. Can job3 run before
job 2 finishes? Comparisons with HIve on this are welcome.

Thanks,
-Gang