You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2010/11/17 02:57:13 UTC
[jira] Created: (PIG-1734) Pig needs a more efficient DAG execution
Pig needs a more efficient DAG execution
----------------------------------------
Key: PIG-1734
URL: https://issues.apache.org/jira/browse/PIG-1734
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
The current code uses Hadoop's Job control to execute one stage at a time. The first stage includes all jobs with no dependencies, the second stage jobs that depend only on jobs completed in the first stage, the third stage contains the jobs that depend on jobs from stage 1 and 2, etc.
The problem with this simplistic approach is that each next stages only starts when the previous stage is over which means means that some branches of the DAG are unnecessarily blocked.
We would need to do our own DAG management to solve this issue which would be a pretty significant undertaking. Something we should look at in the future.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1734) Pig needs a more efficient DAG
execution
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933171#action_12933171 ]
Jeff Hammerbacher commented on PIG-1734:
----------------------------------------
bq. However note that the use of the workflow execution engine should not be enforced but should be optional.
Certainly agree that we shouldn't disrupt existing users.
> Pig needs a more efficient DAG execution
> ----------------------------------------
>
> Key: PIG-1734
> URL: https://issues.apache.org/jira/browse/PIG-1734
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
>
> The current code uses Hadoop's Job control to execute one stage at a time. The first stage includes all jobs with no dependencies, the second stage jobs that depend only on jobs completed in the first stage, the third stage contains the jobs that depend on jobs from stage 1 and 2, etc.
> The problem with this simplistic approach is that each next stages only starts when the previous stage is over which means means that some branches of the DAG are unnecessarily blocked.
> We would need to do our own DAG management to solve this issue which would be a pretty significant undertaking. Something we should look at in the future.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1734) Pig needs a more efficient DAG execution
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1734:
--------------------------------
Fix Version/s: 0.10
> Pig needs a more efficient DAG execution
> ----------------------------------------
>
> Key: PIG-1734
> URL: https://issues.apache.org/jira/browse/PIG-1734
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Fix For: 0.10
>
>
> The current code uses Hadoop's Job control to execute one stage at a time. The first stage includes all jobs with no dependencies, the second stage jobs that depend only on jobs completed in the first stage, the third stage contains the jobs that depend on jobs from stage 1 and 2, etc.
> The problem with this simplistic approach is that each next stages only starts when the previous stage is over which means means that some branches of the DAG are unnecessarily blocked.
> We would need to do our own DAG management to solve this issue which would be a pretty significant undertaking. Something we should look at in the future.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (PIG-1734) Pig needs a more efficient DAG
execution
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933181#action_12933181 ]
Arun C Murthy commented on PIG-1734:
------------------------------------
+1 on a more efficient DAG execution engine, and for exploring common infrastructure between Pig and Hive.
It's hard to keep this in sync with HIVE-549, but I'll try.
Jeff and I came up with some requirements:
# A way to serialize and exchange this DAG (e.g. Avro, JSON, XML)
# A service to execute the DAG and ensure it runs to completion
# Ability to modify the DAG on the fly, potentially in reaction to execution of parents of the nodes.
# Maybe shared infrastructure for ability to restart the necessary components of the DAG etc.
Given the above, I do not believe Oozie is a right answer, I'd agree with Zheng (https://issues.apache.org/jira/browse/HIVE-1107?focusedCommentId=12805351&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12805351) that enhancing JobControl would probably be the sweet spot - this way Pig, Hive and even Oozie can use it.
Russel Jurney has similar views against using Oozie too: https://issues.apache.org/jira/browse/HIVE-1107?focusedCommentId=12888870&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12888870
> Pig needs a more efficient DAG execution
> ----------------------------------------
>
> Key: PIG-1734
> URL: https://issues.apache.org/jira/browse/PIG-1734
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
>
> The current code uses Hadoop's Job control to execute one stage at a time. The first stage includes all jobs with no dependencies, the second stage jobs that depend only on jobs completed in the first stage, the third stage contains the jobs that depend on jobs from stage 1 and 2, etc.
> The problem with this simplistic approach is that each next stages only starts when the previous stage is over which means means that some branches of the DAG are unnecessarily blocked.
> We would need to do our own DAG management to solve this issue which would be a pretty significant undertaking. Something we should look at in the future.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (PIG-1734) Pig needs a more efficient DAG
execution
Posted by "Olga Natkovich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1734:
--------------------------------
Fix Version/s: (was: 0.10)
> Pig needs a more efficient DAG execution
> ----------------------------------------
>
> Key: PIG-1734
> URL: https://issues.apache.org/jira/browse/PIG-1734
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
>
> The current code uses Hadoop's Job control to execute one stage at a time. The first stage includes all jobs with no dependencies, the second stage jobs that depend only on jobs completed in the first stage, the third stage contains the jobs that depend on jobs from stage 1 and 2, etc.
> The problem with this simplistic approach is that each next stages only starts when the previous stage is over which means means that some branches of the DAG are unnecessarily blocked.
> We would need to do our own DAG management to solve this issue which would be a pretty significant undertaking. Something we should look at in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (PIG-1734) Pig needs a more efficient DAG
execution
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933104#action_12933104 ]
Santhosh Srinivasan commented on PIG-1734:
------------------------------------------
+1 on the proposal to move to an external workflow execution engine. However note that the use of the workflow execution engine should not be enforced but should be optional.
> Pig needs a more efficient DAG execution
> ----------------------------------------
>
> Key: PIG-1734
> URL: https://issues.apache.org/jira/browse/PIG-1734
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
>
> The current code uses Hadoop's Job control to execute one stage at a time. The first stage includes all jobs with no dependencies, the second stage jobs that depend only on jobs completed in the first stage, the third stage contains the jobs that depend on jobs from stage 1 and 2, etc.
> The problem with this simplistic approach is that each next stages only starts when the previous stage is over which means means that some branches of the DAG are unnecessarily blocked.
> We would need to do our own DAG management to solve this issue which would be a pretty significant undertaking. Something we should look at in the future.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.