[jira] Commented: (HIVE-1107) Generic parallel execution framework for Hive (and Pig, and ...)

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933170#action_12933170 ] 

Arun C Murthy commented on HIVE-1107:
-------------------------------------

+1 on the direction to get Pig and Hive to use common infrastructure for DAG execution.

{quote}
1) A way to serialize and exchange this DAG (e.g. Avro, JSON, XML)
2) A service to execute the DAG and ensure it runs to completion
{quote}

+1

Some more:

# Ability to modify the DAG on the fly, potentially in reaction to execution of parents of the nodes.
# Maybe shared infrastructure for ability to restart the necessary components of the DAG etc.

----

I agree with Russell that Oozie seems too complicated for this task. 

Potentially, as Zheng suggested, a generalized form of JobControl from Map-Reduce could be the answer, it could be something that Pig, Hive and potentially even Oozie can co-opt.




> Generic parallel execution framework for Hive (and Pig, and ...)
> ----------------------------------------------------------------
>
>                 Key: HIVE-1107
>                 URL: https://issues.apache.org/jira/browse/HIVE-1107
>             Project: Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Carl Steinbach
>
> Pig and Hive each have their own libraries for handling plan execution. As we prepare to invest more time improving Hive's plan execution mechanism we should also start to consider ways of building a generic plan execution mechanism that is capable of supporting the needs of Hive and Pig, as well as other Hadoop data flow programming environments. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.