You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jeff Hammerbacher (JIRA)" <ji...@apache.org> on 2010/08/15 01:49:16 UTC

[jira] Created: (HIVE-1541) More general dataflow execution backend

More general dataflow execution backend
---------------------------------------

                 Key: HIVE-1541
                 URL: https://issues.apache.org/jira/browse/HIVE-1541
             Project: Hadoop Hive
          Issue Type: New Feature
            Reporter: Jeff Hammerbacher


With the recent open source release of Mesos (http://github.com/mesos/mesos), experimentation at the query execution layer has become more feasible. Inspired by more general-purpose dataflow systems like Volcano, Dryad, and Dremel, it would be interesting to explore a more general-purpose dataflow execution system for Hive queries. One potential backend is the Hyracks project from UCI: http://code.google.com/p/hyracks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1541) More general dataflow execution backend

Posted by "Venkatesh S (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898812#action_12898812 ] 

Venkatesh S commented on HIVE-1541:
-----------------------------------

Oozie should be a good candidate as well.

> More general dataflow execution backend
> ---------------------------------------
>
>                 Key: HIVE-1541
>                 URL: https://issues.apache.org/jira/browse/HIVE-1541
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Jeff Hammerbacher
>
> With the recent open source release of Mesos (http://github.com/mesos/mesos), experimentation at the query execution layer has become more feasible. Inspired by more general-purpose dataflow systems like Volcano, Dryad, and Dremel, it would be interesting to explore a more general-purpose dataflow execution system for Hive queries. One potential backend is the Hyracks project from UCI: http://code.google.com/p/hyracks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1541) More general dataflow execution backend

Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898654#action_12898654 ] 

Jeff Hammerbacher commented on HIVE-1541:
-----------------------------------------

In particular, it would be nice to avoid the startup overhead of Hadoop MapReduce with this backend.

> More general dataflow execution backend
> ---------------------------------------
>
>                 Key: HIVE-1541
>                 URL: https://issues.apache.org/jira/browse/HIVE-1541
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Jeff Hammerbacher
>
> With the recent open source release of Mesos (http://github.com/mesos/mesos), experimentation at the query execution layer has become more feasible. Inspired by more general-purpose dataflow systems like Volcano, Dryad, and Dremel, it would be interesting to explore a more general-purpose dataflow execution system for Hive queries. One potential backend is the Hyracks project from UCI: http://code.google.com/p/hyracks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1541) More general dataflow execution backend

Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899290#action_12899290 ] 

Jeff Hammerbacher commented on HIVE-1541:
-----------------------------------------

Hey Venkatesh,

HIVE-1107 is aimed at getting Hive and Pig to express their sequence of MapReduce jobs as an Oozie workflow. For this JIRA, I meant an entirely different initialization routine and set of physical operators, similar to those used by an MPP relational database or Dremel, Whether Oozie is used to describe the workflow tying together these new physical operators is less of a concern to me.

Thanks,
Jeff

> More general dataflow execution backend
> ---------------------------------------
>
>                 Key: HIVE-1541
>                 URL: https://issues.apache.org/jira/browse/HIVE-1541
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Jeff Hammerbacher
>
> With the recent open source release of Mesos (http://github.com/mesos/mesos), experimentation at the query execution layer has become more feasible. Inspired by more general-purpose dataflow systems like Volcano, Dryad, and Dremel, it would be interesting to explore a more general-purpose dataflow execution system for Hive queries. One potential backend is the Hyracks project from UCI: http://code.google.com/p/hyracks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.