You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2012/09/19 21:04:07 UTC

[jira] [Created] (HIVE-3489) Use OutputCommitter to get rid of requirement of client being alive for long running queries

Ashutosh Chauhan created HIVE-3489:
--------------------------------------

             Summary: Use OutputCommitter to get rid of requirement of client being alive for long running queries 
                 Key: HIVE-3489
                 URL: https://issues.apache.org/jira/browse/HIVE-3489
             Project: Hive
          Issue Type: Bug
            Reporter: Ashutosh Chauhan


There is a bunch of work (depending on query) which hive client does once query finishes. This has multiple effects which would be nice to avoid. Few of these work items include:
* Moving temp files / dirs around to their final destination.
* Creating empty files if required.
* Updating metastore with metadata.
* More ?  

If we start using {{OutputCommitter}} then clients can fire and forget, they won't need to be around for all the time query is executing. Plus, since processing is going to happen in cluster, this further makes client lightweight.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3489) Use OutputCommitter to get rid of requirement of client being alive for long running queries

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459321#comment-13459321 ] 

Namit Jain commented on HIVE-3489:
----------------------------------

Also, the post execution hooks need to be run.
                
> Use OutputCommitter to get rid of requirement of client being alive for long running queries 
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3489
>                 URL: https://issues.apache.org/jira/browse/HIVE-3489
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ashutosh Chauhan
>
> There is a bunch of work (depending on query) which hive client does once query finishes. This has multiple effects which would be nice to avoid. Few of these work items include:
> * Moving temp files / dirs around to their final destination.
> * Creating empty files if required.
> * Updating metastore with metadata.
> * More ?  
> If we start using {{OutputCommitter}} then clients can fire and forget, they won't need to be around for all the time query is executing. Plus, since processing is going to happen in cluster, this further makes client lightweight.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3489) Use OutputCommitter to get rid of requirement of client being alive for long running queries

Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459007#comment-13459007 ] 

Edward Capriolo commented on HIVE-3489:
---------------------------------------

I think this is not possible. Right now the client HAS to stay connected. They are the Driver. When a task ends the client launches the dependent tasks. If the job in in its last phase the client could theoretically disconnect, but that is a rare case. HiveServer can be used to disconnect the client from the query. 
                
> Use OutputCommitter to get rid of requirement of client being alive for long running queries 
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3489
>                 URL: https://issues.apache.org/jira/browse/HIVE-3489
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ashutosh Chauhan
>
> There is a bunch of work (depending on query) which hive client does once query finishes. This has multiple effects which would be nice to avoid. Few of these work items include:
> * Moving temp files / dirs around to their final destination.
> * Creating empty files if required.
> * Updating metastore with metadata.
> * More ?  
> If we start using {{OutputCommitter}} then clients can fire and forget, they won't need to be around for all the time query is executing. Plus, since processing is going to happen in cluster, this further makes client lightweight.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira