You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Gang Tim Liu (JIRA)" <ji...@apache.org> on 2012/07/11 19:45:33 UTC

[jira] [Created] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job

Gang Tim Liu created HIVE-3251:
----------------------------------

             Summary: Hive doesn't remove scrach directories while killing running MR job
                 Key: HIVE-3251
                 URL: https://issues.apache.org/jira/browse/HIVE-3251
             Project: Hive
          Issue Type: Bug
          Components: Server Infrastructure
            Reporter: Gang Tim Liu
            Assignee: Gang Tim Liu


While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode.

The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job

Posted by "Hudson (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418836#comment-13418836 ] 

Hudson commented on HIVE-3251:
------------------------------

Integrated in Hive-trunk-h0.21 #1553 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1553/])
    HIVE-3251 Hive doesn't remove scrach directories while killing running MR job
(Gang Tim Liu via namit) (Revision 1363424)

     Result = FAILURE
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363424
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

                
> Hive doesn't remove scrach directories while killing running MR job
> -------------------------------------------------------------------
>
>                 Key: HIVE-3251
>                 URL: https://issues.apache.org/jira/browse/HIVE-3251
>             Project: Hive
>          Issue Type: Bug
>          Components: Server Infrastructure
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: HIVE-3151.patch
>
>
> While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode.
> The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Namit Jain resolved HIVE-3251.
------------------------------

      Resolution: Fixed
    Hadoop Flags: Reviewed

Committed. Thanks Tim
                
> Hive doesn't remove scrach directories while killing running MR job
> -------------------------------------------------------------------
>
>                 Key: HIVE-3251
>                 URL: https://issues.apache.org/jira/browse/HIVE-3251
>             Project: Hive
>          Issue Type: Bug
>          Components: Server Infrastructure
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: HIVE-3151.patch
>
>
> While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode.
> The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Namit Jain updated HIVE-3251:
-----------------------------

    Status: Open  (was: Patch Available)

comments
                
> Hive doesn't remove scrach directories while killing running MR job
> -------------------------------------------------------------------
>
>                 Key: HIVE-3251
>                 URL: https://issues.apache.org/jira/browse/HIVE-3251
>             Project: Hive
>          Issue Type: Bug
>          Components: Server Infrastructure
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: HIVE-3151.patch
>
>
> While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode.
> The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Lisk Bucketing DDL Patch

Posted by Gang Tim Liu <ga...@fb.com>.

Yes, Namit has a great summary. thanks

On 7/27/12 9:09 PM, "Namit Jain" <nj...@fb.com> wrote:

>Yes, that patch will become quiet big to be done a single shot.
>
>Moreover, the skew information can be used by a variety of use-cases.
>
>1. List Bucketing
>2. Skew Joins: https://cwiki.apache.org/Hive/skewed-join-optimization.html
>3. Another variant of skew joins:
>https://issues.apache.org/jira/browse/HIVE-3286
>
>So, the skew may not be limited to be used for list bucketing only.
>
>So, it might be simpler to split into DDL and DML support.
>
>DDL will be common to all the use-cases who want to use/store skew
>information. 
>
>Each use-case can implement the DML/Query separately.
>
>
>Thanks,
>-namit
>
>
>On 7/28/12 7:07 AM, "Carl Steinbach" <ca...@cloudera.com> wrote:
>
>>> Since we are close to release the first patch DDL.
>>
>>In a comment on the design doc you said that the first phase would
>>involve
>>implementing this feature for a single-column end-to-end (DML+DDL). Has
>>that plan changed?
>>
>>Thanks.
>>
>>Carl
>>
>>On Wed, Jul 25, 2012 at 12:31 AM, Gang Tim Liu <ga...@fb.com> wrote:
>>
>>> Dear all hive developers,
>>>
>>> Please review the documentation:
>>>
>>> https://cwiki.apache.org/confluence/display/Hive/ListBucketing
>>>
>>> Since we are close to release the first patch DDL.
>>>
>>> We will continue to update the wiki about new information and in the
>>> meanwhile want to collect your feedback.
>>>
>>> Thanks
>>>
>>> Tim
>>>
>>>
>

Re: Lisk Bucketing DDL Patch

Posted by Namit Jain <nj...@fb.com>.

Yes, that patch will become quiet big to be done a single shot.

Moreover, the skew information can be used by a variety of use-cases.

1. List Bucketing
2. Skew Joins: https://cwiki.apache.org/Hive/skewed-join-optimization.html
3. Another variant of skew joins:
https://issues.apache.org/jira/browse/HIVE-3286

So, the skew may not be limited to be used for list bucketing only.

So, it might be simpler to split into DDL and DML support.

DDL will be common to all the use-cases who want to use/store skew
information. 

Each use-case can implement the DML/Query separately.

Thanks,
-namit

On 7/28/12 7:07 AM, "Carl Steinbach" <ca...@cloudera.com> wrote:

>> Since we are close to release the first patch DDL.
>
>In a comment on the design doc you said that the first phase would involve
>implementing this feature for a single-column end-to-end (DML+DDL). Has
>that plan changed?
>
>Thanks.
>
>Carl
>
>On Wed, Jul 25, 2012 at 12:31 AM, Gang Tim Liu <ga...@fb.com> wrote:
>
>> Dear all hive developers,
>>
>> Please review the documentation:
>>
>> https://cwiki.apache.org/confluence/display/Hive/ListBucketing
>>
>> Since we are close to release the first patch DDL.
>>
>> We will continue to update the wiki about new information and in the
>> meanwhile want to collect your feedback.
>>
>> Thanks
>>
>> Tim
>>
>>

Re: Lisk Bucketing DDL Patch

Posted by Carl Steinbach <ca...@cloudera.com>.

> Since we are close to release the first patch DDL.

In a comment on the design doc you said that the first phase would involve
implementing this feature for a single-column end-to-end (DML+DDL). Has
that plan changed?

Thanks.

Carl

On Wed, Jul 25, 2012 at 12:31 AM, Gang Tim Liu <ga...@fb.com> wrote:

> Dear all hive developers,
>
> Please review the documentation:
>
> https://cwiki.apache.org/confluence/display/Hive/ListBucketing
>
> Since we are close to release the first patch DDL.
>
> We will continue to update the wiki about new information and in the
> meanwhile want to collect your feedback.
>
> Thanks
>
> Tim
>
>

Re: Lisk Bucketing DDL Patch

Posted by Namit Jain <nj...@fb.com>.

Note that, we are also planning to use the same syntax for specifying the
syntax for skew for
optimizing joins.

https://cwiki.apache.org/Hive/skewed-join-optimization.html


Thanks,
-namit



On 7/25/12 1:01 PM, "Gang Tim Liu" <ga...@fb.com> wrote:

>Dear all hive developers,
>
>Please review the documentation:
>
>https://cwiki.apache.org/confluence/display/Hive/ListBucketing
>
>Since we are close to release the first patch DDL.
>
>We will continue to update the wiki about new information and in the
>meanwhile want to collect your feedback.
>
>Thanks
>
>Tim
>