You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "He Yongqiang (JIRA)" <ji...@apache.org> on 2011/09/16 00:46:12 UTC
[jira] [Created] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
move lock retry logic into ZooKeeperHiveLockManager
---------------------------------------------------
Key: HIVE-2450
URL: https://issues.apache.org/jira/browse/HIVE-2450
Project: Hive
Issue Type: Improvement
Reporter: He Yongqiang
Assignee: He Yongqiang
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113857#comment-13113857 ]
jiraposter@reviews.apache.org commented on HIVE-2450:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/#review2061
-----------------------------------------------------------
Could you explain why we want the retry logic down inside of the zookeeper-specific implementation? It seems to me that having it outside is better, since then it doesn't have to be reimplemented in other lock manager implementations as they are added.
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
<https://reviews.apache.org/r/1950/#comment4643>
quorumServers is not used by this method...why has it been added here?
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
<https://reviews.apache.org/r/1950/#comment4642>
But don't we still want to rethrow eventually out of this method? Here you are squelching the exception completely.
- John
On 2011-09-23 21:10:26, Yongqiang He wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1950/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-23 21:10:26)
bq.
bq.
bq. Review request for hive and Ning Zhang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. move lock retry logic into ZooKeeperHiveLockManager
bq.
bq.
bq. This addresses bug HIVE-2450.
bq. https://issues.apache.org/jira/browse/HIVE-2450
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255
bq. trunk/conf/hive-default.xml 1171255
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255
bq.
bq. Diff: https://reviews.apache.org/r/1950/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. will run tests locally
bq.
bq.
bq. Thanks,
bq.
bq. Yongqiang
bq.
bq.
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114999#comment-13114999 ]
John Sichi commented on HIVE-2450:
----------------------------------
+1. Will commit when tests pass.
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-2450:
-------------------------------
Attachment: HIVE-2450.2.patch
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115176#comment-13115176 ]
John Sichi commented on HIVE-2450:
----------------------------------
I got one test failure (lockneg1.q).
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "John Sichi (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi resolved HIVE-2450.
------------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
Hadoop Flags: Reviewed
Committed to trunk. Thanks Yongqiang!
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Fix For: 0.9.0
>
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch, HIVE-2450.4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113769#comment-13113769 ]
jiraposter@reviews.apache.org commented on HIVE-2450:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
-----------------------------------------------------------
(Updated 2011-09-23 21:10:26.466632)
Review request for hive and Ning Zhang.
Changes
-------
update comment in hive-default.xml
Summary
-------
move lock retry logic into ZooKeeperHiveLockManager
This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255
trunk/conf/hive-default.xml 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255
Diff: https://reviews.apache.org/r/1950/diff
Testing
-------
will run tests locally
Thanks,
Yongqiang
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116312#comment-13116312 ]
Hudson commented on HIVE-2450:
------------------------------
Integrated in Hive-trunk-h0.21 #980 (See [https://builds.apache.org/job/Hive-trunk-h0.21/980/])
HIVE-2450. Move retry logic into ZooKeeperHiveLockManager
(Yongqiang He via jvs)
jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176713
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
* /hive/trunk/ql/src/test/queries/clientnegative/lockneg1.q
* /hive/trunk/ql/src/test/queries/clientnegative/lockneg2.q
* /hive/trunk/ql/src/test/queries/clientnegative/lockneg3.q
* /hive/trunk/ql/src/test/queries/clientnegative/lockneg4.q
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Fix For: 0.9.0
>
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch, HIVE-2450.4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "He Yongqiang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-2450:
-------------------------------
Attachment: HIVE-2450.4.patch
a new patch to fix the failing testcase, and added more logging.
One more change is that this patch removes the zookeeper instance recreation from renewZookeeperInstance. It's because once a zookeeper instance is closed, all non-persistent lock will be released. If in future we find out we do need to recreate the zookeeper instance in some cases, we will need to add logics to recreate all non-persistent node after the new zookeeper instance is created.
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch, HIVE-2450.4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-2450:
-------------------------------
Attachment: HIVE-2450.1.patch
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-2450:
-------------------------------
Attachment: HIVE-2450.3.patch
address John's comments. for refactoring the retry logic out, i agree. But i think we can do it later when we do the second lock manager (even we do it now, it will need to change later).
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115933#comment-13115933 ]
jiraposter@reviews.apache.org commented on HIVE-2450:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
-----------------------------------------------------------
(Updated 2011-09-27 21:11:09.644951)
Review request for hive and Ning Zhang.
Changes
-------
fix the failing testcase and more logging
Summary
-------
move lock retry logic into ZooKeeperHiveLockManager
This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255
trunk/conf/hive-default.xml 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255
trunk/ql/src/test/queries/clientnegative/lockneg1.q 1171255
trunk/ql/src/test/queries/clientnegative/lockneg2.q 1171255
trunk/ql/src/test/queries/clientnegative/lockneg3.q 1171255
trunk/ql/src/test/queries/clientnegative/lockneg4.q 1171255
Diff: https://reviews.apache.org/r/1950/diff
Testing
-------
will run tests locally
Thanks,
Yongqiang
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch, HIVE-2450.4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114882#comment-13114882 ]
jiraposter@reviews.apache.org commented on HIVE-2450:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
-----------------------------------------------------------
(Updated 2011-09-26 17:56:19.501365)
Review request for hive and Ning Zhang.
Changes
-------
address comments
Summary
-------
move lock retry logic into ZooKeeperHiveLockManager
This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255
trunk/conf/hive-default.xml 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255
Diff: https://reviews.apache.org/r/1950/diff
Testing
-------
will run tests locally
Thanks,
Yongqiang
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115983#comment-13115983 ]
John Sichi commented on HIVE-2450:
----------------------------------
Retesting now.
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch, HIVE-2450.3.patch, HIVE-2450.4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into
ZooKeeperHiveLockManager
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107636#comment-13107636 ]
jiraposter@reviews.apache.org commented on HIVE-2450:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1950/
-----------------------------------------------------------
Review request for hive and Ning Zhang.
Summary
-------
move lock retry logic into ZooKeeperHiveLockManager
This addresses bug HIVE-2450.
https://issues.apache.org/jira/browse/HIVE-2450
Diffs
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255
trunk/conf/hive-default.xml 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255
trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255
Diff: https://reviews.apache.org/r/1950/diff
Testing
-------
will run tests locally
Thanks,
Yongqiang
> move lock retry logic into ZooKeeperHiveLockManager
> ---------------------------------------------------
>
> Key: HIVE-2450
> URL: https://issues.apache.org/jira/browse/HIVE-2450
> Project: Hive
> Issue Type: Improvement
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: HIVE-2450.1.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira