You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jean Xu (JIRA)" <ji...@apache.org> on 2012/09/18 23:41:07 UTC
[jira] [Created] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Jean Xu created HIVE-3484:
-----------------------------
Summary: RetryingRawStore logic needs to be significantly reworked to support retries within transactions
Key: HIVE-3484
URL: https://issues.apache.org/jira/browse/HIVE-3484
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Jean Xu
Assignee: Jean Xu
Priority: Minor
The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460965#comment-13460965 ]
Ashutosh Chauhan commented on HIVE-3484:
----------------------------------------
I haven't looked at the patch in detail. But can you please explain why are you moving retrial logic to HMSHandler from RawStore. How is it going to help retrial within a transaction? Is this patch part of series of patches or this patch alone is sufficient? Just trying to understand the goals here.
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Jean Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean Xu updated HIVE-3484:
--------------------------
Status: Patch Available (was: Open)
https://reviews.facebook.net/D5535
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467578#comment-13467578 ]
Hudson commented on HIVE-3484:
------------------------------
Integrated in Hive-trunk-h0.21 #1714 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1714/])
HIVE-3484. RetryingRawStore logic needs to be significantly reworked to support retries within transactions (Jean Xu via kevinwilfong) (Revision 1392491)
Result = FAILURE
kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392491
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IHMSHandler.java
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreInit.java
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingRawStore.java
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
> Fix For: 0.10.0
>
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460680#comment-13460680 ]
Kevin Wilfong commented on HIVE-3484:
-------------------------------------
Comments in Phabricator
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
> Priority: Minor
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460998#comment-13460998 ]
Kevin Wilfong commented on HIVE-3484:
-------------------------------------
We have seen in the past errors where the connection to MySQL will be lost and hence the transaction is broken, but the RetryingRawStore happily retries the command to the RawStore, in which the connection is reestablished and the command succeeds, and continues with the rest of the commands. These commands are performed outside a transaction and ultimately the command fails because the number of transactions the metastore tries to close at the end does not match the number of open transactions.
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465808#comment-13465808 ]
Kevin Wilfong commented on HIVE-3484:
-------------------------------------
Looks good for the most part, a couple small comments on the diff.
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465917#comment-13465917 ]
Kevin Wilfong commented on HIVE-3484:
-------------------------------------
+1 This looks good to me.
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461001#comment-13461001 ]
Kevin Wilfong commented on HIVE-3484:
-------------------------------------
There are classes of errors that would be suitable to retry at the raw store level (in which the transaction is maintained open) and classes of errors that should be retried at the Hander level (in which the transaction is rolled back/lost).
It's up to Jean how much she wants to accomplish in this JIRA, but when I suggested she work on this task, I was hoping for at least an implementation of something which could retry outside of transactions, determining those classes of errors that should be handled in either the raw store or the handler could be done here or as part of another JIRA.
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3484:
--------------------------------
Status: Open (was: Patch Available)
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
> Priority: Minor
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3484:
--------------------------------
Priority: Major (was: Minor)
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3484:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.10.0
Status: Resolved (was: Patch Available)
Committed, thanks Jean.
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
> Fix For: 0.10.0
>
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3484:
--------------------------------
Status: Open (was: Patch Available)
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Jean Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean Xu updated HIVE-3484:
--------------------------
Status: Patch Available (was: Open)
https://reviews.facebook.net/D5535
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
> Priority: Minor
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3484) RetryingRawStore logic needs to be
significantly reworked to support retries within transactions
Posted by "Jean Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean Xu updated HIVE-3484:
--------------------------
Status: Patch Available (was: Open)
https://reviews.facebook.net/D5535
> RetryingRawStore logic needs to be significantly reworked to support retries within transactions
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-3484
> URL: https://issues.apache.org/jira/browse/HIVE-3484
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Jean Xu
> Assignee: Jean Xu
>
> The logic for retrying has been moved from RetryingRawStore to RetryingHMSHandler.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira