You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Gregory Chanan (JIRA)" <ji...@apache.org> on 2012/10/02 19:07:07 UTC
[jira] [Created] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Gregory Chanan created HBASE-6920:
-------------------------------------
Summary: On timeout connecting to master, client can get stuck and never make progress
Key: HBASE-6920
URL: https://issues.apache.org/jira/browse/HBASE-6920
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Critical
HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
The issue is here:
{code}
if (tryMaster.isMasterRunning()) {
this.master = tryMaster;
this.masterLock.notifyAll();
break;
}
{code}
If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470885#comment-13470885 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Looks like this is breaking the security build.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6920:
---------------------------------
Attachment: 6920-addendum.txt
Addendum to fix the SecureRpcEngine
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6920:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471048#comment-13471048 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-0.94-security #73 (See [https://builds.apache.org/job/HBase-0.94-security/73/])
HBASE-6920 Addendum2. Remove test code which does not work for the Secure build (Revision 1395093)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gregory Chanan updated HBASE-6920:
----------------------------------
Status: Patch Available (was: Open)
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481780#comment-13481780 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-TRUNK #3472 (See [https://builds.apache.org/job/HBase-TRUNK/3472/])
HBASE-6987 Port HBASE-6920 to trunk (?) (Revision 1401015)
Result = FAILURE
gchanan :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470876#comment-13470876 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-0.94-security #60 (See [https://builds.apache.org/job/HBase-0.94-security/60/])
HBASE-6920 On timeout connecting to master, client can get stuck and never make progress (Revision 1394857)
Result = FAILURE
gchanan :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471374#comment-13471374 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
The security dependency is not at all obvious. I broke the build before that way :)
Fixing the test is probably just a matter of also dealing with SecureRpcEngine (not just WritableRpcEngine).
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470053#comment-13470053 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Yeah, it's the last (for now :) ) open jira for the next RC.
Thanks Gregory.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470770#comment-13470770 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Sorry, looks good, going to check in to 0.94.2 soon.
I need to investigate if this is a problem with 0.96
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467903#comment-13467903 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
I think this is 0.94 only -- at least HBASE-5058 is only in 0.94.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470050#comment-13470050 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Haven't been able to get a good run. Is this holding up the new RC for you? I should be able to tell by tomorrow.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471054#comment-13471054 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-0.94 #512 (See [https://builds.apache.org/job/HBase-0.94/512/])
HBASE-6920 Addendum2. Remove test code which does not work for the Secure build (Revision 1395093)
Result = ABORTED
larsh :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468745#comment-13468745 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Actually, let's just fix the issue you discovered here. +1 on your patch.
We can think about the other change I suggest for 0.94.3. It is time to get 0.94.2 out the door.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467898#comment-13467898 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Also from the explanation I am not quite sure if this a problem in 0.92 as well.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467926#comment-13467926 ]
Hadoop QA commented on HBASE-6920:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12547415/HBASE-6920-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2987//console
This message is automatically generated.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490395#comment-13490395 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/])
HBASE-6920 Addendum2. Remove test code which does not work for the Secure build (Revision 1395093)
HBASE-6920 Addendum - fix SecureRpcEngine (Revision 1394908)
HBASE-6920 On timeout connecting to master, client can get stuck and never make progress (Revision 1394857)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
larsh :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
gchanan :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470892#comment-13470892 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-0.94-security #62 (See [https://builds.apache.org/job/HBase-0.94-security/62/])
HBASE-6920 Addendum - fix SecureRpcEngine (Revision 1394908)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6920:
---------------------------------
Fix Version/s: 0.94.2
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gregory Chanan updated HBASE-6920:
----------------------------------
Attachment: HBASE-6920-v2.patch
v2, which fixes the javadoc issue (actually changes the return value of the code).
I left the LOG as INFO, because that's what it was before; let me know if you disagree.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468835#comment-13468835 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Sounds good. I'm doing some cluster testing today, I'll commit if all looks good.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470784#comment-13470784 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Thanks Gregory! ... Will need to resolve temporarily to get correct release notes for 0.94.2 will reopen soon after (or should 0.96 receive a separate issue now?)
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467889#comment-13467889 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
I don't have any opinion on the DEBUG level; if you think that's better I'll change it.
Good catch on the return value, the javadoc is incorrect.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467873#comment-13467873 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Some other points:
* I have not tried the client app that failed in the description with a patched client yet. I will do that soon.
* This seems like a pretty serious issue (I literally could not get my app to work with a 94 client), though it's not clear how often users will see it, since it only happens on errors. It's also not new with 0.94.2 so I don't know if it needs to sink the RC. Any opinions?
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470888#comment-13470888 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Committed addendum to 0.94
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467891#comment-13467891 ]
Ted Yu commented on HBASE-6920:
-------------------------------
{code}
+ /** Construct a client-side proxy object, specifying an InvocationHandler for testing purposes */
+ VersionedProtocol getProxy(Class<? extends VersionedProtocol> protocol,
{code}
Name the above method getProxyForTesting ?
{code}
+ * RpcEngine that random throws a SocketTimeoutEngine for testing.
{code}
'random' -> 'randomly'
Please correct the spelling for SocketTimeoutEngine
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469800#comment-13469800 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Did you find any more weirdness during your testing?
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470894#comment-13470894 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-0.94 #511 (See [https://builds.apache.org/job/HBase-0.94/511/])
HBASE-6920 Addendum - fix SecureRpcEngine (Revision 1394908)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468215#comment-13468215 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Lars, good question.
There are different failure cases. Basically, does reconnecting to the master require another call to HBaseRPC.getProxy (e.g. yes if the master went down). This only fixes the case that doesn't require another call - e.g. the single RPC call just timed out, but you didn't lose the connection the master. If the master went down and a different master took over then your client is stuck (AFAIK, I'd need to actually test it). This is the same as before HBASE-5058 went in. I kept it this way because of the comment in HBASE-5058:
bq. The effect is that the current behavior is not changed. I.e. for a managed connection we try only once
that is, I didn't want to change the behavior.
What you suggested seems reasonable -- I can try that out. Should it be a different patch?
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468084#comment-13468084 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Hmm... Not offhand.
Looking at the patch again... Will this actually fix the problem? The problem is that masterChecked is true when we successfully connected to a master once. If true, we do not try again.
So we were successful once, we will never enter the retry loop (since masterChecked == true)... Right?
I wonder whether another approach would be to set masterChecked = false at the same spot where master is set to null, but only do that if master was not-null before.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467885#comment-13467885 ]
Ted Yu commented on HBASE-6920:
-------------------------------
Great finding, Gregory.
{code}
+ * @return true if should retry
+ */
+ private boolean logGetMasterAttemptFailure(int tries, Exception e) {
{code}
Please rename the method to reflect the fact that it returns whether we should retry.
{code}
+ LOG.info("getMaster attempt " + tries + " of " + numRetries +
+ " failed; retrying after sleep of " +
+ ConnectionUtils.getPauseTime(this.pause, tries), e);
{code}
Should the above log be at DEBUG level ?
Comparing your code and the original, it seems that the return value of true actually means that we shouldn't retry.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467896#comment-13467896 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Patch looks good.
I don't quite grok the changes in HBaseRPC, RpcEngine, and WritableRpcEngine. Are those needed for this to work?
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468227#comment-13468227 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
I'm fine either way.
If the connection is managed, there is outside logic that will recreate the connection when there was a failure. Maybe that will take care of a restarted master scenario.
For an unmanaged connection neither of these problems exist, because we'll just retry.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470779#comment-13470779 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Thanks for the reviews, Ted and Lars. Committed to 0.94.
Not closing until I investigate trunk.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467874#comment-13467874 ]
Hadoop QA commented on HBASE-6920:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12547405/HBASE-6920.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2986//console
This message is automatically generated.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471367#comment-13471367 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Sounds good to me Lars. Should have tested the security stuff.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467933#comment-13467933 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
+1 on patch after addressing Ted's comments ... (although the RPC changes look a bit scary).
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481963#comment-13481963 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #231 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/231/])
HBASE-6987 Port HBASE-6920 to trunk (?) (Revision 1401015)
Result = FAILURE
gchanan :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467905#comment-13467905 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
The changes in HBaseRPC, RpcEngine, and WritableRpcEngine are just for the test. I wanted to simulate throwing a SocketTimeoutException on a proxy call, but Mockito can't mock a proxy object as far as I can tell.
So all those changes are just so I can setup my own RPC Invoker to run, which just randomly throws SocketTimeoutExceptions.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470859#comment-13470859 ]
Hudson commented on HBASE-6920:
-------------------------------
Integrated in HBase-0.94 #509 (See [https://builds.apache.org/job/HBase-0.94/509/])
HBASE-6920 On timeout connecting to master, client can get stuck and never make progress (Revision 1394857)
Result = FAILURE
gchanan :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ipc/RandomTimeoutRpcEngine.java
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471040#comment-13471040 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
Now TestClientTimeouts always fails in the Security build.
So, here's what I am going to do: I'll revert the test part of this change, then we can file a new ticket for the 0.96 investigation and the testing part.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: 6920-addendum.txt, HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467937#comment-13467937 ]
Gregory Chanan commented on HBASE-6920:
---------------------------------------
Lars -- you have an idea for how to do the test without making RPC changes? I'd prefer that, but couldn't think of a way.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470760#comment-13470760 ]
Lars Hofhansl commented on HBASE-6920:
--------------------------------------
<ping>
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6920.patch, HBASE-6920-v2.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master,
client can get stuck and never make progress
Posted by "Gregory Chanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gregory Chanan updated HBASE-6920:
----------------------------------
Attachment: HBASE-6920.patch
* Attached HBASE-6920.patch *
Here is a patch for this issue and a test. The test is more complicated than I would like; I needed to introduce some functions / change some access controls to be able to use my own RPC Invoker. Any suggestions would be welcome.
> On timeout connecting to master, client can get stuck and never make progress
> -----------------------------------------------------------------------------
>
> Key: HBASE-6920
> URL: https://issues.apache.org/jira/browse/HBASE-6920
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Reporter: Gregory Chanan
> Assignee: Gregory Chanan
> Priority: Critical
> Attachments: HBASE-6920.patch
>
>
> HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized.
> The issue is here:
> {code}
> if (tryMaster.isMasterRunning()) {
> this.master = tryMaster;
> this.masterLock.notifyAll();
> break;
> }
> {code}
> If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application.
> But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect.
> I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira