You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Samir Ahmic (JIRA)" <ji...@apache.org> on 2015/09/14 19:45:46 UTC
[jira] [Updated] (HBASE-14431) AsyncRpcClient#removeConnection()
never removes connection from connections pool if server fails
[ https://issues.apache.org/jira/browse/HBASE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Samir Ahmic updated HBASE-14431:
--------------------------------
Description:
I was playing with master branch in distributed mode (3 rs + master + backup_master) and notice strange behavior when i was testing this sequence of events on single rs: /kill/start/run_balancer while client was writing data to cluster (LoadTestTool).
I have notice that LTT fails with following:
{code}
2015-09-09 11:05:58,364 INFO [main] client.AsyncProcess: #2, waiting for some tasks to finish. Expected max=0, tasksInProgress=35
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: BindException: 1 time,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211)
{code}
After some digging and adding some more logging in code i have notice that following condition in {code}AsyncRpcClient.removeConnection(AsyncRpcChannel connection) {code} is never true:
{code}
if (connectionInPool == connection) {
{code}
causing that {code}AsyncRpcChannel{code} connection is never removed from {code}connections{code} pool in case rs fails.
After changing this condition to:
{code}
if (connectionInPool.address.equals(connection.address)) {
{code}
issue was resolved and client was removing failed server from connections pool.
I will attach patch after running some more tests.
was:
I was playing with master branch in distributed mode (3 rs + master + backup_master) and notice strange behavior when i was testing this sequence of events on single rs: /kill/start/run_balancer while client was writing data to cluster (LoadTestTool).
I have notice that LTT fails with following:
{code}
2015-09-09 11:05:58,364 INFO [main] client.AsyncProcess: #2, waiting for some tasks to finish. Expected max=0, tasksInProgress=35
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: BindException: 1 time,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:186)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:951)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:571)
at org.apache.hadoop.hbase.util.SimpleClient.main(SimpleClient.java:62)
{code}
After some digging and adding some more logging in code i have notice that following condition in {code}AsyncRpcClient.removeConnection(AsyncRpcChannel connection) {code} is never true:
{code}
if (connectionInPool == connection) {
{code}
causing that {code}AsyncRpcChannel{code} connection is never removed from {code}connections{code} pool in case rs fails.
After changing this condition to:
{code}
if (connectionInPool.address.equals(connection.address)) {
{code}
issue was resolved and client was removing failed server from connections pool.
I will attach patch after running some more tests.
> AsyncRpcClient#removeConnection() never removes connection from connections pool if server fails
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-14431
> URL: https://issues.apache.org/jira/browse/HBASE-14431
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
> Affects Versions: 2.0.0, 1.0.2, 1.1.2
> Reporter: Samir Ahmic
> Assignee: Samir Ahmic
> Priority: Critical
>
> I was playing with master branch in distributed mode (3 rs + master + backup_master) and notice strange behavior when i was testing this sequence of events on single rs: /kill/start/run_balancer while client was writing data to cluster (LoadTestTool).
> I have notice that LTT fails with following:
> {code}
> 2015-09-09 11:05:58,364 INFO [main] client.AsyncProcess: #2, waiting for some tasks to finish. Expected max=0, tasksInProgress=35
> Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: BindException: 1 time,
> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211)
> {code}
> After some digging and adding some more logging in code i have notice that following condition in {code}AsyncRpcClient.removeConnection(AsyncRpcChannel connection) {code} is never true:
> {code}
> if (connectionInPool == connection) {
> {code}
> causing that {code}AsyncRpcChannel{code} connection is never removed from {code}connections{code} pool in case rs fails.
> After changing this condition to:
> {code}
> if (connectionInPool.address.equals(connection.address)) {
> {code}
> issue was resolved and client was removing failed server from connections pool.
> I will attach patch after running some more tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)