[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections

["Adrian Muraru (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491169#comment-13491169 ] 

Adrian Muraru commented on HBASE-6956:
--------------------------------------

I'm seeing a similar stack trace when using HTablePool:
{code}
java.lang.RuntimeException: java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5c429be9 closed
        at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38) ~[hbase-0.92.2-26.jar:0.92.2-26]
        at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268) ~[hbase-0.92.2-26.jar:0.92.2-26]
        at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198) ~[hbase-0.92.2-26.jar:0.92.2-26]
        at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173) ~[hbase-0.92.2-26.jar:0.92.2-26]
{code}

The issue is not with a closed HTable instance returned back to the pool.
Instead, following my investigation it is caused by the shared *HConnection* instance being closed. 
Once you get into this state - the HTablePool#getTable() will always fail and never recover.

A possible fix is to drop the HConnection used by HTablePool#getTable() in case of an IOException. 
Not trivial though as it is created based on a private Configuration instance in the pool :)
{code}
this.connection = HConnectionManager.getConnection(conf)
{code}

A more brutal way of escaping this state is to drop the pool altogether a create a new one (not really doable in all cases depending on the usage)

I'll try to come up with a patch to clean up the faulty HConnection and keep the pool up.

                
> Do not return back to HTablePool closed connections
> ---------------------------------------------------
>
>                 Key: HBASE-6956
>                 URL: https://issues.apache.org/jira/browse/HBASE-6956
>             Project: HBase
>          Issue Type: Bug
>          Components: Client
>    Affects Versions: 0.90.6
>            Reporter: Igor Yurinok
>
> Sometimes we see a lot of Exception about closed connections:
> {code}
>  org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068 closed
> org.apache.hadoop.hbase.client.ClosedConnectionException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068 closed
> {code}
> After investigation we assumed that it occurs because closed connection returns back into HTablePool. 
> For our opinion best solution is  check whether the table is closed in method HTablePool.putTable and if true don't add it into the queue and release such HTableInterface.
> But unfortunatly right now there are no access to HTable#closed field through HTableInterface

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira