[jira] [Updated] (HBASE-3692) Handle RejectedExecutionException in HTable

["Andy Jefferson (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andy Jefferson updated HBASE-3692:
----------------------------------

    Attachment: test_datanucleus.zip

I've also seen this when accessing HBase through DataNucleus (JDO or JPA APIs). Never occurs on 0.90.0 (or earlier), yet switching to 0.90.1 through 0.90.4 the same testcase fails. Apologies that I haven't been able to narrow it down to the precise HBase calls, but am attaching a testcase that uses DataNucleus+HBase if that is of use (it's as minimal as I can get) - understand totally if you don't want to have 3rd party libs in a testcase, just that it does reproduce the problem (can you see what HBase API calls are made from the log, for example?).

Stack trace with hbase-0.90.3 and hadoop-core-1.0.0 is as follows 

java.util.concurrent.RejectedExecutionException
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1143)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1241)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:826)
        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:682)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:667)
        at org.datanucleus.store.hbase.HBasePersistenceHandler.insertObject(HBasePersistenceHandler.java:274)

i.e HTable.put() is called. Note this is *single* threaded.

Test is a Maven project, and simply type "mvn clean compile exec:java" to execute it.
                
> Handle RejectedExecutionException in HTable
> -------------------------------------------
>
>                 Key: HBASE-3692
>                 URL: https://issues.apache.org/jira/browse/HBASE-3692
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.90.1
>            Reporter: Jean-Daniel Cryans
>         Attachments: test_datanucleus.zip
>
>
> A user on IRC yesterday had an issue with RejectedExecutionException coming out of HTable sometimes. Apart from being very confusing to the user as it comes with no message at all, it exposes the HTable internals. 
> I think we should handle it and instead throw something like DontUseHTableInMultipleThreadsException or something more clever. In his case, the user had a HTable leak with the pool that he was able to figure out once I told him what to look for.
> It could be an unchecked exception and we could consider adding in 0.90 but marking for 0.92 at the moment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira