[jira] [Created] (TRAFODION-188) LP Bug: 1286349 - Insert..select returns HBASE_ACCESS_ERROR(-705) and then TMF error 97 in org.apache.hadoop.hbase.regionserver.transactional

["Alice Chen (JIRA)" <ji...@apache.org>]

Alice Chen created TRAFODION-188:
------------------------------------

             Summary: LP Bug: 1286349 - Insert..select returns HBASE_ACCESS_ERROR(-705) and then TMF error 97 in org.apache.hadoop.hbase.regionserver.transactional
                 Key: TRAFODION-188
                 URL: https://issues.apache.org/jira/browse/TRAFODION-188
             Project: Apache Trafodion
          Issue Type: Bug
          Components: dtm
            Reporter: Weishiun Tsai
            Priority: Critical


A insert..select statement returns HBASE_ACCSES_ERROR(-705) in org.apache.hadoop.hbase.regionserver.transactional and then TMF error 97 afterwards, as shown here.  This problem is fairly reproducible now with the QA test to create and populate hcubedb through sqlci/JDBC.  This may or may not be related to https://bugs.launchpad.net/trafodion/+bug/1274716 “Getting TM error 97 for INSERT SELECT <hive table>”.  In this case, all tables from the select list are Trafodion/hbase tables, but in the other case, the select table is a hive table.  For now, we use this case to track the problem separately.

SQL>insert into cube1 select t1.a, t6.a, t8.a, t1.a, t6.a, t8.a, 'some text'
from t1, t6, t8 where t8.a < 100;
*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::rowExists returned error HBASE_ACCESS_ERROR(-705). Cause:
org.apache.hadoop.hbase.client.transactional.UnknownTransactionException: org.apache.hadoop.hbase.client.transactional.UnknownTransactionException: transaction: [4294996353], region: [TRAFODION.G_HCUBEDB.CUBE1,,1393442223602.979ac8a950b9df29d1ab4832dd9acb37.]
        at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion.getTransactionState(TransactionalRegion.java:741)
        at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion.checkAndPut(TransactionalRegion.java:941)
        at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.checkAndPut(TransactionalRegionServer.java:449)
        at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428)

sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:526)
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166)
org.apache.hadoop.hbase.client.transactional.TransactionalTable.checkAndPut(TransactionalTable.java:212)
org.apache.hadoop.hbase.client.transactional.RMInterface.checkAndPut(RMInterface.java:164)
org.trafodion.sql.HBaseAccess.HTableClient.checkAndInsertRow(HTableClient.java:537)
. [2014-02-26 11:36:31]
*** ERROR[8610] Transaction subsystem TMF reported error 97 on a waited rollback transaction. [2014-02-26 11:36:31]

This problem has been reproduced on rhel-cdh1 with the following TM tracing and hbase tracing turned on.  Narendra has collected the tracing info for further debugging.

1) Change the logging threshold of the HBase Region Server from ‘INFO’ to ‘DEBUG’: Via the Cloudera Manager:
                                                     i. Click on HBase Service->Configuration tab -> ‘View and Edit’
1. Click on RegionServer -> Logs -> Logging Threshold to ‘DEBUG’
                                                   ii. Restart HBase

2) Trace the Trafodion TM:
 a. In sqf/etc/ms.env (then pdcp to all the nodes):

# DTM Tracing. To enable standard tracing uncomment the following lines.
 TM_TRACE=1
 HBASE_TRACE=0xf
 TM_TRACE_DETAIL=3
 TM_TRACE_UNIQUE=1
 TM_TRACE_FILE=zzttm.
 b. Restart SQ


3) In the file $MY_SQROOT/logs/log4j.dtm.config (all nodes, log file: $MY_SQROOT/sql/scripts/trafodion.dtm.log):

log4j.logger.org.trafodion.dtm=TRACE (change it from DEBUG to TRACE)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)