You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafodion.apache.org by "Alice Chen (JIRA)" <ji...@apache.org> on 2015/07/22 20:14:31 UTC

[jira] [Created] (TRAFODION-182) LP Bug: 1275082 - HBase Region Server Failure Led to Metadata Corruption

Alice Chen created TRAFODION-182:
------------------------------------

             Summary: LP Bug: 1275082 - HBase Region Server Failure Led to Metadata Corruption
                 Key: TRAFODION-182
                 URL: https://issues.apache.org/jira/browse/TRAFODION-182
             Project: Apache Trafodion
          Issue Type: Bug
          Components: sql-general
            Reporter: Apache Trafodion
            Priority: Critical


Testing on Zircon.  Checking the integrity of some ED tables loaded on Wednesday.  The check script failed at scale 22 due to a region server failure (n001). Client output is below.

Region server was restarted.  Repeat of the check immediately fails with failure to connect to HBase.  Client output is below.  Also unable to issue a query through hpdci without getting failure to connect to hbase errors.

Bounced Trafodion.  Now things are really ugly.   Simple count(*) returns the following:

SQL>select count(*) from trafodion.ed.ed_table_01;

*** ERROR[1394] Trafodion need to be reinitialized on this system due to missing or incorrect metadata objects. [2014-01-31 18:37:33]
*** ERROR[4082] Object TRAFODION.ED.ED_TABLE_01 does not exist or is inaccessible. [2014-01-31 18:37:33]


----------------------------
Initial Client Failure:

   Check Table : 2014-01-31 18:07:13
     trafodion.ed.ed_table_21 :     2014-01-31 18:07:13
     Table trafodion.ed.ed_table_21 : Expected Rows = 2097152 , Actual Rows = 1978146
     Elapsed Seconds : 113.865

   Check Table : 2014-01-31 18:09:07
     trafodion.ed.ed_table_22 :     2014-01-31 18:09:07
SQLException:  *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-703). Cause: 
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Fri Jan 31 18:09:07 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:08 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: n001.cm.cluster/172.31.0.1:60020
Fri Jan 31 18:09:09 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:10 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: n001.cm.cluster/172.31.0.1:60020
Fri Jan 31 18:09:12 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:14 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:18 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:22 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:31 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:47 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused

org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:205)
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:120)
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:628)
org.trafodion.sql.HBaseAccess.HTableClient.startScan(HTableClient.java:187)
. [2014-01-31 18:10:26]
Stack Trace of Exception
  HPT4Messages.java:205>> throwSQLException()
  InterfaceResultSet.java:807>> fetch()
  HPT4ResultSet.java:2868>> next()
  EDLoader.java:712>> checkTable()
  EDLoader.java:1629>> main()

   Test Stopping : 2014-01-31 18:10:26

     Elapsed Seconds : 241.558
/home/squser2/JavaBench>


---------------------------
Client failure after Region Server restart


home/squser2/JavaBench> java -Ddbconnect.properties=conf/zircon.seabase.properties EDLoader 30 all check schema trafodion.ed
EDLoader 30 all check schema trafodion.ed 

EDLoader

     Load Starting        : 2014-01-31 18:25:01
     PropertyFile         : conf/zircon.seabase.properties
     Datebase        : seabase
     Schema               : trafodion.ed
     ScaleFactor          : 0
     LoadScope       : all
     Streams              : 1
     Check           : true
     AutoCommit      : true
     BatchSize       : 1
     IntervalLength       : 10

   Check Table : 2014-01-31 18:25:01
     trafodion.ed.ed_table_01 :     2014-01-31 18:25:02
SQLException:  *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-703). Cause: 
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Fri Jan 31 18:25:02 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:03 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:04 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:05 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:07 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:09 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:13 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:17 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:25 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:41 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.

org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:205)
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:120)
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:628)
org.trafodion.sql.HBaseAccess.HTableClient.startScan(HTableClient.java:187)
. [2014-01-31 18:25:41]
Stack Trace of Exception
  HPT4Messages.java:205>> throwSQLException()
  InterfaceResultSet.java:807>> fetch()
  HPT4ResultSet.java:2868>> next()
  EDLoader.java:712>> checkTable()
  EDLoader.java:1629>> main()

   Test Stopping : 2014-01-31 18:25:41

     Elapsed Seconds :  40.398
/home/squser2/JavaBench>

---------------
hpdci failure after Region Server restart.


/home/squser2/JavaBench> hpdci

Welcome to Trafodion Command Interface 
Copyright(C) 2013-2014 Hewlett-Packard Development Company, L.P.


Connected to Data Source: TDM_Default_DataSource

SQL>select count(*) from trafodion.ed.ed_table_01;

*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::coProcAggr returned error HBASE_ACCESS_ERROR(-705). Cause: 
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Fri Jan 31 18:28:58 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:28:59 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:01 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:02 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:04 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:06 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:10 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:14 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:22 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:38 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.

org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
$Proxy7.getRowNum(Unknown Source)
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.call(AggregationClient.java:220)
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.call(AggregationClient.java:217)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1474)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
java.util.concurrent.FutureTask.run(FutureTask.java:166)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
. [2014-01-31 18:29:38]

SQL>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)