You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafodion.apache.org by "Alice Chen (JIRA)" <ji...@apache.org> on 2015/07/22 20:14:31 UTC
[jira] [Created] (TRAFODION-182) LP Bug: 1275082 - HBase Region
Server Failure Led to Metadata Corruption
Alice Chen created TRAFODION-182:
------------------------------------
Summary: LP Bug: 1275082 - HBase Region Server Failure Led to Metadata Corruption
Key: TRAFODION-182
URL: https://issues.apache.org/jira/browse/TRAFODION-182
Project: Apache Trafodion
Issue Type: Bug
Components: sql-general
Reporter: Apache Trafodion
Priority: Critical
Testing on Zircon. Checking the integrity of some ED tables loaded on Wednesday. The check script failed at scale 22 due to a region server failure (n001). Client output is below.
Region server was restarted. Repeat of the check immediately fails with failure to connect to HBase. Client output is below. Also unable to issue a query through hpdci without getting failure to connect to hbase errors.
Bounced Trafodion. Now things are really ugly. Simple count(*) returns the following:
SQL>select count(*) from trafodion.ed.ed_table_01;
*** ERROR[1394] Trafodion need to be reinitialized on this system due to missing or incorrect metadata objects. [2014-01-31 18:37:33]
*** ERROR[4082] Object TRAFODION.ED.ED_TABLE_01 does not exist or is inaccessible. [2014-01-31 18:37:33]
----------------------------
Initial Client Failure:
Check Table : 2014-01-31 18:07:13
trafodion.ed.ed_table_21 : 2014-01-31 18:07:13
Table trafodion.ed.ed_table_21 : Expected Rows = 2097152 , Actual Rows = 1978146
Elapsed Seconds : 113.865
Check Table : 2014-01-31 18:09:07
trafodion.ed.ed_table_22 : 2014-01-31 18:09:07
SQLException: *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-703). Cause:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Fri Jan 31 18:09:07 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:08 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: n001.cm.cluster/172.31.0.1:60020
Fri Jan 31 18:09:09 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:10 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: n001.cm.cluster/172.31.0.1:60020
Fri Jan 31 18:09:12 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:14 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:18 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:22 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:31 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
Fri Jan 31 18:09:47 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@f9a4520, java.net.ConnectException: Connection refused
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:205)
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:120)
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:628)
org.trafodion.sql.HBaseAccess.HTableClient.startScan(HTableClient.java:187)
. [2014-01-31 18:10:26]
Stack Trace of Exception
HPT4Messages.java:205>> throwSQLException()
InterfaceResultSet.java:807>> fetch()
HPT4ResultSet.java:2868>> next()
EDLoader.java:712>> checkTable()
EDLoader.java:1629>> main()
Test Stopping : 2014-01-31 18:10:26
Elapsed Seconds : 241.558
/home/squser2/JavaBench>
---------------------------
Client failure after Region Server restart
home/squser2/JavaBench> java -Ddbconnect.properties=conf/zircon.seabase.properties EDLoader 30 all check schema trafodion.ed
EDLoader 30 all check schema trafodion.ed
EDLoader
Load Starting : 2014-01-31 18:25:01
PropertyFile : conf/zircon.seabase.properties
Datebase : seabase
Schema : trafodion.ed
ScaleFactor : 0
LoadScope : all
Streams : 1
Check : true
AutoCommit : true
BatchSize : 1
IntervalLength : 10
Check Table : 2014-01-31 18:25:01
trafodion.ed.ed_table_01 : 2014-01-31 18:25:02
SQLException: *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::scanOpen returned error HBASE_OPEN_ERROR(-703). Cause:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Fri Jan 31 18:25:02 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:03 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:04 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:05 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:07 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:09 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:13 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:17 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:25 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:25:41 GMT 2014, org.apache.hadoop.hbase.client.ScannerCallable@7f8f8577, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:205)
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:120)
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:628)
org.trafodion.sql.HBaseAccess.HTableClient.startScan(HTableClient.java:187)
. [2014-01-31 18:25:41]
Stack Trace of Exception
HPT4Messages.java:205>> throwSQLException()
InterfaceResultSet.java:807>> fetch()
HPT4ResultSet.java:2868>> next()
EDLoader.java:712>> checkTable()
EDLoader.java:1629>> main()
Test Stopping : 2014-01-31 18:25:41
Elapsed Seconds : 40.398
/home/squser2/JavaBench>
---------------
hpdci failure after Region Server restart.
/home/squser2/JavaBench> hpdci
Welcome to Trafodion Command Interface
Copyright(C) 2013-2014 Hewlett-Packard Development Company, L.P.
Connected to Data Source: TDM_Default_DataSource
SQL>select count(*) from trafodion.ed.ed_table_01;
*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::coProcAggr returned error HBASE_ACCESS_ERROR(-705). Cause:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Fri Jan 31 18:28:58 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:28:59 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:01 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:02 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:04 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:06 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:10 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:14 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:22 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
Fri Jan 31 18:29:38 UTC 2014, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@4755d962, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: TRAFODION.ED.ED_TABLE_01,\x00\x00\x00\x09\x00\x00\x00\x00,1391035830752.0a06a50b09824d0bfc983cada9634fa1.
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
$Proxy7.getRowNum(Unknown Source)
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.call(AggregationClient.java:220)
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$3.call(AggregationClient.java:217)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1474)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
java.util.concurrent.FutureTask.run(FutureTask.java:166)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
. [2014-01-31 18:29:38]
SQL>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)