You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafodion.apache.org by "Alice Chen (JIRA)" <ji...@apache.org> on 2015/07/22 20:15:53 UTC
[jira] [Created] (TRAFODION-392) LP Bug: 1333867 - Create index on
a table with 11M rows causes hbase region server to go down
Alice Chen created TRAFODION-392:
------------------------------------
Summary: LP Bug: 1333867 - Create index on a table with 11M rows causes hbase region server to go down
Key: TRAFODION-392
URL: https://issues.apache.org/jira/browse/TRAFODION-392
Project: Apache Trafodion
Issue Type: Bug
Components: sql-exe
Reporter: Weishiun Tsai
Assignee: Apache Trafodion
Priority: Blocker
Fix For: 0.9 (pre-incubation)
Create index on this table with 11M rows always seem to bring down hbase region servers. From sqlci, the statement returned error 8448, which said that ExpHbaseInterface::insertRows() returned HBASE_ACCESS_ERROR(-705). Before returned the hbase access error, ExpHbaseInterface::insertRows() retried with a certain number, and eventually got a RetriesExhaustedException. The real cause, however, is that some hbase servers (sometimes one, sometimes more than one) have gone down. This can be seen from the Cloudera Manager for a Cloudera machine, or Ambari for a Hortonworks machine.
This might an issue that can be addressed from configuring hbase in a certain way, improving the internal implementation of the index creation, or a combination of both. The problem is fairly reproducible right now on centos-mapr1 (a 6-node Cloudera machine) or rhel-mapr1 (a 4-node HortonWorks machine) on the GIT 0619_0930 build.
Here is the entire script to reproduce this problem (it requires the QA global tables g_tpch2x to be populated first)
-------------------------------------------------------------
log mytest.log clear;
set schema myline3sch;
create table myline3 like trafodion.g_tpch2x.lineitem with constraints with partitions;
invoke trafodion.g_tpch2x.lineitem;
invoke myline3;
upsert using load into myline3 (select * from trafodion.g_tpch2x.lineitem);
select count(*) from myline3;
create index myline3index on myline3 (l_suppkey);
-------------------------------------------------------------
Here is the execution output of the script on centos-mapr1:
[trafodion@centos-mapr1 scripts]$ sqlci | cat
Trafodion Conversational Interface 0.8.1
(c) Copyright 2014 Hewlett-Packard Development Company, LP.
>>obey mytest.sql;
>>log mytest.log clear;
>>
>>set schema myline3sch;
--- SQL operation complete.
>>
>>create table myline3 like trafodion.g_tpch2x.lineitem with constraints with partitions;
--- SQL operation complete.
>>
>>invoke trafodion.g_tpch2x.lineitem;
-- Definition of Trafodion table TRAFODION.G_TPCH2X.LINEITEM
-- Definition current Tue Jun 24 09:03:01 2014
(
"_SALT_" INT UNSIGNED NO DEFAULT NOT NULL NOT
DROPPABLE
, L_ORDERKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_PARTKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_SUPPKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_LINENUMBER INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_QUANTITY NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_EXTENDEDPRICE NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_DISCOUNT NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_TAX NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_RETURNFLAG CHAR(1) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_LINESTATUS CHAR(1) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_SHIPDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
, L_COMMITDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
, L_RECEIPTDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
, L_SHIPINSTRUCT CHAR(25) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_SHIPMODE CHAR(10) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_COMMENT VARCHAR(44) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
)
PRIMARY KEY ("_SALT_" ASC, L_SHIPDATE ASC, L_ORDERKEY ASC, L_LINENUMBER ASC)
--- SQL operation complete.
>>
>>invoke myline3;
-- Definition of Trafodion table TRAFODION.MYLINE3SCH.MYLINE3
-- Definition current Tue Jun 24 09:03:01 2014
(
"_SALT_" INT UNSIGNED NO DEFAULT NOT NULL NOT
DROPPABLE
, L_ORDERKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_PARTKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_SUPPKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_LINENUMBER INT NO DEFAULT NOT NULL NOT DROPPABLE
, L_QUANTITY NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_EXTENDEDPRICE NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_DISCOUNT NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_TAX NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
DROPPABLE
, L_RETURNFLAG CHAR(1) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_LINESTATUS CHAR(1) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_SHIPDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
, L_COMMITDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
, L_RECEIPTDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
, L_SHIPINSTRUCT CHAR(25) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_SHIPMODE CHAR(10) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
, L_COMMENT VARCHAR(44) CHARACTER SET ISO88591 COLLATE
DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
)
PRIMARY KEY ("_SALT_" ASC, L_SHIPDATE ASC, L_ORDERKEY ASC, L_LINENUMBER ASC)
--- SQL operation complete.
>>
>>upsert using load into myline3 (select * from trafodion.g_tpch2x.lineitem);
--- 11997996 row(s) inserted.
>>
>>select count(*) from myline3;
(EXPR)
--------------------
11997996
--- 1 row(s) selected.
>>
>>create index myline3index on myline3 (l_suppkey);
*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::insertRows returned error HBASE_ACCESS_ERROR(-705). Cause:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Tue Jun 24 09:32:04 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.io.IOException: Call to centos-mapr3.hpl.hp.com/15.25.119.71:60020 failed on local exception: java.io.EOFException
Tue Jun 24 09:32:05 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
Tue Jun 24 09:32:06 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: centos-mapr3.hpl.hp.com/15.25.119.71:60020
Tue Jun 24 09:32:08 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
Tue Jun 24 09:32:10 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
Tue Jun 24 09:32:14 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
Tue Jun 24 09:32:18 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
Tue Jun 24 09:32:26 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
Tue Jun 24 09:32:42 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
Tue Jun 24 09:33:14 PDT 2014, org.apache.hadoop.hbase.client.transactional.TransactionalTable$7@19a86a97, java.net.ConnectException: Connection refused
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:186)
org.apache.hadoop.hbase.client.transactional.TransactionalTable.put(TransactionalTable.java:305)
org.apache.hadoop.hbase.client.transactional.RMInterface.put(RMInterface.java:145)
org.trafodion.sql.HBaseAccess.HTableClient.insertRows(HTableClient.java:549)
.
*** ERROR[1389] Object MYLINE3INDEX does not exist in Trafodion.
--- SQL operation failed with errors.
-------------------------------------------------------------
Here are the Critical log entries for the hbase region server on centos-mapr3 before the region server has gone down:
June 24 2014 9:33 AM CRITICAL The health test result for REGION_SERVER_MASTER_CONNECTIVITY has become bad: This RegionServer is not connected to its cluster.
June 24 2014 9:32 AM CRITICAL The health test result for REGION_SERVER_UNEXPECTED_EXITS has become bad: This role encountered 1 unexpected exit(s) in the previous 5 minute(s). Critical threshold: any.
June 24 2014 9:32 AM CRITICAL The health test result for REGION_SERVER_SCM_HEALTH has become bad: This role's process exited. This role is supposed to be started.
June 24 2014 9:12 AM CRITICAL Region TRAFODION.MYLINE3SCH.MYLINE3,\x00\x00\x00\x0C\x00\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00,1403625779307.e60a7f914fcbe58b688fe514fe41c266. has too many store files; delaying flush up to 90000ms
June 24 2014 9:08 AM CRITICAL Region TRAFODION.MYLINE3SCH.MYLINE3,,1403625779306.72573e376c67d7bc03eeb69bcc000484. has too many store files; delaying flush up to 90000ms
Assigned to LaunchPad User khaled Bouaziz
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)