You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "James Taylor (JIRA)" <ji...@apache.org> on 2016/01/24 02:45:39 UTC
[jira] [Resolved] (PHOENIX-1735) Insert data in hbase through
phoenix JDBC when restart hbase service
[ https://issues.apache.org/jira/browse/PHOENIX-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Taylor resolved PHOENIX-1735.
-----------------------------------
Resolution: Cannot Reproduce
Please reopen if the problem persists.
> Insert data in hbase through phoenix JDBC when restart hbase service
> --------------------------------------------------------------------
>
> Key: PHOENIX-1735
> URL: https://issues.apache.org/jira/browse/PHOENIX-1735
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.2.0
> Environment: linux os ( 64G ram,1.5T disk,16 cores) * 4
> Hadoop 2.6.0
> HBase 0.98.6
> Phoenix 4.2.0
> Reporter: wubiandelvbo
> Priority: Critical
> Labels: Region, assigned:, become, for, in, it, still, to, transition,, util.HBaseFsckRepair:, waiting
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> use four clients put data in hbase through phoenix-JDBC when restart hbase service。I discover hbase break down.
> use:
> hbase hbck
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table CARD_INDEX_SALT
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table SYSTEM.CATALOG
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TZJT_CAMERA_SALT
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table T
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TT
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,026 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TT3
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,026 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table hbase:namespace
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TABLENAME
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table SYSTEM.SEQUENCE
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table SYSTEM.STATS
> ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table LOCATION_INDEX_SALT
> 2015-03-14 10:12:01,248 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=30000 watcher=hbase Fsck, quorum=hadoop1:2181,hadoop2:2181,hadoop3:2181, baseZNode=/hbase-unsecure
> 2015-03-14 10:12:01,248 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hadoop1:2181,hadoop2:2181,hadoop3:2181
> 2015-03-14 10:12:01,249 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop1/10.165.97.7:2181. Will not attempt to authenticate using SASL (unknown error)
> 2015-03-14 10:12:01,249 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Socket connection established to hadoop1/10.165.97.7:2181, initiating session
> 2015-03-14 10:12:01,252 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Session establishment complete on server hadoop1/10.165.97.7:2181, sessionid = 0x14c1337679e00f9, negotiated timeout = 30000
> Summary:
> IDX_CC is okay.
> Number of regions: 0
> Deployed on:
> CARD_INDEX is okay.
> Number of regions: 0
> Deployed on:
> US_POPULATION is okay.
> Number of regions: 1
> Deployed on: hadoop4,60020,1426296975241
> EXAMPLE2 is okay.
> Number of regions: 1
> Deployed on: hadoop2,60020,1426296977835
> Table SKIPDEMO is inconsistent.
> Number of regions: 9
> Deployed on: hadoop3,60020,1426296976617 hadoop4,60020,1426296975241
> STUDENT is okay.
> Number of regions: 0
> Deployed on:
> LOCATION_INDEX is okay.
> Number of regions: 0
> Deployed on:
> TZJT_CAMERA is okay.
> Number of regions: 0
> Deployed on:
> hbase:meta is okay.
> Number of regions: 1
> Deployed on: hadoop2,60020,1426296977835
> CARD_INDEX_SALT is okay.
> Number of regions: 0
> Deployed on:
> SYSTEM.CATALOG is okay.
> Number of regions: 0
> Deployed on:
> TZJT_CAMERA_SALT is okay.
> Number of regions: 0
> Deployed on:
> T is okay.
> Number of regions: 0
> Deployed on:
> TT is okay.
> Number of regions: 0
> Deployed on:
> TT3 is okay.
> Number of regions: 0
> Deployed on:
> hbase:namespace is okay.
> Number of regions: 0
> Deployed on:
> TABLENAME is okay.
> Number of regions: 0
> Deployed on:
> SYSTEM.SEQUENCE is okay.
> Number of regions: 0
> Deployed on:
> SYSTEM.STATS is okay.
> Number of regions: 0
> Deployed on:
> LOCATION_INDEX_SALT is okay.
> Number of regions: 0
> Deployed on:
> 462 inconsistencies detected.
> Status: INCONSISTENT
> 2015-03-14 10:12:01,255 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
> so,use 'hbase hbck -fixMeta -fixAssignments hbase:namespace '，'hbase hbck -repair hbase:namespace' show:
> 2015-03-14 10:18:14,988 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}
> 2015-03-14 10:18:15,994 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}
> 2015-03-14 10:18:17,000 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}
> 2015-03-14 10:18:18,005 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}
> 2015-03-14 10:18:19,011 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}
> 2015-03-14 10:18:24,041 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
> 2015-03-14 10:18:24,041 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24c15ce67b80047
> 2015-03-14 10:18:24,045 INFO [main] zookeeper.ZooKeeper: Session: 0x24c15ce67b80047 closed
> 2015-03-14 10:18:24,045 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
> 2015-03-14 10:18:24,045 DEBUG [main] ipc.RpcClient: Stopping rpc client
> Exception in thread "main" java.io.IOException: Region {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''} failed to move out of transition within timeout 120000ms
> at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:141)
> at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1798)
> at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1969)
> at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1579)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:473)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:492)
> at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4213)
> at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4022)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4010)
> ------------------------------
> the hbase still not normal.
> I don't know why ?
> Is bug ?
> My operational problems?
> Can someone help me
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)