You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "chenglei (JIRA)" <ji...@apache.org> on 2017/06/20 13:58:00 UTC
[jira] [Updated] (PHOENIX-3964) Index.preWALRestore should handle
index write failure
[ https://issues.apache.org/jira/browse/PHOENIX-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chenglei updated PHOENIX-3964:
------------------------------
Description:
When I restarted my hbase cluster a certain time, I noticed some region is in FAILED_OPEN state and the RegionServer has error logs as following:
{code:java}
2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] handler.OpenRegionHandler: Failed open of region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5., starting to roll back the global memstore size.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: Failed to write to multiple index tables
at org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
at org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
at org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
> Index.preWALRestore should handle index write failure
> -----------------------------------------------------
>
> Key: PHOENIX-3964
> URL: https://issues.apache.org/jira/browse/PHOENIX-3964
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.10.0
> Reporter: chenglei
>
> When I restarted my hbase cluster a certain time, I noticed some region is in FAILED_OPEN state and the RegionServer has error logs as following:
> {code:java}
> 2017-06-20 12:31:30,493 ERROR [RS_OPEN_REGION-rsync:60020-3] handler.OpenRegionHandler: Failed open of region=BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,0100134e-7ddf-4d13-ab77-6f0d683e5fad_0,1487594358223.57a7be72f9beaeb4285529ac6236f4e5., starting to roll back the global memstore size.
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: Failed to write to multiple index tables
> at org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:221)
> at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
> at org.apache.phoenix.hbase.index.write.RecoveryIndexWriter.write(RecoveryIndexWriter.java:75)
> at org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:554)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1312)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1549)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1308)
> at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3338)
> at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3220)
> at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:823)
> at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:716)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:687)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4596)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4566)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4538)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4494)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4445)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)