You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Alex Batyrshin (Jira)" <ji...@apache.org> on 2020/06/05 15:10:00 UTC
[jira] [Commented] (HBASE-24245) Failed to write replication wal
position
[ https://issues.apache.org/jira/browse/HBASE-24245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126861#comment-17126861 ]
Alex Batyrshin commented on HBASE-24245:
----------------------------------------
Got the same issue at hbase-1.4.12. Can't find anything strage at zookeeper logs
{{Jun 04 21:13:04 hbase01 hbase[54094]: 2020-06-04 21:13:04,378 FATAL [main-EventThread.replicationSource,kafkaproxy.replicationSource.hbase01.intra%2C60020%2C1591106833218.regiongroup-1,kafkaproxy] regionserver.HRegionServer: ABORTING region server hbase01.intra,60020,1591106833218: Failed to write replication wal position (filename=hbase01.intra%2C60020%2C1591106833218.regiongroup-1.1591293776494, position=115406460)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/hbase01.intra,60020,1591106833218/kafkaproxy/hbase01.intra%2C60020%2C1591106833218.regiongroup-1.1591293776494}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:422)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:824)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:874)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:868)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.setLogPosition(ReplicationQueuesZKImpl.java:155)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:194)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.updateLogPosition(ReplicationSource.java:731)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.shipEdits(ReplicationSource.java:702)}}
{{Jun 04 21:13:04 hbase01 hbase[54094]: at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.run(ReplicationSource.java:555)}}
> Failed to write replication wal position
> ----------------------------------------
>
> Key: HBASE-24245
> URL: https://issues.apache.org/jira/browse/HBASE-24245
> Project: HBase
> Issue Type: Bug
> Components: Replication
> Affects Versions: 1.5.0
> Environment: apache-phoenix-4.15.0-HBase-1.5
> hadoop-2.10.0
> hbase-1.5.0-bin
> zookeeper-3.4.14
> Reporter: luodx
> Priority: Critical
> Fix For: 1.4.10
>
>
> I turned on replication for tables, index tables did not, wrote data using phoneix, and hbase regionserver went down after running for a while.
> regionserver log:
> 2020-04-20 19:22:49,531 FATAL [main-EventThread.replicationSource,1.replicationSource.a2-test-regionserver-52-23.sh%2C16020%2C1587378758523,1] regionserver.HRegionServe
> r: ABORTING region server a2- test-regionserver-52-23.sh,16020,1587378758523: Failed to write replication wal position (filename=a2-test-regionserver-52-23.sh%2C16020%2C
> 1587378758523.1587381739689, position=3388929)
> org.apache.zookeeper. KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/a2-test-regionserver-52-23.sh,16020,1587378758523/1/a2-test-reg
> ionserver-52-23.sh%2C16020%2C1587378758523. 1587381739689
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:510)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil .setData(ZKUtil.java:867)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:917)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:911 )
> at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.setLogPosition(ReplicationQueuesZKImpl.java:155)
> at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:197)
> at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.updateLogPosition(ReplicationSource.java:791)
> at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.shipEdits(ReplicationSource.java:762)
> at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.run(ReplicationSource.java:601)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)