You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vinod Gupta Tankala <tv...@readypulse.com> on 2011/09/27 08:26:55 UTC
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 3 actions: WrongRegionException:
Hi,
I have seen similar issue being handled here but i didn't fully understand
how to fix it in my case. this is the earlier thread -
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/18410
i am seeing the below issues in my client -
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
3 ac
tions: WrongRegionException: 3 times, servers with issues:
ip-10-32-61-60.ec2.in
ternal:44911,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
tation.processBatch(HConnectionManager.java:1220)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
tation.processBatchOfPuts(HConnectionManager.java:1234)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
...
also, hbase hbck -details gives the following errors. -fix doesn't do
anything. neither does check_meta.rb.
ERROR: Region
AkStats,277808094:1314921600:daily:Volume,1317052667861.1ecc871503cd827934c3a9077b44e52b.
not deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/3520c0cfbedcf212084379c0e41e7839
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/3af08f3ba88d7c2785fd089702f89241
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/a522e78c7d018547b8979ed6fd381358
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/b89f7bb47e53ea9cc729c86978a7327c
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/fd7ce07003bfd883f53a210bb0985065
on HDFS, but not listed in META or deployed on any region server.
Chain of regions in table AkStats is broken; edges does not contain
277808094:1314921600:daily:Volume
ERROR: Found inconsistency in table AkStats
Summary:
-ROOT- is okay.
Number of regions: 1
Deployed on: ip-10-32-61-60.ec2.internal:55915
.META. is okay.
Number of regions: 1
Deployed on: ip-10-32-61-60.ec2.internal:55915
Chain of regions in table AkStats is broken; edges does not contain
277808094:1314921600:daily:Volume
Table AkStats is inconsistent.
Number of regions: 133
Deployed on: ip-10-32-61-60.ec2.internal:55915
7 inconsistencies detected.
Status: INCONSISTENT
my production system is down on its knees and I need to get unblocked asap.
any help will be highly appreciated.
thanks
vinod
Fwd: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 3 actions: WrongRegionException:
Posted by Vinod Gupta Tankala <tv...@socialyantra.com>.
Hi,
I have seen similar issue being handled here but i didn't fully understand
how to fix it in my case. this is the earlier thread -
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/18410
i am seeing the below issues in my client -
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
3 ac
tions: WrongRegionException: 3 times, servers with issues:
ip-10-32-61-60.ec2.in
ternal:44911,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
tation.processBatch(HConnectionManager.java:1220)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
tation.processBatchOfPuts(HConnectionManager.java:1234)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
...
also, hbase hbck -details gives the following errors. -fix doesn't do
anything. neither does check_meta.rb.
ERROR: Region
AkStats,277808094:1314921600:daily:Volume,1317052667861.1ecc871503cd827934c3a9077b44e52b.
not deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/3520c0cfbedcf212084379c0e41e7839
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/3af08f3ba88d7c2785fd089702f89241
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/a522e78c7d018547b8979ed6fd381358
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/b89f7bb47e53ea9cc729c86978a7327c
on HDFS, but not listed in META or deployed on any region server.
ERROR: Region
file:/media/ephemeral0/hbase-data/AkStats/fd7ce07003bfd883f53a210bb0985065
on HDFS, but not listed in META or deployed on any region server.
Chain of regions in table AkStats is broken; edges does not contain
277808094:1314921600:daily:Volume
ERROR: Found inconsistency in table AkStats
Summary:
-ROOT- is okay.
Number of regions: 1
Deployed on: ip-10-32-61-60.ec2.internal:55915
.META. is okay.
Number of regions: 1
Deployed on: ip-10-32-61-60.ec2.internal:55915
Chain of regions in table AkStats is broken; edges does not contain
277808094:1314921600:daily:Volume
Table AkStats is inconsistent.
Number of regions: 133
Deployed on: ip-10-32-61-60.ec2.internal:55915
7 inconsistencies detected.
Status: INCONSISTENT
my production system is down on its knees and I need to get unblocked asap.
any help will be highly appreciated.
thanks
vinod