You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/04/07 19:12:13 UTC
[jira] Created: (HBASE-1315) when HRS znode expires while HRS is
still running, HRS enters infinite loop
when HRS znode expires while HRS is still running, HRS enters infinite loop
---------------------------------------------------------------------------
Key: HBASE-1315
URL: https://issues.apache.org/jira/browse/HBASE-1315
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.0
Reporter: Andrew Purtell
See HBASE-1314. In addition, the HRS will indefinitely try to check in even though it is getting back LeaseStillHeld exceptions. Both the master and HRS logs are, after a few hours, filled with thousands of these:
>From the HRS log
2009-04-07 10:17:46,136 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Exceeded max retries: 10
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:493)
at java.lang.Thread.run(Thread.java:619)
They come in to the log about 4 times per second.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1315) when HRS znode expires while HRS is
still running, HRS enters infinite loop
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-1315.
-----------------------------------
Resolution: Fixed
Fixed by HBASE-1311
> when HRS znode expires while HRS is still running, HRS enters infinite loop
> ---------------------------------------------------------------------------
>
> Key: HBASE-1315
> URL: https://issues.apache.org/jira/browse/HBASE-1315
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
>
> See HBASE-1314. In addition, the HRS will indefinitely try to check in even though it is getting back LeaseStillHeld exceptions. Both the master and HRS logs are, after a few hours, filled with thousands of these:
> From the HRS log
> 2009-04-07 10:17:46,136 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Exceeded max retries: 10
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:493)
> at java.lang.Thread.run(Thread.java:619)
> They come in to the log about 4 times per second.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1315) when HRS znode expires while HRS is
still running, HRS enters infinite loop
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1315:
----------------------------------
Priority: Blocker (was: Major)
Raise priority to blocker. This prevents any recovery after a ZK session expiry problem.
> when HRS znode expires while HRS is still running, HRS enters infinite loop
> ---------------------------------------------------------------------------
>
> Key: HBASE-1315
> URL: https://issues.apache.org/jira/browse/HBASE-1315
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
>
> See HBASE-1314. In addition, the HRS will indefinitely try to check in even though it is getting back LeaseStillHeld exceptions. Both the master and HRS logs are, after a few hours, filled with thousands of these:
> From the HRS log
> 2009-04-07 10:17:46,136 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Exceeded max retries: 10
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:493)
> at java.lang.Thread.run(Thread.java:619)
> They come in to the log about 4 times per second.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1315) when HRS znode expires while HRS is
still running, HRS enters infinite loop
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707852#action_12707852 ]
Nitay Joffe commented on HBASE-1315:
------------------------------------
This gets fixed by HBASE-1311 as well. They're both just different places of looping because of not handling session expired.
> when HRS znode expires while HRS is still running, HRS enters infinite loop
> ---------------------------------------------------------------------------
>
> Key: HBASE-1315
> URL: https://issues.apache.org/jira/browse/HBASE-1315
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
>
> See HBASE-1314. In addition, the HRS will indefinitely try to check in even though it is getting back LeaseStillHeld exceptions. Both the master and HRS logs are, after a few hours, filled with thousands of these:
> From the HRS log
> 2009-04-07 10:17:46,136 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Exceeded max retries: 10
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:493)
> at java.lang.Thread.run(Thread.java:619)
> They come in to the log about 4 times per second.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1315) when HRS znode expires while HRS is
still running, HRS enters infinite loop
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nitay Joffe updated HBASE-1315:
-------------------------------
Fix Version/s: 0.20.0
> when HRS znode expires while HRS is still running, HRS enters infinite loop
> ---------------------------------------------------------------------------
>
> Key: HBASE-1315
> URL: https://issues.apache.org/jira/browse/HBASE-1315
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Fix For: 0.20.0
>
>
> See HBASE-1314. In addition, the HRS will indefinitely try to check in even though it is getting back LeaseStillHeld exceptions. Both the master and HRS logs are, after a few hours, filled with thousands of these:
> From the HRS log
> 2009-04-07 10:17:46,136 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Exceeded max retries: 10
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:493)
> at java.lang.Thread.run(Thread.java:619)
> They come in to the log about 4 times per second.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.