You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/04/22 02:17:22 UTC
[jira] Created: (HBASE-599) CME iterating return from
ServerManager.getMarkedToClose
CME iterating return from ServerManager.getMarkedToClose
--------------------------------------------------------
Key: HBASE-599
URL: https://issues.apache.org/jira/browse/HBASE-599
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Below is a note from list from David Alves
{code}
I'm seeing some CMEs in the logs they occurred while I still had bad
dfs.replication settings between hadoop and hbase but still thought you
should know.
Trace:
2008-04-21 13:20:46,443 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
java.io.IOException: java.io.IOException:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$ValueIterator.next(HashMap.java:822)
at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:350)
at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:299)
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:217)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:560)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:388)
at java.lang.Thread.run(Thread.java:619)
{code}
Looking at it, the fix would seem to be cloning the Map we give out but the clone needs to be done inside a synchronize of the Map being cloned so some more extensive synchronizing needs to be added to the ServerManager class. Looking too, it seems like regionsToClose is not protected everywhere (maybe I'm not reading it right).
Bryan, you'd probably be best handlng this issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-599) CME iterating return from
ServerManager.getMarkedToClose
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HBASE-599:
-----------------------------------
Assignee: Jim Kellerman
> CME iterating return from ServerManager.getMarkedToClose
> --------------------------------------------------------
>
> Key: HBASE-599
> URL: https://issues.apache.org/jira/browse/HBASE-599
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jim Kellerman
> Fix For: 0.2.0
>
>
> Below is a note from list from David Alves
> {code}
> I'm seeing some CMEs in the logs they occurred while I still had bad
> dfs.replication settings between hadoop and hbase but still thought you
> should know.
> Trace:
> 2008-04-21 13:20:46,443 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$ValueIterator.next(HashMap.java:822)
> at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:350)
> at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:299)
> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:217)
> at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:560)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:388)
> at java.lang.Thread.run(Thread.java:619)
> {code}
> Looking at it, the fix would seem to be cloning the Map we give out but the clone needs to be done inside a synchronize of the Map being cloned so some more extensive synchronizing needs to be added to the ServerManager class. Looking too, it seems like regionsToClose is not protected everywhere (maybe I'm not reading it right).
> Bryan, you'd probably be best handlng this issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-599) CME iterating return from
ServerManager.getMarkedToClose
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592213#action_12592213 ]
Jim Kellerman commented on HBASE-599:
-------------------------------------
Do you have a test case you can submit? I have a patch but no way to test it.
---
Jim Kellerman, Senior Engineer; Powerset
sian.jira.plugin.system.issuetabpanels:all-tabpanel ]
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.23.4/1395 - Release Date: 4/24/2008 7:24 AM
> CME iterating return from ServerManager.getMarkedToClose
> --------------------------------------------------------
>
> Key: HBASE-599
> URL: https://issues.apache.org/jira/browse/HBASE-599
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
>
> Below is a note from list from David Alves
> {code}
> I'm seeing some CMEs in the logs they occurred while I still had bad
> dfs.replication settings between hadoop and hbase but still thought you
> should know.
> Trace:
> 2008-04-21 13:20:46,443 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$ValueIterator.next(HashMap.java:822)
> at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:350)
> at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:299)
> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:217)
> at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:560)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:388)
> at java.lang.Thread.run(Thread.java:619)
> {code}
> Looking at it, the fix would seem to be cloning the Map we give out but the clone needs to be done inside a synchronize of the Map being cloned so some more extensive synchronizing needs to be added to the ServerManager class. Looking too, it seems like regionsToClose is not protected everywhere (maybe I'm not reading it right).
> Bryan, you'd probably be best handlng this issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-599) CME iterating return from
ServerManager.getMarkedToClose
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-599:
--------------------------------
Priority: Critical (was: Major)
This is really bad.
> CME iterating return from ServerManager.getMarkedToClose
> --------------------------------------------------------
>
> Key: HBASE-599
> URL: https://issues.apache.org/jira/browse/HBASE-599
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
>
> Below is a note from list from David Alves
> {code}
> I'm seeing some CMEs in the logs they occurred while I still had bad
> dfs.replication settings between hadoop and hbase but still thought you
> should know.
> Trace:
> 2008-04-21 13:20:46,443 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$ValueIterator.next(HashMap.java:822)
> at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:350)
> at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:299)
> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:217)
> at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:560)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:388)
> at java.lang.Thread.run(Thread.java:619)
> {code}
> Looking at it, the fix would seem to be cloning the Map we give out but the clone needs to be done inside a synchronize of the Map being cloned so some more extensive synchronizing needs to be added to the ServerManager class. Looking too, it seems like regionsToClose is not protected everywhere (maybe I'm not reading it right).
> Bryan, you'd probably be best handlng this issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-599) CME iterating return from
ServerManager.getMarkedToClose
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-599:
------------------------
Fix Version/s: 0.2.0
Fix for 0.2.0. Lots of possibilities for CMEs in SM it seems.
> CME iterating return from ServerManager.getMarkedToClose
> --------------------------------------------------------
>
> Key: HBASE-599
> URL: https://issues.apache.org/jira/browse/HBASE-599
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.2.0
>
>
> Below is a note from list from David Alves
> {code}
> I'm seeing some CMEs in the logs they occurred while I still had bad
> dfs.replication settings between hadoop and hbase but still thought you
> should know.
> Trace:
> 2008-04-21 13:20:46,443 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$ValueIterator.next(HashMap.java:822)
> at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:350)
> at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:299)
> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:217)
> at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:560)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:388)
> at java.lang.Thread.run(Thread.java:619)
> {code}
> Looking at it, the fix would seem to be cloning the Map we give out but the clone needs to be done inside a synchronize of the Map being cloned so some more extensive synchronizing needs to be added to the ServerManager class. Looking too, it seems like regionsToClose is not protected everywhere (maybe I'm not reading it right).
> Bryan, you'd probably be best handlng this issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-599) CME iterating return from
ServerManager.getMarkedToClose
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman resolved HBASE-599.
---------------------------------
Resolution: Fixed
Committed with patch for HBASE-478
> CME iterating return from ServerManager.getMarkedToClose
> --------------------------------------------------------
>
> Key: HBASE-599
> URL: https://issues.apache.org/jira/browse/HBASE-599
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
>
> Below is a note from list from David Alves
> {code}
> I'm seeing some CMEs in the logs they occurred while I still had bad
> dfs.replication settings between hadoop and hbase but still thought you
> should know.
> Trace:
> 2008-04-21 13:20:46,443 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> java.io.IOException: java.io.IOException:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$ValueIterator.next(HashMap.java:822)
> at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:350)
> at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:299)
> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:217)
> at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:560)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:388)
> at java.lang.Thread.run(Thread.java:619)
> {code}
> Looking at it, the fix would seem to be cloning the Map we give out but the clone needs to be done inside a synchronize of the Map being cloned so some more extensive synchronizing needs to be added to the ServerManager class. Looking too, it seems like regionsToClose is not protected everywhere (maybe I'm not reading it right).
> Bryan, you'd probably be best handlng this issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.