[jira] Commented: (HBASE-2169) CME in master regionsAwaitingAssignment

["James Baldassari (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871761#action_12871761 ] 

James Baldassari commented on HBASE-2169:
-----------------------------------------

I've recently seen the following stack trace in our logs (0.20.3):

java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
        at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
        at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:415)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:196)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:482)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:414)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:323)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:724)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:574)
        at java.lang.Thread.run(Thread.java:619)

After looking at the source it seemed like a simple matter of synchronizing on regionsInTransition before iterating over it, so I was going to submit a patch to that effect, but the following comment at the top of the regionsAwaitingAssignment(HServerAddress, boolean) method indicates that someone thought this wasn't necessary: "Note that no synchronization on regionsInTransition is needed because the only caller (assignRegions, whose caller is ServerManager.processMsgs) owns the monitor for RegionManager"

> CME in master regionsAwaitingAssignment
> ---------------------------------------
>
>                 Key: HBASE-2169
>                 URL: https://issues.apache.org/jira/browse/HBASE-2169
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.3
>            Reporter: stack
>
> Got this in a regionserver log:
> {code}
>  3970 java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException
>  3971   at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
>  3972   at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
>  3973   at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:415)
>  3974   at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:196)
>  3975   at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:482)
>  3976   at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:414)
>  3977   at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:323)
>  3978   at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:724)
>  3979   at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>  3980   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  3981   at java.lang.reflect.Method.invoke(Method.java:597)
>  3982   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:652)
>  3983   at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>  3984 
>  3985   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  3986   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  3987   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  3988   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  3989   at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  3990   at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  3991   at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  3992   at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:574)
>  3993   at java.lang.Thread.run(Thread.java:619)
> {code}
> The regionserver was crashing and burning at the time but still.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.