You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Lars George (JIRA)" <ji...@apache.org> on 2009/09/02 12:40:32 UTC
[jira] Created: (HBASE-1810) ConcurrentModificationException in
region assignment
ConcurrentModificationException in region assignment
----------------------------------------------------
Key: HBASE-1810
URL: https://issues.apache.org/jira/browse/HBASE-1810
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.0
Reporter: Lars George
Fix For: 0.20.1
{code}
2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
java.io.IOException: java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
{code}
and
{code}
2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
java.io.IOException: java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
{code}
As discussed with Mathias on IRC:
<hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
<hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
<hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1810) ConcurrentModificationException in
region assignment
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1810:
----------------------------------
Fix Version/s: (was: 0.20.1)
0.20.0
> ConcurrentModificationException in region assignment
> ----------------------------------------------------
>
> Key: HBASE-1810
> URL: https://issues.apache.org/jira/browse/HBASE-1810
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Lars George
> Assignee: Mathias Herberts
> Fix For: 0.20.0
>
> Attachments: HBASE-1810.patch
>
>
> {code}
> 2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> and
> {code}
> 2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> As discussed with Mathias on IRC:
> <hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
> <hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
> <hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1810) ConcurrentModificationException in
region assignment
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1810.
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
I looked at java src. Internally, Collections.synchronizedSet, etc., use the same mutex as is used when you do synchronized(object) where object is the synchronizedSet, etc.
I reviewed the rest of RegionManager. regionsInTransition seems to be safely synchronized with Mathias's patch.
Committed branch and trunk. Thanks for the patch Mathias.
> ConcurrentModificationException in region assignment
> ----------------------------------------------------
>
> Key: HBASE-1810
> URL: https://issues.apache.org/jira/browse/HBASE-1810
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Lars George
> Assignee: Mathias Herberts
> Fix For: 0.20.0
>
> Attachments: HBASE-1810.patch
>
>
> {code}
> 2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> and
> {code}
> 2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> As discussed with Mathias on IRC:
> <hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
> <hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
> <hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1810) ConcurrentModificationException in
region assignment
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars George reassigned HBASE-1810:
----------------------------------
Assignee: Mathias Herberts
> ConcurrentModificationException in region assignment
> ----------------------------------------------------
>
> Key: HBASE-1810
> URL: https://issues.apache.org/jira/browse/HBASE-1810
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Lars George
> Assignee: Mathias Herberts
> Fix For: 0.20.1
>
> Attachments: HBASE-1810.patch
>
>
> {code}
> 2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> and
> {code}
> 2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> As discussed with Mathias on IRC:
> <hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
> <hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
> <hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1810) ConcurrentModificationException in
region assignment
Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mathias Herberts updated HBASE-1810:
------------------------------------
Attachment: HBASE-1810.patch
Enclosed iteration of regionsInTransition in 'synchronized(regionsInTransition)' block.
> ConcurrentModificationException in region assignment
> ----------------------------------------------------
>
> Key: HBASE-1810
> URL: https://issues.apache.org/jira/browse/HBASE-1810
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Lars George
> Fix For: 0.20.1
>
> Attachments: HBASE-1810.patch
>
>
> {code}
> 2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> and
> {code}
> 2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> As discussed with Mathias on IRC:
> <hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
> <hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
> <hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1810) ConcurrentModificationException in
region assignment
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750455#action_12750455 ]
stack commented on HBASE-1810:
------------------------------
hmm.... Looking at the regionsInTransistion set, I see that its synchronized sometimes but not always (I'm talking about before Mathias's patch is applied). Mathias's patch would seem to make sense.... Let me check java src to ensure that same monitor is effected when you do synchronized(regionsInTransistion) as is used when you do Collections.synchronizedSortedMap get /put
> ConcurrentModificationException in region assignment
> ----------------------------------------------------
>
> Key: HBASE-1810
> URL: https://issues.apache.org/jira/browse/HBASE-1810
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Lars George
> Assignee: Mathias Herberts
> Fix For: 0.20.1
>
> Attachments: HBASE-1810.patch
>
>
> {code}
> 2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> and
> {code}
> 2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
> at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> {code}
> As discussed with Mathias on IRC:
> <hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
> <hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
> <hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.