You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/11/11 06:31:44 UTC
[jira] Created: (HBASE-990) NoSuchElementException in
flushSomeRegions
NoSuchElementException in flushSomeRegions
------------------------------------------
Key: HBASE-990
URL: https://issues.apache.org/jira/browse/HBASE-990
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.19.0
>From Billy
{code}
2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
java.io.IOException: java.util.NoSuchElementException
at java.util.TreeMap.key(TreeMap.java:1206)
at java.util.TreeMap.firstKey(TreeMap.java:267)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648504#action_12648504 ]
stack commented on HBASE-990:
-----------------------------
I see. Looks like we succeed the first time through the loop at #229 but if we keep going around without freeing enough memory, eventually we'll empty the Map and get a NSSE. Silly mistake.
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-990.
-------------------------
Resolution: Fixed
Committed.
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648628#action_12648628 ]
Rong-En Fan commented on HBASE-990:
-----------------------------------
Thanks stack, The new patch helps.
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 990.patch
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-990:
------------------------
Attachment: 990.patch
Here is something for you to try Rong-en. Should 'cure' the exception. Next though is to figure why though all regions have flushed, you still do not have enough free memory.
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 990.patch
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648496#action_12648496 ]
stack commented on HBASE-990:
-----------------------------
I'd think the line numbers would at least be different Rong-en? Want to post the stack trace?
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-990.
-------------------------
Resolution: Fixed
Ok. Will try resolving again (Committed).
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 990.patch
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648503#action_12648503 ]
Rong-En Fan commented on HBASE-990:
-----------------------------------
indeed, the line # in flushSomeRegions() is different, I missed it.
java.io.IOException: java.util.NoSuchElementException
at java.util.TreeMap.key(TreeMap.java:1206)
at java.util.TreeMap.firstKey(TreeMap.java:267)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:231)
at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1312)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rong-En Fan reopened HBASE-990:
-------------------------------
I'm still seeing this with default settings regarding memcache / flush size with trunk r718159 (I use hadoop 0.19.0 rc1 instead of the one in the trunk). The stack trace is exactly the same as the one in the description. However, I didn't see the new INFO message in my region server logs. It seems to me that even after flush all online regions, global memory cache is still larger than its low mark.
I'm writing ~2M rows in each run, within 10 runs, each runs has 1~3 errors like this one.
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-990) NoSuchElementException in
flushSomeRegions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-990:
---------------------------
Assignee: stack
> NoSuchElementException in flushSomeRegions
> ------------------------------------------
>
> Key: HBASE-990
> URL: https://issues.apache.org/jira/browse/HBASE-990
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
>
> From Billy
> {code}
> 2008-11-10 18:14:53,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 69 on 60020, call batchUpdates([B@68e97c, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@1b8e8d2) from 192.168.1.201:37077: error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1206)
> at java.util.TreeMap.firstKey(TreeMap.java:267)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:227)
> at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1227)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.