You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/12/29 23:35:44 UTC
[jira] Created: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
---------------------------------------------------------------------------------------------------------------
Key: HBASE-1097
URL: https://issues.apache.org/jira/browse/HBASE-1097
Project: Hadoop HBase
Issue Type: Bug
Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
Reporter: stack
Fix For: 0.19.0
Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659704#action_12659704 ]
Andrew Purtell commented on HBASE-1097:
---------------------------------------
I'm running with this patch applied now.
> SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1097
> URL: https://issues.apache.org/jira/browse/HBASE-1097
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
> Reporter: stack
> Fix For: 0.19.0
>
> Attachments: mf.patch
>
>
> Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659829#action_12659829 ]
Andrew Purtell commented on HBASE-1097:
---------------------------------------
+1 for commit.
As you might expect, heaps on our HRS are running 40-60% smaller than before. :-) Great work Stack, bringing MapFile and SequenceFile local was a great move.
> SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1097
> URL: https://issues.apache.org/jira/browse/HBASE-1097
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
> Reporter: stack
> Fix For: 0.19.0
>
> Attachments: mf.patch
>
>
> Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659860#action_12659860 ]
stack commented on HBASE-1097:
------------------------------
Good. Let me make sure there are no migration implications before committing above patch.
> SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1097
> URL: https://issues.apache.org/jira/browse/HBASE-1097
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
> Reporter: stack
> Fix For: 0.19.0
>
> Attachments: mf.patch
>
>
> Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1097:
-------------------------
Attachment: mf.patch
Bring down MapFile, SequenceFile and DataOutputBuffer. Fix DOB reset so we put in place initial buffer on invocation. Fix is like HADOOP-4802.
> SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1097
> URL: https://issues.apache.org/jira/browse/HBASE-1097
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
> Reporter: stack
> Fix For: 0.19.0
>
> Attachments: mf.patch
>
>
> Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659883#action_12659883 ]
stack commented on HBASE-1097:
------------------------------
Thanks for testing Andrew.
> SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1097
> URL: https://issues.apache.org/jira/browse/HBASE-1097
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: mf.patch
>
>
> Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-1097:
----------------------------
Assignee: stack
> SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1097
> URL: https://issues.apache.org/jira/browse/HBASE-1097
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: mf.patch
>
>
> Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1097) SequenceFile.Reader keeps around
buffer whose size is that of largest item read -> results in lots of dead
heap
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1097.
--------------------------
Resolution: Fixed
Release Note: Address OOMEing in presence of big cells -- MBs -- when there is scanning going on
Hadoop Flags: [Reviewed]
No apparent migration issues.
> SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1097
> URL: https://issues.apache.org/jira/browse/HBASE-1097
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
> Reporter: stack
> Fix For: 0.19.0
>
> Attachments: mf.patch
>
>
> Andrew is OOMEing again. Looking at some of his heaps, I can count Reader with DataOutputBuffers of ~600MB in a 2G heap. Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it an internal Buffer class which is based on ByteArrayOutputStream. Reset of the DOB eventually goes through to the BAOS reset. This just sets the position. It keeps the buffer sized to whatever it grew to last time this BAOS was used (Figuring this was a little complicated by the fact that DOB does some fancy footwork in a reset override to avoid copies).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.