You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@harmony.apache.org by "deven you (JIRA)" <ji...@apache.org> on 2010/03/12 08:33:27 UTC
[jira] Created: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
[classlib][luni] Scanner will exhaust all heap memory when parse a large file
-----------------------------------------------------------------------------
Key: HARMONY-6470
URL: https://issues.apache.org/jira/browse/HARMONY-6470
Project: Harmony
Issue Type: Bug
Components: Classlib
Affects Versions: 5.0M14
Reporter: deven you
Fix For: 5.0M14
Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "deven you (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
deven you updated HARMONY-6470:
-------------------------------
Attachment: HARMONY-6470_v3.diff
Hi Tim,
Following your comments, I have modified the test case and now it can only use memory to simulate file reading. Please check it. Thanks a lot!
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M14
> Reporter: deven you
> Fix For: 5.0M14
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878948#action_12878948 ]
Hudson commented on HARMONY-6470:
---------------------------------
Integrated in Harmony-1.5-head-linux-x86_64 #855 (See [http://hudson.zones.apache.org/hudson/job/Harmony-1.5-head-linux-x86_64/855/])
Apply patch for HARMONY-6470 ([classlib][luni] Scanner will exhaust all heap memory when parse a large file)
Includes a (relatively) long running test case.
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M15
> Reporter: deven you
> Assignee: Tim Ellison
> Fix For: 5.0M15
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "Tim Ellison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Ellison resolved HARMONY-6470.
----------------------------------
Assignee: Tim Ellison
Resolution: Fixed
Thanks Deven,
Patch applied to LUNI module at repo revision r954817.
Please check it was applied as you expected.
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M15
> Reporter: deven you
> Assignee: Tim Ellison
> Fix For: 5.0M15
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "Tim Ellison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844530#action_12844530 ]
Tim Ellison commented on HARMONY-6470:
--------------------------------------
Deven,
Thanks for the patch.
Rather than create a temp file of 8 << 21 (16777216) * 11 bytes and then reading it back into the scanner, do you think it would be better to have a Reader that returns the bytes in memory? I think the test will be more robust and faster that way.
Something like this:
public class MyReader extends Reader {
static final char[] CONTENT = "Large file!\n".toCharArray();
@Override
public void close() throws IOException {
}
@Override
public int read(char[] buf, int offset, int count) throws IOException {
for (int i = offset, j = 0; j < count; i++, j++) {
buf[i] = CONTENT[j % CONTENT.length];
}
return count;
}
}
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M14
> Reporter: deven you
> Fix For: 5.0M14
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "Mark Hindess (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Hindess updated HARMONY-6470:
----------------------------------
Affects Version/s: 5.0M15
(was: 5.0M14)
Deferring until next release.
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M15
> Reporter: deven you
> Fix For: 5.0M14
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "Mark Hindess (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Hindess updated HARMONY-6470:
----------------------------------
Fix Version/s: 5.0M15
(was: 5.0M14)
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M15
> Reporter: deven you
> Fix For: 5.0M15
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "deven you (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
deven you updated HARMONY-6470:
-------------------------------
Attachment: HARMONY-6470_v2.diff
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M14
> Reporter: deven you
> Fix For: 5.0M14
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "deven you (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
deven you updated HARMONY-6470:
-------------------------------
Attachment: HARMONY-6470.diff
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M14
> Reporter: deven you
> Fix For: 5.0M14
>
> Attachments: HARMONY-6470.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878960#action_12878960 ]
Hudson commented on HARMONY-6470:
---------------------------------
Integrated in Harmony-select-1.5-head-linux-x86_64 #35 (See [http://hudson.zones.apache.org/hudson/job/Harmony-select-1.5-head-linux-x86_64/35/])
Apply patch for HARMONY-6470 ([classlib][luni] Scanner will exhaust all heap memory when parse a large file)
Includes a (relatively) long running test case.
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M15
> Reporter: deven you
> Assignee: Tim Ellison
> Fix For: 5.0M15
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "deven you (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845667#action_12845667 ]
deven you commented on HARMONY-6470:
------------------------------------
Since this path is a little complex, let me give some some explains.
The main modification is in Scanner.expandBuffer(), originally it simply uses a new larger char buffer to replace the old one, and copy all from the old one, so all properties (charbuffer length, capacity, limit and Scanner's findStartIndex, preStartIndex, cacheHasNextIndex and etc. )related to the old buffer won't change.
However in this patch, Scanner will try to discard the contends which are not used any longer in the current char buffer(which also may be replaced by a larger one) and only keep that will used for next Scanner.hasNext() or Scanner.next(), so the properties related to the char buffer also need be changed.
The next point is this patch add 2 instance variables tokenStart and tokenEnd which were just 2 local vars in Scanner.setTokenRegion(), since the char buffer's properties now may be changed at any time, now methods(like findPreDelimiter() and findPostDelimiter() etc.) which track the start index and end index need change this 2 vars at any moment.
if any one would like to try this patch? Thanks a lot!
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M14
> Reporter: deven you
> Fix For: 5.0M14
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (HARMONY-6470) [classlib][luni] Scanner will exhaust
all heap memory when parse a large file
Posted by "deven you (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
deven you closed HARMONY-6470.
------------------------------
This jira has been verified, Thanks!
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M15
> Reporter: deven you
> Assignee: Tim Ellison
> Fix For: 5.0M15
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff, HARMONY-6470_v3.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HARMONY-6470) [classlib][luni] Scanner will
exhaust all heap memory when parse a large file
Posted by "Tim Ellison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844547#action_12844547 ]
Tim Ellison commented on HARMONY-6470:
--------------------------------------
Sorry Deven, of course if you want the Reader to reach an end, you'll have to keep track of "char's served" too, and return -1 when you reach the limit -- I'll leave that as an exercise :-)
> [classlib][luni] Scanner will exhaust all heap memory when parse a large file
> -----------------------------------------------------------------------------
>
> Key: HARMONY-6470
> URL: https://issues.apache.org/jira/browse/HARMONY-6470
> Project: Harmony
> Issue Type: Bug
> Components: Classlib
> Affects Versions: 5.0M14
> Reporter: deven you
> Fix For: 5.0M14
>
> Attachments: HARMONY-6470.diff, HARMONY-6470_v2.diff
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Please see the attached file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.