You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by "Ortwin Glück (JIRA)" <ji...@apache.org> on 2006/08/09 09:50:13 UTC
[jira] Created: (SANDBOX-166) Improve memory use
Improve memory use
------------------
Key: SANDBOX-166
URL: http://issues.apache.org/jira/browse/SANDBOX-166
Project: Commons Sandbox
Issue Type: Improvement
Components: CSV
Affects Versions: Nightly Builds
Reporter: Ortwin Glück
Attachments: profile.png
The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Updated: (SANDBOX-166) Improve memory use
Posted by "Ortwin Glück (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SANDBOX-166?page=all ]
Ortwin Glück updated SANDBOX-166:
---------------------------------
Attachment: patch.txt
This patch reduces the number of object allocations drastically.
> Improve memory use
> ------------------
>
> Key: SANDBOX-166
> URL: http://issues.apache.org/jira/browse/SANDBOX-166
> Project: Commons Sandbox
> Issue Type: Improvement
> Components: CSV
> Affects Versions: Nightly Builds
> Reporter: Ortwin Glück
> Attachments: patch.txt, profile.png
>
>
> The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (SANDBOX-166) Improve memory use
Posted by "Ortwin Glück (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SANDBOX-166?page=comments#action_12426905 ]
Ortwin Glück commented on SANDBOX-166:
--------------------------------------
The most important optimization is to reuse Token (and their StringBuffer) objects.
CSV files usually contain the same number of columns throughout the file. The parser should adapt itself dynamically after the first line and size its internal arrays correctly. Also the columns have maximum lengths. The parser should adapt itself dynamically and size it's StringBuffers correctly.
Because of JDK 1.3 compatibility there is StringBuffer.append(StringBuffer.toString()) which copies data twice. Using a better character buffer can alleviate the problem.
CSVParser:
getLine(): String[0] is immutable and should be a constant. Token objects should be reused!
nextToken(): reuse intermediate StringBuffer wsBuf. Don't create a new instance on every call
simpleTokenLexer(): reuse intermediate StringBuffer wsBuf. Don't create a new instance on every call
I'll attach a patch that addresses those issues shortly.
> Improve memory use
> ------------------
>
> Key: SANDBOX-166
> URL: http://issues.apache.org/jira/browse/SANDBOX-166
> Project: Commons Sandbox
> Issue Type: Improvement
> Components: CSV
> Affects Versions: Nightly Builds
> Reporter: Ortwin Glück
> Attachments: profile.png
>
>
> The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (SANDBOX-166) Improve memory use
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SANDBOX-166?page=comments#action_12427099 ]
Henri Yandell commented on SANDBOX-166:
---------------------------------------
Looks good to me. I think we'll want to attack the CharBuffer class for bugs a bit as it's at the heart of the system, and it might be valuable for Lang (or there might be something already in there that does the same thing); but the looking at the patch all the changes look good.
Do you want to go ahead and do the commits Ortwin?
You'll find that I accidentally committed the CsvUtils class before realising that I really should be leaving it for you to do the commit - sorry about that.
> Improve memory use
> ------------------
>
> Key: SANDBOX-166
> URL: http://issues.apache.org/jira/browse/SANDBOX-166
> Project: Commons Sandbox
> Issue Type: Improvement
> Components: CSV
> Affects Versions: Nightly Builds
> Reporter: Ortwin Glück
> Attachments: optimized.png, patch.txt, profile.png
>
>
> The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Updated: (SANDBOX-166) Improve memory use
Posted by "Ortwin Glück (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SANDBOX-166?page=all ]
Ortwin Glück updated SANDBOX-166:
---------------------------------
Attachment: optimized.png
These are the figures parsing the same file as before but with the patch applied.
The file contained 47632 rows of 10 columns each. The total file size is around 4MB.
You can clearly see:
- char[] are now down to the number of fields in the file: 476.741 now vs. 1.620.351 befor
- A single CSVParser$Token is now reused: 1 now vs. 476.315 before
- The intermediate String/StringBuffer objects are not reported, but they basically are the same number as char[] and now 4 CharBuffer instances are used only.
> Improve memory use
> ------------------
>
> Key: SANDBOX-166
> URL: http://issues.apache.org/jira/browse/SANDBOX-166
> Project: Commons Sandbox
> Issue Type: Improvement
> Components: CSV
> Affects Versions: Nightly Builds
> Reporter: Ortwin Glück
> Attachments: optimized.png, patch.txt, profile.png
>
>
> The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (SANDBOX-166) Improve memory use
Posted by "Ortwin Glück (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SANDBOX-166?page=comments#action_12427212 ]
Ortwin Glück commented on SANDBOX-166:
--------------------------------------
As to "CharBuffer ... might be valuable for Lang":
If anyone wants it he may copy it over to Lang. Maybe the CharArrayBuffer / ByteArrayBuffer from the HttpComponents projects is a better choice though.
What must not happen is, that CSV uses the CharBuffer class from Lang! This implementation is as you said "at the heart of the system" and it is important that it fits the CSV needs 100%. Especially the linear and minimalistic growing characteristics (as opposed to the exponential growing of a StringBuffer) may not be what anyone else wants. So please keep this CSV local copy, so it can be changed as needed.
> Improve memory use
> ------------------
>
> Key: SANDBOX-166
> URL: http://issues.apache.org/jira/browse/SANDBOX-166
> Project: Commons Sandbox
> Issue Type: Improvement
> Components: CSV
> Affects Versions: Nightly Builds
> Reporter: Ortwin Glück
> Assigned To: Ortwin Glück
> Fix For: Nightly Builds
>
> Attachments: optimized.png, patch.txt, profile.png
>
>
> The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Resolved: (SANDBOX-166) Improve memory use
Posted by "Ortwin Glück (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SANDBOX-166?page=all ]
Ortwin Glück resolved SANDBOX-166.
----------------------------------
Fix Version/s: Nightly Builds
Resolution: Fixed
Assignee: Ortwin Glück
Patch committed.
I have added some more test coverage for the CharBuffer class, recycling some of Oleg's test code for a related class in the HttpComponents project. Now the CharBuffer handles null appends safely.
> Improve memory use
> ------------------
>
> Key: SANDBOX-166
> URL: http://issues.apache.org/jira/browse/SANDBOX-166
> Project: Commons Sandbox
> Issue Type: Improvement
> Components: CSV
> Affects Versions: Nightly Builds
> Reporter: Ortwin Glück
> Assigned To: Ortwin Glück
> Fix For: Nightly Builds
>
> Attachments: optimized.png, patch.txt, profile.png
>
>
> The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (SANDBOX-166) Improve memory use
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SANDBOX-166?page=comments#action_12426973 ]
Henri Yandell commented on SANDBOX-166:
---------------------------------------
Very cool :) Thanks Ortwin.
I'll take a look tonight (hopefully).
> Improve memory use
> ------------------
>
> Key: SANDBOX-166
> URL: http://issues.apache.org/jira/browse/SANDBOX-166
> Project: Commons Sandbox
> Issue Type: Improvement
> Components: CSV
> Affects Versions: Nightly Builds
> Reporter: Ortwin Glück
> Attachments: optimized.png, patch.txt, profile.png
>
>
> The parser is currently a real memory burner. I fed it a 4MB CSV file and ran the TPTP profiler. As you can see the parser creates around 100MB of garbage whereas it could (in really optimized) use around 4MBs. Such figures are not acceptable within a server environment. Please attach insights and patches to this issue report.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org