You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Adriano Crestani (JIRA)" <ji...@apache.org> on 2010/07/23 06:55:49 UTC
[jira] Created: (LUCENE-2556) CharTermAttribute cloning memory
consumption
CharTermAttribute cloning memory consumption
--------------------------------------------
Key: LUCENE-2556
URL: https://issues.apache.org/jira/browse/LUCENE-2556
Project: Lucene - Java
Issue Type: Improvement
Components: Analysis
Affects Versions: 3.0.2
Reporter: Adriano Crestani
Priority: Minor
Fix For: 3.1
The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler reassigned LUCENE-2556:
-------------------------------------
Assignee: Uwe Schindler
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Adriano Crestani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adriano Crestani updated LUCENE-2556:
-------------------------------------
Attachment: lucene_2556_adriano_crestani_07_24_2010.patch
So, if you guys think my thoughts above are OK, this is a simple patch with code that creates a new internal buffer strictly equals to the term length
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2556) (Char)TermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2556:
----------------------------------
Attachment: LUCENE-2556-3.0.patch
Patch for 3.0 for reference
> (Char)TermAttribute cloning memory consumption
> ----------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556-3.0.patch, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a (Char)TermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922596#action_12922596 ]
Uwe Schindler commented on LUCENE-2556:
---------------------------------------
There is also no problem with reusing the cloned attribute, as ArrayUtils grows the array correctly also in that case. So you can still apend stuff as usual, with a *small* perf penalty maybe.
I think I will commit this later this evening to trunk and 3.x. Merging is little harder because of sophisticated backwards layer in 3.x. :-)
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Adriano Crestani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922559#action_12922559 ]
Adriano Crestani commented on LUCENE-2556:
------------------------------------------
Robert is right, in the actual use case, a cloned TermAttribute is never modified, it's at max copied back to the TokenStream's TermAttribute (when using TokenStream.restoreState(State)), so the only TermAttribute instance that needs to grow is the TokenStream's.
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2556:
----------------------------------
Attachment: LUCENE-2556.patch
Here the patch, I see no problem with applying it to 3.x and trunk.
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2556) (Char)TermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2556:
----------------------------------
Description: The memory consumption problem with cloning a (Char)TermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z (was: The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z)
Fix Version/s: 3.0.3
2.9.4
Summary: (Char)TermAttribute cloning memory consumption (was: CharTermAttribute cloning memory consumption)
Chaning title
> (Char)TermAttribute cloning memory consumption
> ----------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a (Char)TermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-2556.
-----------------------------------
Resolution: Fixed
Fix Version/s: 4.0
Committed trunk revision: 1024408
Committed 3.x revision: 1024409
Thanks Adriano!
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Adriano Crestani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922753#action_12922753 ]
Adriano Crestani commented on LUCENE-2556:
------------------------------------------
Thank you for committing the patch ;)
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922523#action_12922523 ]
Robert Muir commented on LUCENE-2556:
-------------------------------------
bq. The only thing: It still thin it should use an array size calculated by ArrayUtils.oversize() on the clone?
I don't think we should. the clone might never be used again:
So if the clone is never reused, there is no evidence the array will ever grow (most analysis processes, stemming and folding, etc actually shorten the term text)
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Adriano Crestani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891500#action_12891500 ]
Adriano Crestani commented on LUCENE-2556:
------------------------------------------
I was checking State.clone() method usage and it's just used store the current AttributeSource state for later use, when it's restored by invoking AttributeSource.restoreState(), which only copies the valid chars from the stored state to the current CharTermAttribute object the AttributeSource holds. In the end, I see no reason for stored states (the cloned ones) to hold an internal buffer greater than it needs too, once it will never be actually used/changed by any AttributeSource user.
Please, let me know if I'm missing something.
Thanks,
Adriano Crestani
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922446#action_12922446 ]
Uwe Schindler commented on LUCENE-2556:
---------------------------------------
I have no problem with it.
The only thing: It still thin it should use an array size calculated by ArrayUtils.oversize() on the clone?
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2556) (Char)TermAttribute
cloning memory consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926568#action_12926568 ]
Uwe Schindler edited comment on LUCENE-2556 at 10/30/10 7:37 AM:
-----------------------------------------------------------------
Committed branch 3.0 revision: 1029022
Committed branch 2.9 revision: 1029027
was (Author: thetaphi):
Committed branch 3.0 revision: 1024409
Committed branch 2.9 revision: 1029027
> (Char)TermAttribute cloning memory consumption
> ----------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556-3.0.patch, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a (Char)TermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Adriano Crestani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891494#action_12891494 ]
Adriano Crestani commented on LUCENE-2556:
------------------------------------------
Hi Uwe,
Thanks for the quick reply, your patch looks good enough for me. I just added that optimization part in case somebody complains about poor performance of arrayCopy for long arrays, but I agree with you, mostly terms are short and wouldn't require such optimization.
+1 to also apply the patch to trunk
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2556) (Char)TermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-2556.
-----------------------------------
Resolution: Fixed
Committed branch 3.0 revision: 1024409
Committed branch 2.9 revision: 1029027
> (Char)TermAttribute cloning memory consumption
> ----------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556-3.0.patch, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a (Char)TermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922365#action_12922365 ]
Robert Muir commented on LUCENE-2556:
-------------------------------------
Uwe, whats happening with this issue?
Indexing terms are typically small, I think we should commit Adriano's last patch.
I have problems with clone() being slow with -client on my jvm, I think this is a good improvement.
And, can prevent some memory issues (the original intent of the issue it seems)
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler reopened LUCENE-2556:
-----------------------------------
Backport to 2.9 and 3.0.
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch, lucene_2556_adriano_crestani_07_24_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Adriano Crestani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adriano Crestani updated LUCENE-2556:
-------------------------------------
Attachment: lucene_2556_adriano_crestani_07_23_2010.patch
This patch optimizes the cloning of the CharTermAttributeImpl internal buffer. It keeps using clone() to clone the internal buffer when CharTermAttribute.length() is at least 150 and at least 75% and of the internal buffer length, otherwise, it uses System.arrayCopy(...) to clone it using CharTermAttribute.length() as the new internal buffer size.
It's performing the optimization, because in some scenarios, like cloning long arrays, clone() is usually faster than System.arrayCopy(...).
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, lucene_2556_adriano_crestani_07_23_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891481#action_12891481 ]
Uwe Schindler commented on LUCENE-2556:
---------------------------------------
{quote}
This patch optimizes the cloning of the CharTermAttributeImpl internal buffer. It keeps using clone() to clone the internal buffer when CharTermAttribute.length() is at least 150 and at least 75% and of the internal buffer length, otherwise, it uses System.arrayCopy(...) to clone it using CharTermAttribute.length() as the new internal buffer size.
It's performing the optimization, because in some scenarios, like cloning long arrays, clone() is usually faster than System.arrayCopy(...).
{quote}
Haven't seen your patch yet. I dont know if the two extra calculations rectify the barnching, because terms are mostly short...
If we take your patch, the allocations should in all cases be done with ArrayUtils.oversize() to be consistent with the allocation strategy of the rest of CTA.
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java, LUCENE-2556.patch, lucene_2556_adriano_crestani_07_23_2010.patch
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2556) CharTermAttribute cloning memory
consumption
Posted by "Adriano Crestani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adriano Crestani updated LUCENE-2556:
-------------------------------------
Attachment: CharTermAttributeMemoryConsumptionDemo.java
This java application demonstrates how much memory CharTermAttributeImpl.clone() might consume in some scenarios.
> CharTermAttribute cloning memory consumption
> --------------------------------------------
>
> Key: LUCENE-2556
> URL: https://issues.apache.org/jira/browse/LUCENE-2556
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0.2
> Reporter: Adriano Crestani
> Priority: Minor
> Fix For: 3.1
>
> Attachments: CharTermAttributeMemoryConsumptionDemo.java
>
>
> The memory consumption problem with cloning a CharTermAttributeImpl object was raised on thread http://markmail.org/thread/bybuerugbk5w2u6z
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org