You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2010/04/29 22:01:57 UTC
[jira] Created: (LUCENE-2422) don't reuse byte[] in
IndexInput/Output for read/writeString
don't reuse byte[] in IndexInput/Output for read/writeString
------------------------------------------------------------
Key: LUCENE-2422
URL: https://issues.apache.org/jira/browse/LUCENE-2422
Project: Lucene - Java
Issue Type: Bug
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: 2.9.3, 3.0.2, 3.1, 4.0.0
IndexInput now holds a private "byte[] bytes", which it re-uses for reading strings. Likewise, IndexOutput holds a UTF8Result (which holds "byte[] bytes"), re-used for writing strings.
These are both dangerous, since on reading or writing immense strings, we never free this storage.
We don't use read/writeString in very perf sensitive parts of the code, so, I think we should not reuse the byte[] at all.
I think this is likely the cause of the recent "IndexWriter and memory usage" thread, started by Ross Woolf on java-user@.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2422) don't reuse byte[] in
IndexInput/Output for read/writeString
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-2422.
----------------------------------------
Resolution: Fixed
> don't reuse byte[] in IndexInput/Output for read/writeString
> ------------------------------------------------------------
>
> Key: LUCENE-2422
> URL: https://issues.apache.org/jira/browse/LUCENE-2422
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.9.3, 3.0.2, 3.1, 4.0
>
> Attachments: LUCENE-2422.patch
>
>
> IndexInput now holds a private "byte[] bytes", which it re-uses for reading strings. Likewise, IndexOutput holds a UTF8Result (which holds "byte[] bytes"), re-used for writing strings.
> These are both dangerous, since on reading or writing immense strings, we never free this storage.
> We don't use read/writeString in very perf sensitive parts of the code, so, I think we should not reuse the byte[] at all.
> I think this is likely the cause of the recent "IndexWriter and memory usage" thread, started by Ross Woolf on java-user@.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2422) don't reuse byte[] in
IndexInput/Output for read/writeString
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862606#action_12862606 ]
Michael McCandless commented on LUCENE-2422:
--------------------------------------------
bq. Mike - this patch is against an old revision?
Yes, sorry, the patch applies to 2.9.x. I think we should fix it in 2.9.x (and all branches after -- 3.0, trunk).
In trunk these reused byte[] have been moved to DataInput/Output.
> don't reuse byte[] in IndexInput/Output for read/writeString
> ------------------------------------------------------------
>
> Key: LUCENE-2422
> URL: https://issues.apache.org/jira/browse/LUCENE-2422
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.9.3, 3.0.2, 3.1, 4.0.0
>
> Attachments: LUCENE-2422.patch
>
>
> IndexInput now holds a private "byte[] bytes", which it re-uses for reading strings. Likewise, IndexOutput holds a UTF8Result (which holds "byte[] bytes"), re-used for writing strings.
> These are both dangerous, since on reading or writing immense strings, we never free this storage.
> We don't use read/writeString in very perf sensitive parts of the code, so, I think we should not reuse the byte[] at all.
> I think this is likely the cause of the recent "IndexWriter and memory usage" thread, started by Ross Woolf on java-user@.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2422) don't reuse byte[] in
IndexInput/Output for read/writeString
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2422:
---------------------------------------
Attachment: LUCENE-2422.patch
> don't reuse byte[] in IndexInput/Output for read/writeString
> ------------------------------------------------------------
>
> Key: LUCENE-2422
> URL: https://issues.apache.org/jira/browse/LUCENE-2422
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.9.3, 3.0.2, 3.1, 4.0.0
>
> Attachments: LUCENE-2422.patch
>
>
> IndexInput now holds a private "byte[] bytes", which it re-uses for reading strings. Likewise, IndexOutput holds a UTF8Result (which holds "byte[] bytes"), re-used for writing strings.
> These are both dangerous, since on reading or writing immense strings, we never free this storage.
> We don't use read/writeString in very perf sensitive parts of the code, so, I think we should not reuse the byte[] at all.
> I think this is likely the cause of the recent "IndexWriter and memory usage" thread, started by Ross Woolf on java-user@.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2422) don't reuse byte[] in
IndexInput/Output for read/writeString
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862530#action_12862530 ]
Shai Erera commented on LUCENE-2422:
------------------------------------
Mike - this patch is against an old revision? I'm up to the latest and IndexInput/Output don't include any field, just abstract methods. This seems to be relevant to 3.0.1 (and before?) If so, where does this need to be fixed post 3.0.1?
> don't reuse byte[] in IndexInput/Output for read/writeString
> ------------------------------------------------------------
>
> Key: LUCENE-2422
> URL: https://issues.apache.org/jira/browse/LUCENE-2422
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.9.3, 3.0.2, 3.1, 4.0.0
>
> Attachments: LUCENE-2422.patch
>
>
> IndexInput now holds a private "byte[] bytes", which it re-uses for reading strings. Likewise, IndexOutput holds a UTF8Result (which holds "byte[] bytes"), re-used for writing strings.
> These are both dangerous, since on reading or writing immense strings, we never free this storage.
> We don't use read/writeString in very perf sensitive parts of the code, so, I think we should not reuse the byte[] at all.
> I think this is likely the cause of the recent "IndexWriter and memory usage" thread, started by Ross Woolf on java-user@.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org