You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Alex Chow (JIRA)" <ji...@apache.org> on 2015/09/24 12:34:04 UTC
[jira] [Comment Edited] (LUCENE-6814) PatternTokenizer should free
heap after it's done
[ https://issues.apache.org/jira/browse/LUCENE-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906171#comment-14906171 ]
Alex Chow edited comment on LUCENE-6814 at 9/24/15 10:33 AM:
-------------------------------------------------------------
Thanks for taking a look at this, [~mikemccand]!
It'd be great if this can get added to 4.10 so elasticsearch 1.x can pull it in too.
I'm curious why there shouldn't there be some trimming in `end()` as well? Or is a `TokenStream` meant to be used only once (no multiple `reset()`, `incrementToken()`, `end()` on the same `TokenStream`)?
Elasticsearch seems to never reinstantiate Tokenizers and just reuses them for each field in an index, though I may be wrong. Or elasticsearch is using TokenStream the wrong way?
was (Author: chow):
Thanks for taking a look at this, [~mikemccand]!
I'm curious why there shouldn't there be some trimming in `end()` as well? Or is a `TokenStream` meant to be used only once (no multiple `reset()`, `incrementToken()`, `end()` on the same `TokenStream`)?
Elasticsearch seems to never reinstantiate Tokenizers and just reuses them for each field in an index, though I may be wrong. Or elasticsearch is using TokenStream the wrong way?
> PatternTokenizer should free heap after it's done
> -------------------------------------------------
>
> Key: LUCENE-6814
> URL: https://issues.apache.org/jira/browse/LUCENE-6814
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: Trunk, 5.4
>
> Attachments: LUCENE-6814.patch, LUCENE-6814.patch
>
>
> Caught by Alex Chow in this Elasticsearch issue: https://github.com/elastic/elasticsearch/issues/13721
> Today, PatternTokenizer reuses a single StringBuilder, but it doesn't free its heap usage after tokenizing is done. We can either stop reusing, or ask it to {{.trimToSize}} when we are done ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org