You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Alex Chow (JIRA)" <ji...@apache.org> on 2015/09/24 23:21:04 UTC
[jira] [Commented] (LUCENE-6814) PatternTokenizer indefinitely
holds heap equal to max document it has ever tokenized
[ https://issues.apache.org/jira/browse/LUCENE-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907054#comment-14907054 ]
Alex Chow commented on LUCENE-6814:
-----------------------------------
bq. Hmm, how many {{PatternTokenizer}} instances do you have? With 24 bulk indexing threads you should (I think?) only have at most 24 instances * 4 MB max should be ~100 MB.
{{PatternTokenizer}} instances also scale with the number of indices (technically shards?) if you use your own analyzer based on a {{PatternAnalyzer}}. Custom analyzers get instantiated per index since they have their own mappings. We have 168 indices, so that's a _lot_. One heap dump I was poking around had 3432 instances, but it'd probably have grown more since some analyzers haven't had to create their {{TokenStreamComponents}} yet.
> PatternTokenizer indefinitely holds heap equal to max document it has ever tokenized
> ------------------------------------------------------------------------------------
>
> Key: LUCENE-6814
> URL: https://issues.apache.org/jira/browse/LUCENE-6814
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: Trunk, 5.4
>
> Attachments: LUCENE-6814.patch, LUCENE-6814.patch
>
>
> Caught by Alex Chow in this Elasticsearch issue: https://github.com/elastic/elasticsearch/issues/13721
> Today, PatternTokenizer reuses a single StringBuilder, but it doesn't free its heap usage after tokenizing is done. We can either stop reusing, or ask it to {{.trimToSize}} when we are done ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org