[jira] [Resolved] (LUCENE-6779) Reduce memory allocated by CompressingStoredFieldsWriter to write large strings

["Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shalin Shekhar Mangar resolved LUCENE-6779.
-------------------------------------------
       Resolution: Fixed
         Assignee: Shalin Shekhar Mangar
    Fix Version/s: 5.4
                   Trunk

Thanks for the reviews Dawid and Robert!

> Reduce memory allocated by CompressingStoredFieldsWriter to write large strings
> -------------------------------------------------------------------------------
>
>                 Key: LUCENE-6779
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6779
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/codecs
>            Reporter: Shalin Shekhar Mangar
>            Assignee: Shalin Shekhar Mangar
>             Fix For: Trunk, 5.4
>
>         Attachments: LUCENE-6779.patch, LUCENE-6779.patch, LUCENE-6779.patch, LUCENE-6779.patch, LUCENE-6779_alt.patch
>
>
> In SOLR-7927, I am trying to reduce the memory required to index very large documents (between 10 to 100MB) and one of the places which allocate a lot of heap is the UTF8 encoding in CompressingStoredFieldsWriter. The same problem existed in JavaBinCodec and we reduced its memory allocation by falling back to a double pass approach in SOLR-7971 when the utf8 size of the string is greater than 64KB.
> I propose to make the same changes to CompressingStoredFieldsWriter as we made to JavaBinCodec in SOLR-7971.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org