[jira] [Updated] (SOLR-18237) Collection state deserialization duplicates content for no good reason

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SOLR-18237:
----------------------------------
    Labels: pull-request-available  (was: )

> Collection state deserialization duplicates content for no good reason
> ----------------------------------------------------------------------
>
>                 Key: SOLR-18237
>                 URL: https://issues.apache.org/jira/browse/SOLR-18237
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 9.10, 10.0
>            Reporter: Pierre Salagnac
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When deserializing collection state from Zookeeper, we create large char arrays to convert UTF8 data into Java strings.
> In Utils.java
> {code:java}
> CharArr chars = new CharArr();
> ByteUtils.UTF8toUTF16(utf8, offset, length, chars);
> {code}
> This consumes a lot of memory several threads concurrently deserialize a big collection. We had OOM error because of this. This is a corner case since the OOM were raised with a non full heap (~50%). My understanding is this is because big arrays are allocated directly in the _Old_ space, and we end with fragmented heap where one more big array does not fit, even if the heap is not full.
>  
> This may easily replaced by a reader that does the UTF8 decoding on the fly, with a much much smaller buffer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org