You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/06/17 12:07:00 UTC
[jira] [Commented] (SOLR-18237) Collection state deserialization duplicates content for no good reason
[ https://issues.apache.org/jira/browse/SOLR-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089644#comment-18089644 ]
ASF subversion and git services commented on SOLR-18237:
--------------------------------------------------------
Commit 2c8cb3235e16f3b0c45df50c3ceb218d093095d4 in solr's branch refs/heads/main from Pierre Salagnac
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=2c8cb3235e1 ]
SOLR-18237: Do not copy data before noggit parsing (#4415)
This removes the char[] array buffer when parsing collection states and other Json data, since allocation of this large buffer is a cause of OOM errors.
Instead, the UTF8 decoding is done on the fly using JVM tolling.
> Collection state deserialization duplicates content for no good reason
> ----------------------------------------------------------------------
>
> Key: SOLR-18237
> URL: https://issues.apache.org/jira/browse/SOLR-18237
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10, 10.0
> Reporter: Pierre Salagnac
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When deserializing collection state from Zookeeper, we create large char arrays to convert UTF8 data into Java strings.
> In Utils.java
> {code:java}
> CharArr chars = new CharArr();
> ByteUtils.UTF8toUTF16(utf8, offset, length, chars);
> {code}
> This consumes a lot of memory several threads concurrently deserialize a big collection. We had OOM error because of this. This is a corner case since the OOM were raised with a non full heap (~50%). My understanding is this is because big arrays are allocated directly in the _Old_ space, and we end with fragmented heap where one more big array does not fit, even if the heap is not full.
>
> This may easily replaced by a reader that does the UTF8 decoding on the fly, with a much much smaller buffer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org