You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "countmdm (via GitHub)" <gi...@apache.org> on 2026/01/31 01:09:39 UTC
[PR] Lazily allocate ByteArrayDataInputs in SegmentTermsEnumFrame [lucene]
countmdm opened a new pull request, #15637:
URL: https://github.com/apache/lucene/pull/15637
### Description
In my previous PR https://github.com/apache/lucene/pull/15618 I changed SegementTermsEnumFrame to lazily initialize byte[] arrays that it manages. After this change, we measured memory allocation of one of apps at LinkedIn that use Lucene and found that (a) its memory allocation rate went down by ~25%, as expected, but (b) still ~10% of that reduced memory allocation was due to ByteArrayDataInput objects eagerly allocated by SegmentTermsEnumFrame code. Previously we determined that a high percentage of SegmentTermsEnumFrame objects are allocated but not used by our app. Thus, the same percentage of ByteArrayDataInputs managed by STEF are wasted. This PR addresses that problem by switching to lazy allocation of these objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Lazily allocate ByteArrayDataInputs in SegmentTermsEnumFrame [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss merged PR #15637:
URL: https://github.com/apache/lucene/pull/15637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Lazily allocate ByteArrayDataInputs in SegmentTermsEnumFrame [lucene]
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #15637:
URL: https://github.com/apache/lucene/pull/15637#issuecomment-3861626181
Merged to main and branch_10x. Please add an entry to CHANGES.txt under the right branch, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Lazily allocate ByteArrayDataInputs in SegmentTermsEnumFrame [lucene]
Posted by "zhaih (via GitHub)" <gi...@apache.org>.
zhaih commented on PR #15637:
URL: https://github.com/apache/lucene/pull/15637#issuecomment-3858456497
> Can somebody advise definitively
Can you follow your previous PR #15618? There's 2 files changed there and only one here, so I think Mike is suggesting modifying the other file (which is CHANGES.txt) to add a change entry there :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Lazily allocate ByteArrayDataInputs in SegmentTermsEnumFrame [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #15637:
URL: https://github.com/apache/lucene/pull/15637#issuecomment-3859558128
+1 to add an entry to `CHANGES.txt` for this exciting allocation reduction. On upgrade Lucene users would like to see such optimizations are happening :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Lazily allocate ByteArrayDataInputs in SegmentTermsEnumFrame [lucene]
Posted by "countmdm (via GitHub)" <gi...@apache.org>.
countmdm commented on PR #15637:
URL: https://github.com/apache/lucene/pull/15637#issuecomment-3842784880
> Thanks! LGTM: I only wonder if we should also mention this in CHANGES.txt?
I don't know this project's rules about that. Can somebody advise definitively, or otherwise merge this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Lazily allocate ByteArrayDataInputs in SegmentTermsEnumFrame [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #15637:
URL: https://github.com/apache/lucene/pull/15637#issuecomment-3841447755
Thanks! LGTM: I only wonder if we should also mention this in CHANGES.txt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org