You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "javanna (via GitHub)" <gi...@apache.org> on 2026/04/30 10:24:53 UTC
[PR] Clarify intent of Accountable#ramBytesUsed [lucene]
javanna opened a new pull request, #16000:
URL: https://github.com/apache/lucene/pull/16000
The memory tracking is meant for heap memory and not off-heap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna merged PR #16000:
URL: https://github.com/apache/lucene/pull/16000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #16000:
URL: https://github.com/apache/lucene/pull/16000#issuecomment-4351663095
If we agree on this, a possible follow-up or alternative change is to rename the method, but such change has a wide blast radius as opposed to just adjusting docs. On the other hand this class is marked internal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #16000:
URL: https://github.com/apache/lucene/pull/16000#issuecomment-4476816432
Thanks for taking a look @benwtrent and @jainankitk , much appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #16000:
URL: https://github.com/apache/lucene/pull/16000#issuecomment-4351911220
> That said, there is nothing preventing somebody from using this interface in their own way and making it refer to off heap as well.
Agreeed.
The reason why I thought it's good to clarify is that there's data structures that may use heap and off-heap memory, in which case it is not always clear what `ramBytesUsed` should report on. This can lead to inconsistencies and difficulties interpreting sizing.
A concrete example: completion FSTs. They moved a few versions ago from on-heap to off-heap. Their accounting reflects that. Some folks may wonder if this is expected, or whether the reporting is inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #16000:
URL: https://github.com/apache/lucene/pull/16000#issuecomment-4358446611
> Do we have other examples where something is moved off-heap?
One that comes to mind is `KnnVectorsReader` , which does not implement `Accountable` but has a specific method to retrieve the off-heap size (`getOffHeapByteSize`). That seems like a conscious choice.
There may be more, I was hoping that other devs would bring those up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #16000:
URL: https://github.com/apache/lucene/pull/16000#issuecomment-4355157434
Do we have other examples where something is moved off-heap? Does off-heap always mean "accessed via IO" (the FST example), or do we ever do something like `malloc(...)` as "off-heap"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #16000:
URL: https://github.com/apache/lucene/pull/16000#issuecomment-4386250404
Thanks for the feedback so far! I plan on merging this soon if there's no objections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Clarify intent of Accountable#ramBytesUsed [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #16000:
URL: https://github.com/apache/lucene/pull/16000#issuecomment-4405383587
I did some more research and found maybe some discrepancy in `ByteBuffersDataInput` and `ByteBuffersDataOutput` (and indirectly `PrefixCodedTerm` as well as `NRTCachingDirectory`):
these are allocator-agnostic and would count off-heap if constructed with a direct allocator.
These look more like edge cases / exceptions, as the vast majority of the `Accountable` implementations either only use heap memory, or consciously leave out the off-heap memory they use in their `ramBytesUsed` implementation. (`OffHeapFSTStore` , `DirectPackedReader`, `LegacyDirectMonotonicReader`). VectorsReader impls all consistently report on heap only by design.
`KnnVectorDict.ramBytesUsed()` unambiguously includes off-heap (mmap) bytes today, but it does not implement `Accountable`. That's perhaps a good candidate for renaming to avoid confusion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org