You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jtao15 (via GitHub)" <gi...@apache.org> on 2025/02/27 16:53:53 UTC
[I] High Memory Usage in Mutable Dictionary Index When Cardinality Increases Significantly [pinot]
jtao15 opened a new issue, #15147:
URL: https://github.com/apache/pinot/issues/15147
We've observed excessive heap memory allocation when new columns with a dictionary forward index are added first, and Kafka values are populated later. This issue arises due to static buffer size allocation based on column cardinality from `StatsHistory` (e.g., 1, due to the default value).
When Kafka ingestion begins and the actual cardinality increases significantly, small buffers are repeatedly allocated, leading to a long [list of bufferReader]( https://github.com/apache/pinot/blob/8b2d5b6d9ee980ee8004d44edb0c0e32ad5f7db0/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/forward/FixedByteSVMutableForwardIndex.java#L54). Since the implementation uses a `CopyOnWriteArrayList`, adding new bufferReader entries is expensive, leading to increased memory usage, excessive garbage collection, and potentially causing out-of-memory (OOM) issues.
Ideally, buffer size allocation should be dynamically adjusted at consumption time rather than relying only on `StatsHistory`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] High Memory Usage in Mutable Dictionary Index When Cardinality Increases Significantly [pinot]
Posted by "jtao15 (via GitHub)" <gi...@apache.org>.
jtao15 commented on issue #15147:
URL: https://github.com/apache/pinot/issues/15147#issuecomment-2689367098
cc @vvivekiyer @sajjad-moradi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] High Memory Usage in Mutable Dictionary Index When Cardinality Increases Significantly [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15147:
URL: https://github.com/apache/pinot/issues/15147#issuecomment-4348591427
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] High Memory Usage in Mutable Dictionary Index When Cardinality Increases Significantly [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15147:
URL: https://github.com/apache/pinot/issues/15147#issuecomment-4686359171
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] High Memory Usage in Mutable Dictionary Index When Cardinality Increases Significantly [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15147: High Memory Usage in Mutable Dictionary Index When Cardinality Increases Significantly
URL: https://github.com/apache/pinot/issues/15147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org