You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2025/03/26 22:12:32 UTC
[I] Add native memory manager [pinot]
Jackie-Jiang opened a new issue, #15381:
URL: https://github.com/apache/pinot/issues/15381
As we add more and more off-heap data structures (e.g. #15205), we should track the native memory usage in a central place, and emit metrics for the native memory usage. This can be useful to detect memory leaks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15381:
URL: https://github.com/apache/pinot/issues/15381#issuecomment-4340081660
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on issue #15381:
URL: https://github.com/apache/pinot/issues/15381#issuecomment-2774498064
Is Project Panama officially supported in a JDK ?
I would recommend looking at Netty's memory manager -- Pool based and Unpool
Apache Arrow uses Netty (for RPC as well) to implements a full fledged memory manager, accountant etc. The abstraction is ByteBuf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "mghildiy (via GitHub)" <gi...@apache.org>.
mghildiy commented on issue #15381:
URL: https://github.com/apache/pinot/issues/15381#issuecomment-2761443018
Fomr initial research, I see for ByteBuffer based allocations, we can make use of BufferPoolMXBean to hook into jvm to track the 0ff-heap memory. For Unsafe, we may need to track manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15381:
URL: https://github.com/apache/pinot/issues/15381#issuecomment-4676146916
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "mghildiy (via GitHub)" <gi...@apache.org>.
mghildiy commented on issue #15381:
URL: https://github.com/apache/pinot/issues/15381#issuecomment-2765183331
I was wondering if using MemorySegment(project Panama) is a better approach than using Unsafe for off-heap memory allocations. It performa automatic memory deallocation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15381: Add native memory manager
URL: https://github.com/apache/pinot/issues/15381
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "mghildiy (via GitHub)" <gi...@apache.org>.
mghildiy commented on issue #15381:
URL: https://github.com/apache/pinot/issues/15381#issuecomment-2758153669
I am not sure how much in-depth understanding of pinot codebase is needed for it. I may give it a try, may be would need guidance from experienced members.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add native memory manager [pinot]
Posted by "mghildiy (via GitHub)" <gi...@apache.org>.
mghildiy commented on issue #15381:
URL: https://github.com/apache/pinot/issues/15381#issuecomment-2774512598
Panama is still not GA.
I would take a look at ByteBuf.
Manish
On Thu, 3 Apr, 2025, 10:30 am Siddharth Teotia, ***@***.***>
wrote:
> Is Project Panama officially supported in a JDK ?
>
> I would recommend looking at Netty's memory manager -- Pool based and
> Unpool
>
> Apache Arrow uses Netty (for RPC as well) to implement a full fledged
> memory manager, accountant etc. The abstraction is ByteBuf
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/pinot/issues/15381#issuecomment-2774498064>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACMXFVQ3YMGL6JDR5WTKXGL2XS575AVCNFSM6AAAAABZ3O5SPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONZUGQ4TQMBWGQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
> [image: siddharthteotia]*siddharthteotia* left a comment
> (apache/pinot#15381)
> <https://github.com/apache/pinot/issues/15381#issuecomment-2774498064>
>
> Is Project Panama officially supported in a JDK ?
>
> I would recommend looking at Netty's memory manager -- Pool based and
> Unpool
>
> Apache Arrow uses Netty (for RPC as well) to implement a full fledged
> memory manager, accountant etc. The abstraction is ByteBuf
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/pinot/issues/15381#issuecomment-2774498064>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACMXFVQ3YMGL6JDR5WTKXGL2XS575AVCNFSM6AAAAABZ3O5SPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONZUGQ4TQMBWGQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org