You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2024/02/07 19:08:40 UTC
Re: [PR] make thread local variable static to avoid the memory leaking issue [pinot]
wirybeaver commented on PR #12242:
URL: https://github.com/apache/pinot/pull/12242#issuecomment-1932695513
I made an A/B testing to compare the performance option 1(static thread local) vs option 2(move bytes array to context) in two clusters with same size and configuration. It turns out that the option 1 is much better than the option 2 in terms of the query latency and memory usage efficiency when the QPS is high and queries consist of json_indexing over large json blobs and reading large payload column.
| Option | Young Gen CountAve per 5min | Ave Young Gen Time | Max Supervised Space After GC | Peak heap usage | Server Up Time | P90 Query Latency |
|--------------------------------------|-----------------------------|--------------------|-------------------------------|-----------------|----------------------------|-------------------|
| 1 (Preferred) static thread local | 1 | 0.8ms | 430MB | 91GB | Min: > 2 day Max: infinite | 4s |
| 2 Move byte array to reader context | 5 | 5ms | 2.3GB | 186GB | All nodes: < ~1 day | 20s |
After I rollback the option 2 to option 1, the heap memory usage drop downs by half.
<a href="https://imgur.com/ZfYMseP"><img src="https://i.imgur.com/ZfYMseP.png" title="source: imgur.com" /></a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org