You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Lvnszn (via GitHub)" <gi...@apache.org> on 2024/02/27 02:32:41 UTC
[I] Configurable heap size for query and realtime segment persistence. [pinot]
Lvnszn opened a new issue, #12503:
URL: https://github.com/apache/pinot/issues/12503
When pinot query data, pinot will load the segment from disk to offheap, but when persisting segment, pinot need to use offheap. This memory will interact with each other and have a probability of causing write failure. The current way to alleviate it is to configure the write segment amount.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Posted by "Lvnszn (via GitHub)" <gi...@apache.org>.
Lvnszn commented on issue #12503:
URL: https://github.com/apache/pinot/issues/12503#issuecomment-1968119315
Here is another error log in server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Posted by "Lvnszn (via GitHub)" <gi...@apache.org>.
Lvnszn commented on issue #12503:
URL: https://github.com/apache/pinot/issues/12503#issuecomment-1972735574
> Is the problem resolved automatically? We'll need more context in order to debug it
Add a server and rebalance table to a new server then the problem was resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Posted by "Lvnszn (via GitHub)" <gi...@apache.org>.
Lvnszn commented on issue #12503:
URL: https://github.com/apache/pinot/issues/12503#issuecomment-1968116146
Here is my jvm config,
`javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms6G -Xmx14G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:/opt/pinot/gc-pinot-server.log -Dlog4j2.configurationFile=/opt/pinot/conf/pinot-server-log4j2.xml -Dplugins.dir=/opt/pinot/plugins -Dio.grpc.netty.shaded.io.netty.tryReflectionSetAccessible=true -Dpinot.admin.system.exit=false -Djute.maxbuffer=10485760`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12503:
URL: https://github.com/apache/pinot/issues/12503#issuecomment-1972254102
Is the problem resolved automatically? We'll need more context in order to debug it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12503:
URL: https://github.com/apache/pinot/issues/12503#issuecomment-1967854085
Did some research and seems this usually indicates that the server is super low on direct memory. How much direct memory is configured in your JVM? Can you check the server log and see if you can get more context?
One thing I'm suspecting is that server might try to build some large inverted indexes, which will use direct memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Posted by "Lvnszn (via GitHub)" <gi...@apache.org>.
Lvnszn commented on issue #12503:
URL: https://github.com/apache/pinot/issues/12503#issuecomment-1970895940
> Do you have access to the server node? K8s log might not be enough because it won't log the actual stack trace, and it doesn't contain the non-error logs.
The log has been recycled. . What I recorded here is OOM: map fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12503:
URL: https://github.com/apache/pinot/issues/12503#issuecomment-1970033581
Do you have access to the server node? K8s log might not be enough because it won't log the actual stack trace, and it doesn't contain the non-error logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org