You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/11 22:17:40 UTC
[GitHub] [pinot] abhs50 opened a new issue, #9788: Logging Limit
abhs50 opened a new issue, #9788:
URL: https://github.com/apache/pinot/issues/9788
Currently 1 log on the broker is followed by N logs (1 per server). So every query gets N + 1 logs. For a high QPS system, this can turn into a lot of data.
Reduce amount of logging in broker, server using a config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1324317790
On the server side, we have `pinot.query.scheduler.query.log.maxRatePerSecond` which is similar to the config on the broker side. More configs can be found here: https://docs.pinot.apache.org/configuration-reference/server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1338037448
> Apologies for delay. @Jackie-Jiang : Is there a config which controls this :
>
> https://github.com/apache/pinot/blob/dcc496f63b4488989808ec144a7e2ffcc4697c75/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L212
>
> We see a bunch of logs in our server pods due to this [line](https://github.com/apache/pinot/blob/dcc496f63b4488989808ec144a7e2ffcc4697c75/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L1575) . We want to limit it to a lower value
Seems it is not configurable as of now. Do you want to help contribute a PR to make it configurable?
> Additionally is there way to keep Pinot Server in ERROR logging mode instead of INFO by default. We are using helm installation way of deploying Pinot.
@xiangfu0 do you know how to change the default logging level with helm installation? I think it is doable with a customized log4j config file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1322647083
We already have some rate limit config, see `QueryLogger` class for details. Do you think we need to add more configs?
cc @agavra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
abhs50 commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1353586652
@Jackie-Jiang : This can be controlled via Helm envs as here : https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/values.yaml#L167
In that case we wont this MR and we are moving to ERROR logging only in production.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
abhs50 commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1335933999
Apologies for delay. @Jackie-Jiang : Is there a config which controls this : https://github.com/apache/pinot/blob/dcc496f63b4488989808ec144a7e2ffcc4697c75/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L212
We see a bunch of logs in our server pods due to this [line](https://github.com/apache/pinot/blob/dcc496f63b4488989808ec144a7e2ffcc4697c75/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L1575) :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
abhs50 commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1335934766
Additionally is there way to keep Pinot Server in Error logging mode instead of `info` by default. We are using helm installation way of deploying Pinot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
abhs50 commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1338280258
@Jackie-Jiang : Yes, I can work on an PR to make it configurable.
I will research more on logging config with helm installation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] bagipriyank commented on issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
bagipriyank commented on issue #9788:
URL: https://github.com/apache/pinot/issues/9788#issuecomment-1322801100
@Jackie-Jiang rate limiting option in `QueryLogger` is very helpful to limit the logs from broker. However right now, there is a log per query from each server that is involved in responding to the query. We are looking at options to reduce logging on the server side, and not just broker.
We also want to look at reducing log volume from ingestion from kafka. Looks like options for controlling that are hard coded in the code right now as shared by @abhs50 above. It would be helpful to have them configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 closed issue #9788: Logging Limit
Posted by GitBox <gi...@apache.org>.
abhs50 closed issue #9788: Logging Limit
URL: https://github.com/apache/pinot/issues/9788
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org