You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/08 23:44:21 UTC
[GitHub] [pinot] abhs50 opened a new issue, #8869: Pinot Broker logs to show source_ip of host/user who executed the query
abhs50 opened a new issue, #8869:
URL: https://github.com/apache/pinot/issues/8869
Currently in Pinot broker we see logs as below : <Sample log>
`requestId=130,table<redacted>,timeMs=23,docs=72/108615312,entries=2741105/792,segments(queried/processed/matched/consuming/unavailable):256/253/8/32/0,consumingFreshnessTimeMs=1654707653215,servers=6/6,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs,RequestSentDelayMs);pinot-server-2_R=0,4,1817,0,1;pinot-server-5_R=0,5,1817,1,1;pinot-server-1_R=0,20,1820,0,1;pinot-server-4_R=1,21,1820,0,1;pinot-server-0_R=1,4,1818,0,1;pinot-server-3_R=1,5,1817,0,1,offlineThreadCpuTimeNs(total/thread/sysActivity/resSer):0/0/0/0,realtimeThreadCpuTimeNs(total/thread/sysActivity/resSer):0/0/0/0,query=<redacted>`
Another useful param to add here is source_ip of host/user who executed the query. This will be useful to track usage of Pinot from a particular user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8869: Pinot Broker logs to show source_ip of host/user who executed the query
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8869:
URL: https://github.com/apache/pinot/issues/8869#issuecomment-1154342292
@abhs50 You may take a look at `PinotClientRequest` class where the `HttpRequesterIdentity` is created from the request context. This identity is passed to the `BrokerRequestHandler.handleRequest()` where you can log the source info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 commented on issue #8869: Pinot Broker logs to show source_ip of host/user who executed the query
Posted by GitBox <gi...@apache.org>.
abhs50 commented on issue #8869:
URL: https://github.com/apache/pinot/issues/8869#issuecomment-1157878364
@Jackie-Jiang : Please have a look : https://github.com/apache/pinot/pull/8907
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #8869: Pinot Broker logs to show source_ip of host/user who executed the query
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8869:
URL: https://github.com/apache/pinot/issues/8869#issuecomment-1158139746
I would like to make sure that this is a MUST have feature. We have worked hard to keep to exactly ONE log per happy-path query on the broker, and kept the log in a parseable fashion. Increasing log size is a concern both while debugging production problems and storage as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 commented on issue #8869: Pinot Broker logs to show source_ip of host/user who executed the query
Posted by GitBox <gi...@apache.org>.
abhs50 commented on issue #8869:
URL: https://github.com/apache/pinot/issues/8869#issuecomment-1157729246
@Jackie-Jiang : There are couple of we can add it :
1. Log all the headers
`((HttpRequesterIdentity) requesterIdentity).getHttpHeaders().toString()`
2. Parse out specific value from each header. In this case we can log just : `X-Forwarded-For`
Will 1 become too verbose for logging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8869: Pinot Broker logs to show source_ip of host/user who executed the query
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8869:
URL: https://github.com/apache/pinot/issues/8869#issuecomment-1152659364
Good suggestion. We can add the remote host info to the `HttpRequesterIdentity` and log it accordingly if it is provided to the `BrokerRequestHandler`. Do you want to help contribute this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhs50 commented on issue #8869: Pinot Broker logs to show source_ip of host/user who executed the query
Posted by GitBox <gi...@apache.org>.
abhs50 commented on issue #8869:
URL: https://github.com/apache/pinot/issues/8869#issuecomment-1152712956
Hi @Jackie-Jiang: Yes, I will be more than happy to contribute. Can you help me with few details on where to look and how to get started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org