You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2023/04/13 18:31:35 UTC
[GitHub] [pinot] ankitsultana opened a new issue, #10606: Add Support for Pluggable Query Event Info Listeners
ankitsultana opened a new issue, #10606:
URL: https://github.com/apache/pinot/issues/10606
Query event info listeners are available in engines like Presto and are really helpful in improving observability.
We have written a proposal for supporting the same in Pinot here: https://docs.google.com/document/d/1zWHM3SwGCRTzaxm9YF7PViALclm99L99n5rH4XDWJa0/edit
cc: @Jackie-Jiang @yupeng9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #10606: Add Support for Pluggable Query Event Info Listeners
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #10606:
URL: https://github.com/apache/pinot/issues/10606#issuecomment-1509407442
There'll be events emitted in both brokers and servers.
Our goal is to make it very flexible: some users may only be interested in broker events, some may be interested in both broker and server events, so we'll have separate interfaces for these and users can plug in howsoever they wish.
The interface can take in the query-info in a dedicated POJO, which can make it easy to add more details later without having to worry about backwards incompatibility. e.g. `BrokerQueryEventInfo::trigger(QueryInfo)`.
`QueryInfo` here may be something like:
```
QueryInfo {
private String _sqlQuery;
private long _requestId;
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on issue #10606: Add Support for Pluggable Query Event Info Listeners
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on issue #10606:
URL: https://github.com/apache/pinot/issues/10606#issuecomment-1633106208
I am working with @ankitsultana on the design doc for this.
Please assign to me! cc @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10606: Add Support for Pluggable Query Event Info Listeners
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10606:
URL: https://github.com/apache/pinot/issues/10606#issuecomment-1509374058
@mcvsubbu @siddharthteotia I think query events are already tracked at LinkedIn. Can you share how you achieve that and also help review the design?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10606: Add Support for Pluggable Query Event Info Listeners
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10606:
URL: https://github.com/apache/pinot/issues/10606#issuecomment-1509400665
Right now, we use the query statistics returned from the broker. Overall, a listener seems to be a good idea. We need to get the schema correctly, so that the result has all the elements we need, whether they be emitted as a result of V1 or V2 engine (in stages).
Even in V2 engine, we will need the aggregate across all servers on the event emitted in the broker (at least aggregate thread-cpu used across consuming/completed/offline segments, with posibility of adding other things later). So, i assume that even if V2 engine is used, there will be a final event emitted from the broker right? (a cursory look at th edesign doc says it is only emitted from the servers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on issue #10606: Add Support for Pluggable Query Event Info Listeners
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on issue #10606:
URL: https://github.com/apache/pinot/issues/10606#issuecomment-1646182333
Hey folks, this is the design doc for this: https://docs.google.com/document/d/11UeU6eNTlyEJ9G096j9XT57QuWgbuCtwnfTC5TQpQ8M/edit
Please review! Thanks!
cc @ankitsultana @Jackie-Jiang @walterddr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org