You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/02/24 22:40:12 UTC
[GitHub] [incubator-pinot] akshayrai opened a new pull request #5091:
[TE][subscription] Dimension alerter should be resilient to partial errors;
Emit metrics to track emails and jiras
akshayrai opened a new pull request #5091: [TE][subscription] Dimension alerter should be resilient to partial errors; Emit metrics to track emails and jiras
URL: https://github.com/apache/incubator-pinot/pull/5091
Current setup of dimension alerter has the following issue:
If one of the recipients under dimension recipients has an error(ex: assignee invalid) then,
1. We will mark the task as FAILED and not proceed with the remaining recipients even if they are correctly configured.
2. Along with marking the task as FAILED, we also do not update the watermarks. Hence the next time the alerter runs, a subset of the dimension recipients will be notified again until we reach the error.
Proposal:
* Dimension alerter should not fail the pipeline if one of the dimension recipient has some issue.
* Also added counters to track the number of email and jira metrics along with breakdown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on issue #5091:
[TE][subscription] Dimension alerter should be resilient to partial errors;
Emit metrics to track emails and jiras
Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on issue #5091: [TE][subscription] Dimension alerter should be resilient to partial errors; Emit metrics to track emails and jiras
URL: https://github.com/apache/incubator-pinot/pull/5091#issuecomment-590621634
Thanks for the fix. @akshayrai .
One question is: If users don't receive all of the alerts how do we let them know?
Can we add something in the email email says these are only partial alerts and some alerts are not sent?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai edited a comment on issue #5091:
[TE][subscription] Dimension alerter should be resilient to partial errors;
Emit metrics to track emails and jiras
Posted by GitBox <gi...@apache.org>.
akshayrai edited a comment on issue #5091: [TE][subscription] Dimension alerter should be resilient to partial errors; Emit metrics to track emails and jiras
URL: https://github.com/apache/incubator-pinot/pull/5091#issuecomment-591012020
> Thanks for the fix. @akshayrai .
> One question is: If users don't receive all of the alerts how do we let them know?
> Can we add something in the email email says these are only partial alerts and some alerts are not sent?
@xiaohui-sun
This is a pretty interesting problem but notifying users is a bit non-trivial and tricky. Note that each dimension recipient is independent of each other (each dimension recipient receives a completely different email), and there is no point in notifying other dimension recipients about the error. We could probably send an error email to the specific concerned recipients which would be helpful even in other normal cases. For example, if the detection failed or our default subscription config has an issue then we can send an error email. We can discuss this further offline.
In addition to this, what I have been thinking for a while is that we introduce the concept of subscription health similar to detection health and use that to kind of track how it performs (maybe even make it part of the email).
Besides, this specific PR attempts to address this issue by logging and emitting metrics which can be tracked and hopefully we can introduce some more sophisticated offline validations on the config so that such things do not occur, at least not by misconfiguration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai merged pull request #5091:
[TE][subscription] Dimension alerter should be resilient to partial errors;
Emit metrics to track emails and jiras
Posted by GitBox <gi...@apache.org>.
akshayrai merged pull request #5091: [TE][subscription] Dimension alerter should be resilient to partial errors; Emit metrics to track emails and jiras
URL: https://github.com/apache/incubator-pinot/pull/5091
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on issue #5091:
[TE][subscription] Dimension alerter should be resilient to partial errors;
Emit metrics to track emails and jiras
Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on issue #5091: [TE][subscription] Dimension alerter should be resilient to partial errors; Emit metrics to track emails and jiras
URL: https://github.com/apache/incubator-pinot/pull/5091#issuecomment-591226451
> Besides, this specific PR attempts to address this issue by logging and emitting metrics which can be tracked and hopefully we can introduce some more sophisticated offline validations on the config so that such things do not occur, at least not by misconfiguration.
Like the idea of monitoring using autometrics.
It is a little different with detection health monitoring. For detection pipeline if we find the MAPE degrades we can retrain the model. There is nothing we can do except fixing the configurations for notification. We should aiming at 100% success for notifications.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai commented on issue #5091:
[TE][subscription] Dimension alerter should be resilient to partial errors;
Emit metrics to track emails and jiras
Posted by GitBox <gi...@apache.org>.
akshayrai commented on issue #5091: [TE][subscription] Dimension alerter should be resilient to partial errors; Emit metrics to track emails and jiras
URL: https://github.com/apache/incubator-pinot/pull/5091#issuecomment-591012020
> Thanks for the fix. @akshayrai .
> One question is: If users don't receive all of the alerts how do we let them know?
> Can we add something in the email email says these are only partial alerts and some alerts are not sent?
This is a pretty interesting problem but notifying users is a bit non-trivial and tricky. Note that each dimension recipient is independent of each other (each dimension recipient receives a completely different email), and there is no point in notifying other dimension recipients about the error. We could probably send an error email to the specific concerned recipients which would be helpful even in other normal cases. For example, if the detection failed or our default subscription config has an issue then we can send an error email. We can discuss this further offline.
In addition to this, what I have been thinking for a while is that we introduce the concept of subscription health similar to detection health and use that to kind of track how it performs (maybe even make it part of the email).
Besides, this specific PR attempts to address this issue by logging and emitting metrics which can be tracked and hopefully we can introduce some more sophisticated offline validations on the config so that such things do not occur, at least not by misconfiguration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org