You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/01 20:36:14 UTC
[GitHub] [incubator-pinot] fx19880617 opened a new issue #6516: Set error messages in helix for state transition failues
fx19880617 opened a new issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516
It can help access the errors from the controller UI without searching server logs for segment errors or broker/server instances go into error status.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-771206290
Yes, right now we only know that segment in error state from external view. The goal is to also set the stacktrace for segment loading and easy for users to retrieve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-771818395
I think Helix preserves the error stack/messages for a few days? State stored in segmentDataManager can go away on restart. I think there is value in retrieving helix messages (if they exist) as well.
On a recent issue in Linkedin, it turned out that the state transitions did not succeed and caused a performance hit on pinot servers. The errors went away upon restart. I can totally imagine an admin restarting the servers to get the system up & running.
It may make sense to have an API gather all debug information available and output it as a a blob of text. We can enhance this as we feel like, without regarding to backward compat, etc.
curl controller:port/tables/{tableName}/debugState 1> saved 2>&1
can be done before any restart is attempted. This can also retrieve states from each of the servers that have segments for the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-771145481
can you please elaborate "set error messages in helix"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-771818395
I think Helix preserves the error stack/messages for a few days? State stored in segmentDataManager can go away on restart. I think there is value in retrieving helix messages (if they exist) as well.
On a recent issue in Linkedin, it turned out that the state transitions did not succeed and caused a performance hit on pinot servers. The errors went away upon restart. I can totally imagine an admin restarting the servers to get the system up & running.
It may make sense to have an API gather all debug information available and output it as a a blob of text. We can enhance this as we feel like, without regarding to backward compat, etc.
curl controller:port/tables/{tableName}/debugState 1> saved 2>&1
can be done before any restart is attempted. This can also retrieve states from each of the servers that have segments for the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-771178201
If I understand what @fx19880617 is saying, it is to provide an API in the controller to retrieve Helix state transition error messages and perhaps history as well. I concur that it is a useful feature to have. Once we have the servers and the error messages, we can look into the server for a specific cause.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] dharakk commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
dharakk commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-781020414
Hi, from experience, this will be a really useful feature to me. I would love to give it a try if nobody is working on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-771387950
Got it. One thing to keep in mind is that state transition error messages get cleaned up automatically by Helix. we should probably do something like this
- each server stores the error message/segment status in segmentDataManager
- exposes an api to get the status along with message
- controller can invoke the server api on demand
Please don't save the error stacktrace in Helix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #6516: Set error messages in helix for state transition failues
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #6516:
URL: https://github.com/apache/incubator-pinot/issues/6516#issuecomment-771387950
Got it. One thing to keep in mind is that state transition error messages get cleaned up automatically by Helix. we should probably do something like this
- each server stores the error message/segment status in segmentDataManager
- exposes an api to get the status along with message
- controller can invoke the server api on demand
Please don't save the error stacktrace in Helix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org