You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/11/10 20:49:25 UTC
[GitHub] [pinot] dongxiaoman opened a new issue #7740: Transient Status 305 error from recent master version (10/29/2021)
dongxiaoman opened a new issue #7740:
URL: https://github.com/apache/pinot/issues/7740
For the past a few days, our production cluster has seen several cases of this exception, causing queries to fail.
```
Caused by: org.apache.pinot.client.PinotClientException: Query had processing exceptions: \n[{\"errorCode\":305,\"message\":\"2358 segments [point_entry__108__1331__20211110T1138Z, point_entry__142__1331__20211110T1044Z, point_entry__34__3196__20211110T0944Z, point_entry__46__3207__20211110T1945Z, point_entry__112__1329__20211110T0843Z, point_entry__34__3205__20211110T1845Z, point_entry__70__1331__20211110T1339Z, point_entry__188__1334__20211110T1449Z, point_entry__36__3190__20211110T0249Z, point_entry__156__1329__20211110T0955Z, point_entry__68__1331__20211110T1145Z, point_entry__176__1328__20211110T0844Z, point_entry__74__1322__20211110T0239Z, point_entry__0__3197__20211110T0938Z, point_entry__38__3198__20211110T1244Z, point_entry__6__3202__20211110T1645Z, point_entry__4__3197__20211110T1043Z, point_entry__178__1322__20211110T0149Z, point_entry__22__3197__20211110T1050Z, point_entry__116__1328__20211110T0939Z,...] (truncated by Splunk)
```
When I tried to query `point_entry__188__1334__20211110T1449Z` it is clear that those realtime segments were deleted by Pinot retention manager already, somehow the query is still hitting those segments.
This bug is hard to reproduce, the first time we hit it, we did not find out which server has the problem, we have to restart all Pinotserver and the problem is gone.
This second time it seems better that it self resolved.
It is something worth investigating
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7740: Transient Status 305 error from recent master version (10/29/2021)
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7740:
URL: https://github.com/apache/pinot/issues/7740#issuecomment-965850770
I feel this issue might be related to the ZK callback lag/inconsistency. Can you please check the ZK server system usage and see if the ZK servers are overloaded when the exception happens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] dongxiaoman commented on issue #7740: Transient Status 305 error from recent master version (10/29/2021)
Posted by GitBox <gi...@apache.org>.
dongxiaoman commented on issue #7740:
URL: https://github.com/apache/pinot/issues/7740#issuecomment-965739323
Some extra logs in `pinotbroker`
```
Failed to find servers hosting segment: point_entry__108__1331__20211110T1138Z for table: point_entry_REALTIME (all ONLINE/CONSUMING instances: [] and OFFLINE instances: [] are disabled, counting segment as unavailable)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] dongxiaoman closed issue #7740: Transient Status 305 error from recent master version (10/29/2021)
Posted by GitBox <gi...@apache.org>.
dongxiaoman closed issue #7740:
URL: https://github.com/apache/pinot/issues/7740
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] dongxiaoman commented on issue #7740: Transient Status 305 error from recent master version (10/29/2021)
Posted by GitBox <gi...@apache.org>.
dongxiaoman commented on issue #7740:
URL: https://github.com/apache/pinot/issues/7740#issuecomment-966626022
@Jackie-Jiang After some debugging, it turns out to be a one-time problem. We have clusters in two regions, and we only restarted one region.
It is not easy to reproduce that one-time issue anymore. Maybe it is related to a "kill all controllers" restart that should be avoided in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org