You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/09/09 22:29:49 UTC
[GitHub] [pinot] ankitsultana opened a new issue, #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
ankitsultana opened a new issue, #9371:
URL: https://github.com/apache/pinot/issues/9371
We recently enabled the feature to return exceptions for cases when some segments are missing (as opposed to returning partial results).
This feature was rolled out in #7823.
After enabling it we have started seeing the following error occasionally for our Realtime tables. On digging a bit we found that this error comes when a segment is deleted due to retention.
```
ServerSegmentMissing:1 segments [<segment-name>] missing on server: <server-name>, errorCode:235
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] chenboat closed issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
Posted by GitBox <gi...@apache.org>.
chenboat closed issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
URL: https://github.com/apache/pinot/issues/9371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
Posted by GitBox <gi...@apache.org>.
ankitsultana commented on issue #9371:
URL: https://github.com/apache/pinot/issues/9371#issuecomment-1242641878
Another solution may be to extend TimeSegmentPruner or create a new Pruner to prune segments which are older than the configured retention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #9371:
URL: https://github.com/apache/pinot/issues/9371#issuecomment-1242544786
routing table is re-built when the external view notification comes in. That means any segment that is there now (and was not there before) is deleted, and any segment that was not there before (and is there now) are added. Also, if segments are moved, then that is included in the routing reconstruction as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9371:
URL: https://github.com/apache/pinot/issues/9371#issuecomment-1242559488
Hmm, this means server processes the callbacks faster than the broker. We have #7818 trying to address the similar issue, but if the version you are running already includes this PR, that means broker is too slow, and before it processes the IS change, server already deleted the segment.
In that case, one possible solution is to introduce a delay on the server before dropping the segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
Posted by GitBox <gi...@apache.org>.
ankitsultana commented on issue #9371:
URL: https://github.com/apache/pinot/issues/9371#issuecomment-1247131002
@Jackie-Jiang suggested that we can keep a cache of the deleted segments in Pinot Servers and ignore this error if the missing segment is a recently deleted segment. We can try this approach out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] chenboat commented on issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
Posted by GitBox <gi...@apache.org>.
chenboat commented on issue #9371:
URL: https://github.com/apache/pinot/issues/9371#issuecomment-1242539231
@Jackie-Jiang @mcvsubbu How does Pinot handle the case after the broker gets the external view segment to server mapping, the segments get deleted right after that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9371: Segments Deleted Due to Retention Can Cause Query Failures with ServerSegmentMissing Error
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9371:
URL: https://github.com/apache/pinot/issues/9371#issuecomment-1245014264
This can happen when segment is manually deleted as well, so I feel we want a more general solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org