You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "satwik-pachigolla (via GitHub)" <gi...@apache.org> on 2026/07/25 05:55:58 UTC
[I] Broker routing can briefly reference segments that servers no longer serve [pinot]
satwik-pachigolla opened a new issue, #19081:
URL: https://github.com/apache/pinot/issues/19081
Several Pinot workflows rely on broker routing converging with server-side segment state changes before the old serving path becomes unavailable. A stale routing entry can send a query to a server after that server has dropped the segment, producing SERVER_SEGMENT_MISSING.
1. Server readiness versus broker readiness
[Issue #16565](https://github.com/apache/pinot/issues/16565) describes a server returning healthy before brokers had resumed routing to it. During a rolling restart, this allowed the next server to be restarted while neither replica was queryable.
2. Segment deletion / retention
Deletion has the inverse ordering requirement. The controller updates IdealState, servers asynchronously process the resulting transition and remove the segment, and brokers asynchronously rebuild routing. If a broker routes a query using the old assignment after the server has removed the segment, the query fails.
This is less likely for ordinary retention when time pruning excludes expired segments, but it remains possible for queries spanning the retention boundary and for explicit segment deletion.
3. Rebalance transition
Even with add-before-remove rebalance, a query can be dispatched immediately before the source server’s state transition reaches broker routing. If the source has removed the segment by request handling time, it returns:
Query processing exceptions:
{235=1 segments [<segment>] missing on server: <server>}
The common requirement is that brokers must stop routing a segment to a server before that server drops the segment. An external workflow can enforce this by waiting until every broker routing table excludes the target segments before deletion; Pinot currently does not provide a controller-level acknowledgement or ordering guarantee for that invariant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Broker routing can briefly reference segments that servers no longer serve [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #19081:
URL: https://github.com/apache/pinot/issues/19081#issuecomment-5199648825
https://github.com/apache/pinot/issues/18902#issuecomment-5156173796
this is linked as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Broker routing can briefly reference segments that servers no longer serve [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #19081:
URL: https://github.com/apache/pinot/issues/19081#issuecomment-5208652024
Server does track `_recentlyDeletedSegments` in `BaseTableDataManager`. There might be a small race window when server just started. We need to find a solution for that.
cc @J-HowHuang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org