You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/04/12 21:40:29 UTC

[GitHub] [pinot] chenboat opened a new issue, #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

chenboat opened a new issue, #8519:
URL: https://github.com/apache/pinot/issues/8519

   Today if an empty box Pinot server joins the cluster to replace an existing server (i.e., the two servers share the same id), it can take 10s of minutes for Helix to instruct the new server to download all the segments and get them online. During the above process, Pinot brokers still route queries to the new server, this behavior results in partial and incorrect query results.
   
   We should fix this bug by not routing the query to the server until it loads all the segments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] mcvsubbu commented on issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8519:
URL: https://github.com/apache/pinot/issues/8519#issuecomment-1097433786

   But then only those segments that are already loaded (and are ONLINE or CONSUMING in externalview on the new server) should get queries right? Is it a performance issue that you are facing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] chenboat commented on issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

Posted by GitBox <gi...@apache.org>.
chenboat commented on issue #8519:
URL: https://github.com/apache/pinot/issues/8519#issuecomment-1201858785

   @mcvsubbu This is not a performance issue but rather a query correctness issue. I also prefer to use the ExternalView as the single place to control the query routing decision rather using an additional flag. However, today it is possible that a Pinot segment is in ONLINE/CONSUMING state but the segment is still yet fully downloaded to the server during the segment replacement process I mentioned above. To me it is a bug as well. cc @Jackie-Jiang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] mcvsubbu commented on issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #8519:
URL: https://github.com/apache/pinot/issues/8519#issuecomment-1201882228

   One thing that can be done is as a part of new server workflow, set the `queriesDisabled` flag in Helix to true. This flag behaves the same as `shutdownInProgress`, except that the server does not change the flag back any time. It is used purely for admin purpose.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang closed issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments 
URL: https://github.com/apache/pinot/issues/8519


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on issue #8519: [Bug]Pinot broker routes queries to servers while they are still downloading segments

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8519:
URL: https://github.com/apache/pinot/issues/8519#issuecomment-1097437319

   @mcvsubbu The issue here is that for new servers, the `shutdownInProgress` flag is not set, and broker will start routing queries to it whenever the segment becomes ONLINE/CONSUMING in the external view. We should set the flag if it is not set yet when a server starts, so that all the servers (new server, existing server that encounters hard failure etc.) have the same behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org