You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/09/17 22:06:06 UTC
[GitHub] [incubator-pinot] mayankshriv opened a new issue #6029: Partitioning mis-match can lead to bad query results in Realtime
mayankshriv opened a new issue #6029:
URL: https://github.com/apache/incubator-pinot/issues/6029
The current code simply assumes partitionId from kafka partition to be in accordance with the partition function defined in table config. In case there is a mis-match on kafka partition and what's set in table config, that leads to incorrect pruning of segments causing bad results for query.
There was indeed an old behavior were we wiped out the incorrect partition metadata when committing the segment, which would limit the scope to consuming queries. At the very minimum, we should bring back that behavior. This is important as we can run into these issues in various situations:
* Increase in number of partitions from Kafka side.
* Partition function change on either side (accidental or intentional).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6029: Partitioning mis-match can lead to bad query results in Realtime
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6029:
URL: https://github.com/apache/incubator-pinot/issues/6029#issuecomment-694552301
Fix for the issue: #6031
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6029: Partitioning mis-match can lead to bad query results in Realtime
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6029:
URL: https://github.com/apache/incubator-pinot/issues/6029#issuecomment-694552426
Perf testing branch which removes the partition info for consuming segments: https://github.com/apache/incubator-pinot/tree/remove_consuming_partition_info
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang closed issue #6029: Partitioning mis-match can lead to bad query results in Realtime
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #6029:
URL: https://github.com/apache/incubator-pinot/issues/6029
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6029: Partitioning mis-match can lead to bad query results in Realtime
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6029:
URL: https://github.com/apache/incubator-pinot/issues/6029#issuecomment-701106090
Removing the partition info for consuming segments could cause performance degradation for extremely high QPS use cases. Filed #6078 as a work-around without performance impact.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org