You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/09/21 19:15:32 UTC
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #6040: Remove the partition info from the consuming segment ZK metadata
Jackie-Jiang opened a new pull request #6040:
URL: https://github.com/apache/incubator-pinot/pull/6040
## Description
For issue: #6029
Do not persist the partition info to the consuming segment ZK metadata because it might not reflect the correct segment partitioning when the data is not partitioned correctly in the stream, or the partitions changed during the consumption. Persisting the partition info could cause segment to be mis-pruned, thus cause wrong result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6040: Remove the partition info from the consuming segment ZK metadata
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6040:
URL: https://github.com/apache/incubator-pinot/pull/6040#issuecomment-696415308
@mcvsubbu The segment partition pruning for consuming segment will happen on server side instead of broker side. Yes there will be some overhead, and we need to measure the impact with a high QPS use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6040: Remove the partition info from the consuming segment ZK metadata
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6040:
URL: https://github.com/apache/incubator-pinot/pull/6040#issuecomment-701105519
After some benchmark we found this change might cause some performance degradation for extremely high QPS use cases. Closing this PR and filed #6078 as a work-around without overhead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6040: Remove the partition info from the consuming segment ZK metadata
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6040:
URL: https://github.com/apache/incubator-pinot/pull/6040#issuecomment-696415308
@mcvsubbu The segment partition pruning for consuming segment will happen on server side instead of broker side. Yes there will be some overhead, and we need to measure the impact with a high QPS use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang closed pull request #6040: Remove the partition info from the consuming segment ZK metadata
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed pull request #6040:
URL: https://github.com/apache/incubator-pinot/pull/6040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6040: Remove the partition info from the consuming segment ZK metadata
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6040:
URL: https://github.com/apache/incubator-pinot/pull/6040#issuecomment-696415308
@mcvsubbu The segment partition pruning for consuming segment will happen on server side instead of broker side. Yes there will be some overhead, and we need to measure the impact with a high QPS use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org