You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/08 15:08:36 UTC
[GitHub] [incubator-pinot] Aka-shi opened a new issue #6555: [Pinot]API to reset a stream to earliest/latest offset
Aka-shi opened a new issue #6555:
URL: https://github.com/apache/incubator-pinot/issues/6555
In scenarios where the topic is corrupted or recreated or when migrating to a new cluster, pinot streaming consumption is affected because of how pinot manages offsets. It would be nice to have an API where we can reset the stream to start from the earliest/latest offset based on the configuration.
Druid provides a similar API to reset a supervisor, which does the offset and streaming tasks management, which makes these situations easier to handle. Reference - https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#resetting-supervisors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6555: [Pinot]API to reset a stream to earliest/latest offset
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6555:
URL: https://github.com/apache/incubator-pinot/issues/6555#issuecomment-775330365
In general, offsets will not match across clusters. Assuming that the new cluster/topic is a replica of the old one, it is not reasonable to assume that the two clusters are exactly in sync either. So, if such an API exists, pinot could receive either duplicate rows, or miss rows altogether. Is that acceptable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org