You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/01/10 20:50:48 UTC
[GitHub] [pinot] navina opened a new issue #7988: Enhance Pinot Streaming Consumer Interface
navina opened a new issue #7988:
URL: https://github.com/apache/pinot/issues/7988
There are streaming systems which require additional client-initiated operations to guarantee the availability of data for replay. For example, Google Pubsub does **not** expose any notion of `offsets` for messages. It only delivers the messages that were published after a consumer is created. In order to replay or rewind, the client has to manually create/manage `snapshots` (aka checkpoints), which serves as an alternate mechanism to replay messages from a particular point in the stream.
Pinot's realtime streaming ingestion is very closely modeled to kafka's semantics. In order to accommodate other streaming systems, it would be useful to expand the existing `PartitionGroupConsumer`.
Proposal is to provide a mechanism to `commit` the state to the streaming source. For details on the proposal, please see - https://docs.google.com/document/d/17eywG2zYHv72ye3nBwVSdxlF7JIcTlqcNkov2TCqaGs/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #7988: Enhance Pinot Streaming Consumer Interface
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #7988:
URL: https://github.com/apache/pinot/issues/7988#issuecomment-1009400224
FYI @mcvsubbu @snleee @sajjad-moradi .
Thanks @navina . This will be certainly useful for streams like PubSub.
From your doc I see ``commit` will be invoked whenever the controller requests the server to commit a segment`. More specifically, will the commit be invoked before server gets back to the controller for metadata commit or after?
If it's the latter, some server failure scenarios might get tricky? If the former, any gotcha failure scenarios to consider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #7988: Enhance Pinot Streaming Consumer Interface
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #7988:
URL: https://github.com/apache/pinot/issues/7988#issuecomment-1009474097
I hope we are discussing in the design document. As of now, it is very minimal. I have left some comments there. Please address them, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on issue #7988: Enhance Pinot Streaming Consumer Interface
Posted by GitBox <gi...@apache.org>.
navina commented on issue #7988:
URL: https://github.com/apache/pinot/issues/7988#issuecomment-1009468977
@npawar : The commit will be invoked *before* the server gets back to the controller. Basically, when the server enters `COMMIT` state, it will first checkpoint to the stream, followed by building a segment and responding to the controller with metadata.
Checkpoint is used as a "marker" in the stream. So, even if the server fails or enters commit state again, it should re-create (or use the existing) checkpoint and return the new metadata to the controller.
I can't think of a gotcha failure scenarios as of now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar closed issue #7988: Enhance Pinot Streaming Consumer Interface
Posted by GitBox <gi...@apache.org>.
npawar closed issue #7988:
URL: https://github.com/apache/pinot/issues/7988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org