[GitHub] [incubator-pinot] Aka-shi commented on issue #6302: [pinot]Support for pausing the realtime consumption without disabling the table.

[GitBox <gi...@apache.org>]

Aka-shi commented on issue #6302:
URL: https://github.com/apache/incubator-pinot/issues/6302#issuecomment-778922524


   @mcvsubbu 
   >Would you want all partitions of the stream to pause , or just some specific partitions??
   
   For all partitions. The consumption for the table itself should be stopped. I was thinking something like this. 
   1. User pauses stream -> pinot server commits the current consuming segments(for all partitions) along with offsets. 
   2. User resumes stream -> Pinot starts consumption from all partitions from previously committed offsets. 
   
   >Would you want all partitions/tables in a given server to pause?
   
   I was expecting the pause option to be at a table level. Because of this #6555 . If the pause and reset APIs are available at a table level, then the user can pause the current stream(which would commit current segments and pause the stream), reset the offsets, and resume consumption  from earliest/latest offsets as per config. 
   
   >After a pause, if the server is restarted, how do you desire the server should come back up? Should it consume up to the exact same paused place again? Should it simply not consume after the last completed/committed point? Should it forget that it was paused?
   
   If a table is paused and server restarts after it, then considering the previous consuming segments were already committed when the table is paused, I would expect the server to not consume after the restart too. My understanding is when we pause the stream, we are changing the state of the table and it should not start consuming until the user resumes the stream himself. 
   
   >Further if there are multiple replicas, do you want them all to pause at the same place after a pause command?
   
   Yes. That's what I would expect when I pause the stream. No replica of any partition consumer should be active is what I feel. 
   
   PS: Just putting it out here. When we pause the table, the table should still be available for querying the already consumed data. If not, this would be more or less like the enable/disable API. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org