[GitHub] [incubator-pinot] fpj commented on issue #5928: Add connector for Pravega

[GitBox <gi...@apache.org>]

fpj commented on issue #5928:
URL: https://github.com/apache/incubator-pinot/issues/5928#issuecomment-698823572


   Hi @npawar,  I missed your message, sorry about that. 
   
   > is "set of segments" like partitions within the stream?
   
   yes, but we do not expose segments individually. a group of readers coordinate among themselves to assign enabled segments. keep in mind that the set of segments in a pravega stream can change over time because of stream scaling.
   
    > Can we use them to instead add a PartitionLevelConsumer implementation?
   
   We can't with the stream API because of the reason I stated above. The Batch API, which does not follow any order of segments and simply gives iterators for a stream, can be used, is more aligned with what you are asking for, but it is not event driven. The iterators will read the events up to a point in the stream.
   
    > Is there any doc where i can read the detailed design of Pravega stream APIs and particularly the checkpointing?
   
   You may want to explore the documentation here:
   
   https://pravega.io/docs/latest/
   
   and in particular the "Developing Pravega Applications" section. You may also want to check the javadocs:
   
   https://pravega.io/docs/latest/javadoc/clients/io/pravega/client/stream/ReaderGroup.html#initiateCheckpoint-java.lang.String-java.util.concurrent.ScheduledExecutorService-
   
   The best example currently for the use of checkpoints is Apache Flink. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org