[GitHub] [druid] aditya-r-m commented on issue #5773: gcloud pubsub support

[GitBox <gi...@apache.org>]

aditya-r-m commented on issue #5773: gcloud pubsub support
URL: https://github.com/apache/druid/issues/5773#issuecomment-611474005
 
 
   @marklu2018 agreed, you are right in that ordering won't be needed to ingest data into druid from pubsub.
   
   To brief you on the context, in the druid code base, SeekableStream supervisor service relies on ordering of packets & sequence numbers for checkpointing. & task replicas are created to ensure that all the partitions are reliably being read from.
   
   Pubsub being a global queue without any hardwired partitioning, we will not need task replicas & thus, sequence number based check-pointing shouldn't be required. Sensible acks should do the job.
   
   This has implications on how pubsub indexing implementation will differ from Kafka & Kinesis. So we can't derive another child class from SeekableStream package & use it for Pubsub as was suggested by @glasser.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org