[GitHub] erankor opened a new issue #6963: Tuning Kafka Indexing Service for real time

[GitBox <gi...@apache.org>]

erankor opened a new issue #6963: Tuning Kafka Indexing Service for real time
URL: https://github.com/apache/incubator-druid/issues/6963
 
 
   Hi all,
   
   Sorry if this is already answered somewhere, tried to google for it and didn't find a clear explanation...
   
   One of our data sources is highly real time in nature = most queries include an interval very close to the live edge. 
   We are using:
   - KIS for ingestion
   - Query granularity of 10 sec (1 sec in Druid, rounded to 10 sec outside druid)
   - Segment granularity of 1 hour
   
   Most of our 1 hour segments fall in the range 300-900 MB after merging the partitions using Hadoop. This is very close to recommendations, which probably means the 1 hour segment granularity is not a very bad choice. 
   However, since most queries touch the live edge, I think it may be beneficial to have KIS generate smaller segments (maybe 5 min), in order to have a larger portion of the query answered by finalized/immutable segments. These segments will later be merged anyway in offline, in order to get them close to the recommended size.
   
   My questions are - 
   1. Does this logic make sense?
   2. What is the relation between the KIS taskDuration and segmentGranularity? do I need to change both? what happens if segmentGranularity is 5 min while taskDuration is 1 hour?
   3. Read in a few places that windowPeriod is not needed for KIS, if I change the segment duration to 5 min and an event arrives 2 min late, what happens with it?
   
   Thank you!
   
   Eran

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org