[GitHub] [incubator-druid] zhangxinyu1 commented on issue #7119: [materialized view] The last day of month and the last 24h are not ingested by the Supervisor

[GitBox <gi...@apache.org>]

zhangxinyu1 commented on issue #7119: [materialized view] The last day of month and the last 24h are not ingested by the Supervisor
URL: https://github.com/apache/incubator-druid/issues/7119#issuecomment-469101926
 
 
   If the data of base dataSource is ingested by batch ingestion, a small value of `minDataLagMs` is suggested. e.g. 0.
   
   However, if the data of base dataSource is ingested by stream ingestion, e.g. kafka-indexing-servcie, the segments of based dataSource are generated each `taskDuration`. 
   
   If the value of `minDataLagMs` is too small, the derivative dataSource supervisor will submit tasks frequently along with the new ingested segments of base dataSource. I think the frequently submitted tasks may waste computing resource, so `minDataLagMs` is used to avoid a large number of tasks.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org