[GitHub] [incubator-druid] justinborromeo edited a comment on issue #7238: [PROPOSAL] Add segment limit for native and streaming index tasks

[GitBox <gi...@apache.org>]

justinborromeo edited a comment on issue #7238: [PROPOSAL] Add segment limit for native and streaming index tasks
URL: https://github.com/apache/incubator-druid/issues/7238#issuecomment-481041614
 
 
   @jihoonson I think `maxRowsPerSegment`, `maxTotalRows`, and `maxTotalSegments` are all necessary since they protect against different cases.
   
   - `maxRowsPerSegment` exists so that a specific segment doesn't have too much data
   - `maxTotalRows` exists for the case where there is a moderately high number of segments each with a large amount of data
   - `maxTotalSegments` exists for the cases when there is an extremely high number of segments with a small amount of data
   
   As an example, if we use the `maxTotalRows = maxTotalSegments * maxRowsPerSegment` calculation with the existing 5M `maxRowsPerSegment` default and the proposed 1K `maxTotalSegments` default, our calculated `maxTotalRows` is 5B which seems a bit high.
   
   Do you agree?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org