[GitHub] jihoonson commented on issue #6336: update insert pending segments logic to synchronous

[GitBox <gi...@apache.org>]

jihoonson commented on issue #6336: update insert pending segments logic to synchronous
URL: https://github.com/apache/incubator-druid/pull/6336#issuecomment-450603775
 
 
   @QiuMM thank you for reporting! I think `TaskLockbox` potentially has this problem for all methods because they are coordinated by a single `giant` lock. I guess this becomes significant in large clusters when `allocatePendingSegment` is executed inside of `doInCriticalSection` because it's a heavy method.
   
   I think this PR is still useful because we can avoid the slow pending segment allocation problem caused by the race in `allocatePendingSegment`, but the solution may need to be improved. How about improving the lock granularity for taskLockbox? There's no need to use a single lock, but it can use more granular lock per dataSource per interval.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org