You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/05/27 21:02:40 UTC
[GitHub] [druid] kroeders opened a new issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
kroeders opened a new issue #9940:
URL: https://github.com/apache/druid/issues/9940
### Description
In [KillUnusedSegmentsTask](https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java) unused segments are deleted in serial. The proposal is to use an ExecutorService to kill the segments in parallel. A numThreads parameter will be added to the task spec with a default to 1 so that existing functionality is not changed. This parameter is also exposed in the URL parameters used by the UI, e.g. /coordinator/v1/datasources/wikipedia1?kill=true&interval=1000/3000&numThreads=10
A fork and test results will be added here soon.
### Motivation
In deployments with very large numbers of segments, deleting unused segments can be painfully slow. This is particularly noticeable in large HDFS deployments as each request to the cluster could be parallelized in a straightforward way. This was a client request and it would be nice to hear from others with similar issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kroeders commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
kroeders commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-675072728
migrating over to the other issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kroeders commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
kroeders commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-653153635
Hi again @a2l007 @jihoonson @FrankChen021 I got access to our staging cluster and put together a [prototype](https://github.com/apache/druid/compare/master...kroeders:feature-parallel-killtask) and did some testing and so I wanted to share what I found.
The test cluster is reasonably large with some datasources in the TB range. I did tests up to around a few hundred GB and segments up to 500MB and did tests up to a few thousand segments per datasource. I did scaled down testing with local storage as well. I found that the delete times were very fast and for HDFS not proportional to segment size - which makes sense as an HDFS delete is a namenode operation. I saw an expected speedup running in parallel, but overall the deletes were very fast, around 10ms to delete a segment in our cluster.
In cases where there is more of a delay I saw the expected speedup, but I had to put an artificial delay to see an appreciable difference. I'm still investigating with users whether anyone is still experiencing the original issue, but it looks like changes at the HDFS layer have improved the situation dramatically.
Regarding the discussion about resource scheduling, I think that delete segments would be a different type than the parallel subtask for indexing. These deep storage tasks are delegating to another service so I think it makes sense to let them run in parallel, even for local storage cases.
For the kill task specifically, I also saw the discussion with problems related to a very large number of segments aside from accumulated deep storage delays. I noticed in the parallel indexing tasks, we still end up with task items in the ingestion UI. It would be pretty straightforward to set a threshold/batch size and just reissue independent kill tasks without introducing a new subtask framework. Maybe these are two general "dials" in a general resource management framework, number of segments per task and number of concurrent deep storage operations?
Jihoon, regarding your question on druid.coordinator.kill.on - I asked our users and they indicated kill tasks are issued manually primarily for policy reasons. Apparently, under certain situation there are rules regarding the retention of data so there are regular schedules for pulling and deleting data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kroeders closed issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
kroeders closed issue #9940:
URL: https://github.com/apache/druid/issues/9940
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 edited a comment on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
FrankChen021 edited a comment on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-636332701
I think the task slot model( a task per process) is too heavy.
Issuing multiple tasks simultaneously reaches the same goal as an executor service proposed here, but the problem of task slot is the number of task slots are fixed at the configuration file level. As the datasources grows, and the index task grows, there're less task slots left to do the compact or kill tasks. If we want to balance it, we have to enlarge the number of task slots and reboot the middle manager. This is the problem, if druid could provide a configuration center, to change the number of slots would not involve rebooting middle managers.
In this scenario, operators could breaks down the kill task by smaller intervals to utilize multiple task slots. But from the user's view, user should not care about how to utilize these task slots, they just want that job to finish ASAP, so the break down should leave to druid. I think this is why this proposal is here.
Considering the task model here, maybe another viable way is the overlord breaks down the kill task into smaller tasks, each of which perform a kill in a smaller intervals.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-634940806
Hi @kroeders thank you for raising the issue. Can you tell me what the benefits of using multiple threads are vs issuing multiple kill tasks? In Druid's indexing service, the resource allocated per task is abstracted as the _task slot_. Multiple task slots can be configured per middleManager and a task occupies one task slot when it is assigned to a middleManager. This model is to simplify resource control and management.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-640714445
@jihoonson I agree with you, it's druid's responsibility to break down a large task into smaller tasks and schedule these tasks simultaneously.
One of what limits this idea is the static configuration of number of task slots. Can the numbers of subtasks plus the number of task being executed exceed the number of task slots ? If can't, what if there's only one task slot left at the time a large kill task is submitted? It involves many details.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
a2l007 commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-642111692
@jihoonson I believe the rationale behind having a multi-threaded Kill Task is with regard to clusters that heavily rely on just Hadoop based batch indexing. In this case, the cluster may not have several middlemanagers or the middlemanagers may not be tuned for native parallel indexing and thus might not have several task slots. If we end up making kill tasks spawn subtasks, it might use up more task slots which can cause subsequent indexing tasks to wait.
I agree that your idea of a task slot model for kill tasks is better in terms of fine grained resource management and is in line with the community's effort towards native parallel indexing tasks, but considering the usecase I mentioned above, it might be useful to support multi threading within a Kill Task itself as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kroeders commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
kroeders commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-635367098
hi @jihoonson thank you for the feedback, I think this is the key design issue here. For some context, we run an internal fork of Druid where the users are in another team. From the user perspective they're regularly cleaning up stale data. At a high level the problem statement is that a logical kill operation (e.g. delete all traffic from the last day) leads to thousands of segments being deleted in serial which is visibly slow.
So, we could check the segment count and issue multiple kill tasks (specifying a batch size parameter instead of a thread count parameter) or even one task per segment. As a general rule I want to use existing concurrency features whenever possible, but if I understand Tasks properly they seem pretty heavyweight, being launched into other processes and visible in the ingestion window. I didn't want to end up in a degenerate case where there was one task per segment, so we looked at a more lightweight approach to speeding up this task. From a user perspective, parallelizing inside the task keeps the UI output in synch with the user input and uses minimal overhead
I'm new to both Druid and this team, but we have a pretty large existing cluster with HDFS deep storage. Delete should just be issuing a request to HDFS and a pretty lightweight operation for Druid when configured this way - local storage will be a different story.
I should have access to a dev cluster shortly where I can provide metrics and try different approaches.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-642347494
@a2l007 @jihoonson This is the problem I worry about. Number of task slot is fixed at configuration level. Currently, there's no way to enlarge the number of task slots without rebooting middle managers to take the new number in effect, which means tuning the number is not easy in production. And In order not to block the index tasks, a complicated algorithm is needed.
I think threading model is a simple enough way but without violating any current design principles.
Task slots provides a way to parallelize the task between middle managers, threading model provides a way to parallel the task inside a task. They're not conflict.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-640696940
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-636332701
I think the task slot model( a task per process) is too heavy.
Issuing multiple tasks simultaneously reaches the same goal as an executor service proposed here, but the problem of task slot is the number of task slots are fixed at the configuration file level. As the datasources grows, and the index task grows, there're less task slots left to do the compact or kill tasks. If we want to balance it, we have to enlarge the number of task slots and reboot the middle manager. This is the problem, if druid could provide a configuration center, to change the number of slots would not involve rebooting middle managers.
In this scenario, operators could breaks down the kill task by smaller intervals to utilize multiple task slots. But from the user's view, user should not care about how to utilize these task slots, they just want to that job finishes ASAP, so the break down should leave to druid. I think this is why this proposal is here.
Considering the task model here, maybe another viable way is the overlord breaks down the kill task into smaller tasks, each of which perform a kill in a smaller intervals.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 edited a comment on issue #9940: Improve Performance With Multithreaded Kill Unused Segments Task
Posted by GitBox <gi...@apache.org>.
FrankChen021 edited a comment on issue #9940:
URL: https://github.com/apache/druid/issues/9940#issuecomment-636332701
I think the task slot model( a task per process) is too heavy.
Issuing multiple tasks simultaneously reaches the same goal as an executor service proposed here, but the problem of task slot is the number of task slots are fixed at the configuration file level. As the datasources grows, and the index task grows, there're less task slots left to do the compact or kill tasks. If we want to balance it, we have to enlarge the number of task slots and reboot the middle manager. This is the problem, if druid could provide a configuration center, to change the number of slots would not involve rebooting middle managers.
In this scenario, operators could breaks down the kill task by smaller intervals to utilize multiple task slots. But from the user's view, user should not care about how to utilize these task slots, they just want that job to finish ASAP, so the break down should leave to druid. I think this is why this proposal is here.
Considering the task model here, maybe another viable way is the overlord breaks down the kill task into smaller tasks, each of which performs a kill in a smaller intervals.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org