You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/05/26 05:46:34 UTC
[GitHub] [druid] suneet-s opened a new pull request, #12571: Set druid.processing.fifo to true by default
suneet-s opened a new pull request, #12571:
URL: https://github.com/apache/druid/pull/12571
`druid.processing.fifo=true` seems like a better default for more predictable performance when tasks of equal priority are placed on the processing queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm merged pull request #12571: Set druid.processing.fifo to true by default
Posted by GitBox <gi...@apache.org>.
gianm merged PR #12571:
URL: https://github.com/apache/druid/pull/12571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on pull request #12571: Set druid.processing.fifo to true by default
Posted by GitBox <gi...@apache.org>.
suneet-s commented on PR #12571:
URL: https://github.com/apache/druid/pull/12571#issuecomment-1177850920
Thanks for the review @capistrant I found this blog - https://metamarkets.com/2016/impact-on-query-speed-from-forced-processing-ordering-in-druid/ - that talked about the concerns mentioned by the implementer.
I think if we call out this change in the release notes, then operators with heterogeneous workloads can change this setting if they have long running queries mixed in with hot queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] capistrant commented on pull request #12571: Set druid.processing.fifo to true by default
Posted by GitBox <gi...@apache.org>.
capistrant commented on PR #12571:
URL: https://github.com/apache/druid/pull/12571#issuecomment-1178199675
> Thanks for the review @capistrant I found this blog - https://metamarkets.com/2016/impact-on-query-speed-from-forced-processing-ordering-in-druid/ - that talked about the concerns mentioned by the implementer.
>
> I think if we call out this change in the release notes, then operators with heterogeneous workloads can change this setting if they have long running queries mixed in with hot queries.
From the blog: So the lesson learned here is when there is a large backlog of work, having orderly execution helps everything get through in a predictable manner. However, when speed and nimbleness are key, let chaos reign.
This note makes me sway back to keeping this configuration disabled by default. If enabling hurt the interactive query performance of a prod cluster, it would make me think twice about enabling by default.
However the blog is from 2016 and the druid version was 0.9 at the time. Perhaps there is a need to re-create that study with current Druid to see if the results tell the same story.
For now, I will have to revoke the approval out of doubt about this after reading the blog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] imply-cheddar commented on pull request #12571: Set druid.processing.fifo to true by default
Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on PR #12571:
URL: https://github.com/apache/druid/pull/12571#issuecomment-1206683385
I'm approving this. While the blog post does show a bimodal distribution where it looks like queries are faster, what we've seen most recently is that this bimodal distribution is effectively non-deterministic. I.e. some queries get impacted, others don't. When we are running Druid, supporting Druid, using Druid in product contexts, non-determinism basically equates to an inability to reason about how the system will behave. So, we have fallen into a common pattern
1) Be told that a cluster experiences slow queries sometimes
2) Look at metrics and identify that there is a non-deterministic case of queries being randomly delayed in query/wait, but only for a subset of segments and not holistically
3) Be like, "omg, is it fifo again?" and then be like, "oh yeah, we have no clue because it's non-deterministic"
4) Well, let's set it to fifo and then see how things work out
5) Now that fifo is set to true, we can actually aee what's going on, yay!
6) Improve the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] capistrant commented on pull request #12571: Set druid.processing.fifo to true by default
Posted by GitBox <gi...@apache.org>.
capistrant commented on PR #12571:
URL: https://github.com/apache/druid/pull/12571#issuecomment-1176658417
https://github.com/apache/druid/pull/1809
Went back in time to see if there was reasoning behind setting it to false by default. It seems like that was done simply to leave default behavior the same across versions. There is an note in there from implementer in PR description:
> One downside to the approach in this PR is that a particularly long-running query has the ability to starve out cluster resources, where the prior implementation could interleave (though randomly) other tasks into a long-running query.
This seems like a reasonable change in default to me, but I do wonder how many people may notice odd side effects from this after upgrade and struggle to put their finger on the cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org