You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Aditya Kishore <ad...@gmail.com> on 2014/05/29 08:15:16 UTC
Review Request 22013: DRILL-855: Improve work assignment parallelization
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22013/
-----------------------------------------------------------
Review request for drill, Jacques Nadeau and Steven Phillips.
Repository: drill-git
Description
-------
Currently if only a small number of endpoints have data affinity, all the work units get assigned to this small subset leaving other endpoints in the cluster underutilized.
It will lead to poor query performance since the level of parallelization is reduced below what is otherwise possible.
This JIRA intend to modify the assignment to do a slightly more equitable distribution of work fragments among all available nodes while slightly favoring the nodes with data affinity.
This *favor* is controlled by the setting "drill.exec.work.affinity.factor" (defaults to 1.2f).
As an example, let's say that there are 2 endpoints of which only one is with affinity and 10 work fragments. In the current scheme of things, all of these work fragments will get assigned to the single node with affinity. With the modified scheme and with default affinity factor of 1.2, it will get 20% more assignment that it would get under equal distribution (i.e. 6:4 instead of 5:5).
Diffs
-----
distribution/src/resources/drill-override.conf a9316a9
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java d9e0833
exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java f3059ae
exec/java-exec/src/main/java/org/apache/drill/exec/planner/SimpleExecPlanner.java 4da6500
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java 313a81d
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 8602bf0
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java eb1d738
exec/java-exec/src/main/resources/drill-module.conf 26205bd
exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 1b38dce
Diff: https://reviews.apache.org/r/22013/diff/
Testing
-------
Thanks,
Aditya Kishore
Re: Review Request 22013: DRILL-855: Improve work assignment parallelization
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22013/#review44243
-----------------------------------------------------------
exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java
<https://reviews.apache.org/r/22013/#comment78580>
Can we at least have a unit test testing exactly what you added?
I don't see where we verify the behavior you described.
- Timothy Chen
On May 29, 2014, 6:15 a.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22013/
> -----------------------------------------------------------
>
> (Updated May 29, 2014, 6:15 a.m.)
>
>
> Review request for drill, Jacques Nadeau and Steven Phillips.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Currently if only a small number of endpoints have data affinity, all the work units get assigned to this small subset leaving other endpoints in the cluster underutilized.
>
> It will lead to poor query performance since the level of parallelization is reduced below what is otherwise possible.
>
> This JIRA intend to modify the assignment to do a slightly more equitable distribution of work fragments among all available nodes while slightly favoring the nodes with data affinity.
>
> This *favor* is controlled by the setting "drill.exec.work.affinity.factor" (defaults to 1.2f).
>
> As an example, let's say that there are 2 endpoints of which only one is with affinity and 10 work fragments. In the current scheme of things, all of these work fragments will get assigned to the single node with affinity. With the modified scheme and with default affinity factor of 1.2, it will get 20% more assignment that it would get under equal distribution (i.e. 6:4 instead of 5:5).
>
>
> Diffs
> -----
>
> distribution/src/resources/drill-override.conf a9316a9
> exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java d9e0833
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java f3059ae
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/SimpleExecPlanner.java 4da6500
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java 313a81d
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 8602bf0
> exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java eb1d738
> exec/java-exec/src/main/resources/drill-module.conf 26205bd
> exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 1b38dce
>
> Diff: https://reviews.apache.org/r/22013/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 22013: DRILL-855: Improve work assignment parallelization
Posted by Aditya Kishore <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22013/#review44251
-----------------------------------------------------------
exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java
<https://reviews.apache.org/r/22013/#comment78599>
Will add a test as part of HBase scan distribution that I am working atm.
- Aditya Kishore
On May 28, 2014, 11:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22013/
> -----------------------------------------------------------
>
> (Updated May 28, 2014, 11:15 p.m.)
>
>
> Review request for drill, Jacques Nadeau and Steven Phillips.
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Currently if only a small number of endpoints have data affinity, all the work units get assigned to this small subset leaving other endpoints in the cluster underutilized.
>
> It will lead to poor query performance since the level of parallelization is reduced below what is otherwise possible.
>
> This JIRA intend to modify the assignment to do a slightly more equitable distribution of work fragments among all available nodes while slightly favoring the nodes with data affinity.
>
> This *favor* is controlled by the setting "drill.exec.work.affinity.factor" (defaults to 1.2f).
>
> As an example, let's say that there are 2 endpoints of which only one is with affinity and 10 work fragments. In the current scheme of things, all of these work fragments will get assigned to the single node with affinity. With the modified scheme and with default affinity factor of 1.2, it will get 20% more assignment that it would get under equal distribution (i.e. 6:4 instead of 5:5).
>
>
> Diffs
> -----
>
> distribution/src/resources/drill-override.conf a9316a9
> exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java d9e0833
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java f3059ae
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/SimpleExecPlanner.java 4da6500
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java 313a81d
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 8602bf0
> exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java eb1d738
> exec/java-exec/src/main/resources/drill-module.conf 26205bd
> exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 1b38dce
>
> Diff: https://reviews.apache.org/r/22013/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>