You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yunikorn.apache.org by "Craig Condit (Jira)" <ji...@apache.org> on 2021/08/19 15:03:00 UTC
[jira] [Created] (YUNIKORN-807) Improve performance of node sorting
Craig Condit created YUNIKORN-807:
-------------------------------------
Summary: Improve performance of node sorting
Key: YUNIKORN-807
URL: https://issues.apache.org/jira/browse/YUNIKORN-807
Project: Apache YuniKorn
Issue Type: Improvement
Components: core - scheduler
Reporter: Craig Condit
Assignee: Craig Condit
Attachments: Node Sorting Performance Improvement.pdf
YuniKorn currently sorts all nodes on demand whenever scheduling of a container occurs. This causes significant performance degradation as the number of nodes increases.
If we replace the on-demand sorting with a B-Tree sorted proactively, we can improve performance considerably.
This is a similar approach to YUNIKORN-21, but without the associated behavioral changes.
I've attached a design document with the details of the approach and the performance improvement gained.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org
Re: Design review request: (YUNIKORN-807) Improve performance of node sorting
Posted by Weiwei Yang <ww...@apache.org>.
Hi Craig
The result looks amazing, this is a long wanted optimization (more than 1
year?)
I will spend some time this week reviewing the PR, thank you for the
efforts!
On Thu, Aug 19, 2021 at 8:24 AM Craig Condit <ap...@craigcondit.com> wrote:
> Just wanted to bring this to the attention of the community due to the
> large performance delta we observing with the new patch (30-70x in some
> cases). I’ve tried to keep the changes as minimal as possible, but there
> are some refactorings done to allow the sorting code to be more localized.
> This is (IMO) a simplified approach vs. YUNIKORN-21 and hopefully carries
> less risk. Any review or comments would be greatly appreciated.
>
> Thanks,
>
> Craig Condit
>
>
> > On Aug 19, 2021, at 10:03 AM, Craig Condit (Jira) <ji...@apache.org>
> wrote:
> >
> > Craig Condit created YUNIKORN-807:
> > -------------------------------------
> >
> > Summary: Improve performance of node sorting
> > Key: YUNIKORN-807
> > URL: https://issues.apache.org/jira/browse/YUNIKORN-807
> > Project: Apache YuniKorn
> > Issue Type: Improvement
> > Components: core - scheduler
> > Reporter: Craig Condit
> > Assignee: Craig Condit
> > Attachments: Node Sorting Performance Improvement.pdf
> >
> > YuniKorn currently sorts all nodes on demand whenever scheduling of a
> container occurs. This causes significant performance degradation as the
> number of nodes increases.
> >
> > If we replace the on-demand sorting with a B-Tree sorted proactively, we
> can improve performance considerably.
> >
> > This is a similar approach to YUNIKORN-21, but without the associated
> behavioral changes.
> >
> > I've attached a design document with the details of the approach and the
> performance improvement gained.
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
> > For additional commands, e-mail: dev-help@yunikorn.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
> For additional commands, e-mail: dev-help@yunikorn.apache.org
>
>
Design review request: (YUNIKORN-807) Improve performance of node
sorting
Posted by Craig Condit <ap...@craigcondit.com>.
Just wanted to bring this to the attention of the community due to the large performance delta we observing with the new patch (30-70x in some cases). I’ve tried to keep the changes as minimal as possible, but there are some refactorings done to allow the sorting code to be more localized. This is (IMO) a simplified approach vs. YUNIKORN-21 and hopefully carries less risk. Any review or comments would be greatly appreciated.
Thanks,
Craig Condit
> On Aug 19, 2021, at 10:03 AM, Craig Condit (Jira) <ji...@apache.org> wrote:
>
> Craig Condit created YUNIKORN-807:
> -------------------------------------
>
> Summary: Improve performance of node sorting
> Key: YUNIKORN-807
> URL: https://issues.apache.org/jira/browse/YUNIKORN-807
> Project: Apache YuniKorn
> Issue Type: Improvement
> Components: core - scheduler
> Reporter: Craig Condit
> Assignee: Craig Condit
> Attachments: Node Sorting Performance Improvement.pdf
>
> YuniKorn currently sorts all nodes on demand whenever scheduling of a container occurs. This causes significant performance degradation as the number of nodes increases.
>
> If we replace the on-demand sorting with a B-Tree sorted proactively, we can improve performance considerably.
>
> This is a similar approach to YUNIKORN-21, but without the associated behavioral changes.
>
> I've attached a design document with the details of the approach and the performance improvement gained.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
> For additional commands, e-mail: dev-help@yunikorn.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org