You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Jonathan Hung <jy...@gmail.com> on 2019/03/29 19:14:05 UTC
[DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Hello devs,
Starting a discuss thread to merge resource types/native GPU scheduling
support to branch-3.0 and branch-2. The resource types work was done in
trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
is to merge GPU support into branch-3.0 and both resource types/GPU support
to branch-2.
Internally we've been running resource types/GPU support off a fork of
branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
well. Also for completeness we verified that everything going into branch-2
also exists in branch-3.0.
The specific list of patches to merge is in feature branch
YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
If there's no issues from the community I'll start a merge vote next week.
Thanks.
Jonathan Hung
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Thanks for following up on the rolling upgrades!
We are definitely interested in seeing this move forward.
On Wed, Aug 21, 2019 at 2:37 PM Jonathan Hung <jy...@gmail.com> wrote:
> Reviving this thread: we tested YARN RU starting with a cluster running
> 2.7.4, to running branch-2 + YARN-8200. Ran some simple MR/Spark jobs
> concurrently with the RM/NM upgrades and did not see any issues.
>
> If no other concerns I'll continue with a vote.
>
> Jonathan Hung
>
>
> On Thu, Apr 18, 2019 at 5:12 PM Jonathan Hung <jy...@gmail.com>
> wrote:
>
>> Sorry for the delay, had to deprioritize this. Hoping to get to this next
>> week.
>>
>> Jonathan
>>
>> ------------------------------
>> *From:* Jim Brennan <ja...@verizonmedia.com>
>> *Sent:* Thursday, April 18, 2019 7:28 AM
>> *To:* Jonathan Hung
>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>
>> Hi Jonathan,
>>
>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>>> issue, but we’ll try it out and report back.
>>
>>
>> Any update on this?
>> Jim
>>
>>
>> On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being
>>> an issue, but we’ll try it out and report back.
>>>
>>> Jonathan
>>>
>>> ------------------------------
>>> *From:* Jim Brennan <ja...@verizonmedia.com>
>>> *Sent:* Tuesday, April 2, 2019 9:17 AM
>>> *To:* Jonathan Hung
>>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>>
>>> Thanks for working on this!
>>> One concern for us is support for a rolling upgrade. If we are running
>>> a cluster based on branch-2.8, will we be able to do a rolling upgrade (no
>>> cluster down-time) to a branch containing these changes? Have you tested
>>> rolling upgrades?
>>>
>>> Thanks.
>>> Jim
>>>
>>> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
>>> wrote:
>>>
>>>> Hello devs,
>>>>
>>>> Starting a discuss thread to merge resource types/native GPU scheduling
>>>> support to branch-3.0 and branch-2. The resource types work was done in
>>>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the
>>>> proposal
>>>> is to merge GPU support into branch-3.0 and both resource types/GPU
>>>> support
>>>> to branch-2.
>>>>
>>>> Internally we've been running resource types/GPU support off a fork of
>>>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has
>>>> worked
>>>> well. Also for completeness we verified that everything going into
>>>> branch-2
>>>> also exists in branch-3.0.
>>>>
>>>> The specific list of patches to merge is in feature branch
>>>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>>>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>>>> diff
>>>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>>>
>>>> If there's no issues from the community I'll start a merge vote next
>>>> week.
>>>> Thanks.
>>>>
>>>> Jonathan Hung
>>>>
>>>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Thanks for following up on the rolling upgrades!
We are definitely interested in seeing this move forward.
On Wed, Aug 21, 2019 at 2:37 PM Jonathan Hung <jy...@gmail.com> wrote:
> Reviving this thread: we tested YARN RU starting with a cluster running
> 2.7.4, to running branch-2 + YARN-8200. Ran some simple MR/Spark jobs
> concurrently with the RM/NM upgrades and did not see any issues.
>
> If no other concerns I'll continue with a vote.
>
> Jonathan Hung
>
>
> On Thu, Apr 18, 2019 at 5:12 PM Jonathan Hung <jy...@gmail.com>
> wrote:
>
>> Sorry for the delay, had to deprioritize this. Hoping to get to this next
>> week.
>>
>> Jonathan
>>
>> ------------------------------
>> *From:* Jim Brennan <ja...@verizonmedia.com>
>> *Sent:* Thursday, April 18, 2019 7:28 AM
>> *To:* Jonathan Hung
>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>
>> Hi Jonathan,
>>
>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>>> issue, but we’ll try it out and report back.
>>
>>
>> Any update on this?
>> Jim
>>
>>
>> On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being
>>> an issue, but we’ll try it out and report back.
>>>
>>> Jonathan
>>>
>>> ------------------------------
>>> *From:* Jim Brennan <ja...@verizonmedia.com>
>>> *Sent:* Tuesday, April 2, 2019 9:17 AM
>>> *To:* Jonathan Hung
>>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>>
>>> Thanks for working on this!
>>> One concern for us is support for a rolling upgrade. If we are running
>>> a cluster based on branch-2.8, will we be able to do a rolling upgrade (no
>>> cluster down-time) to a branch containing these changes? Have you tested
>>> rolling upgrades?
>>>
>>> Thanks.
>>> Jim
>>>
>>> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
>>> wrote:
>>>
>>>> Hello devs,
>>>>
>>>> Starting a discuss thread to merge resource types/native GPU scheduling
>>>> support to branch-3.0 and branch-2. The resource types work was done in
>>>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the
>>>> proposal
>>>> is to merge GPU support into branch-3.0 and both resource types/GPU
>>>> support
>>>> to branch-2.
>>>>
>>>> Internally we've been running resource types/GPU support off a fork of
>>>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has
>>>> worked
>>>> well. Also for completeness we verified that everything going into
>>>> branch-2
>>>> also exists in branch-3.0.
>>>>
>>>> The specific list of patches to merge is in feature branch
>>>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>>>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>>>> diff
>>>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>>>
>>>> If there's no issues from the community I'll start a merge vote next
>>>> week.
>>>> Thanks.
>>>>
>>>> Jonathan Hung
>>>>
>>>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Reviving this thread: we tested YARN RU starting with a cluster running
2.7.4, to running branch-2 + YARN-8200. Ran some simple MR/Spark jobs
concurrently with the RM/NM upgrades and did not see any issues.
If no other concerns I'll continue with a vote.
Jonathan Hung
On Thu, Apr 18, 2019 at 5:12 PM Jonathan Hung <jy...@gmail.com> wrote:
> Sorry for the delay, had to deprioritize this. Hoping to get to this next
> week.
>
> Jonathan
>
> ------------------------------
> *From:* Jim Brennan <ja...@verizonmedia.com>
> *Sent:* Thursday, April 18, 2019 7:28 AM
> *To:* Jonathan Hung
> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>
> Hi Jonathan,
>
> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>> issue, but we’ll try it out and report back.
>
>
> Any update on this?
> Jim
>
>
> On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com> wrote:
>
>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>> issue, but we’ll try it out and report back.
>>
>> Jonathan
>>
>> ------------------------------
>> *From:* Jim Brennan <ja...@verizonmedia.com>
>> *Sent:* Tuesday, April 2, 2019 9:17 AM
>> *To:* Jonathan Hung
>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>
>> Thanks for working on this!
>> One concern for us is support for a rolling upgrade. If we are running a
>> cluster based on branch-2.8, will we be able to do a rolling upgrade (no
>> cluster down-time) to a branch containing these changes? Have you tested
>> rolling upgrades?
>>
>> Thanks.
>> Jim
>>
>> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>>> Hello devs,
>>>
>>> Starting a discuss thread to merge resource types/native GPU scheduling
>>> support to branch-3.0 and branch-2. The resource types work was done in
>>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the
>>> proposal
>>> is to merge GPU support into branch-3.0 and both resource types/GPU
>>> support
>>> to branch-2.
>>>
>>> Internally we've been running resource types/GPU support off a fork of
>>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has
>>> worked
>>> well. Also for completeness we verified that everything going into
>>> branch-2
>>> also exists in branch-3.0.
>>>
>>> The specific list of patches to merge is in feature branch
>>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>>> diff
>>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>>
>>> If there's no issues from the community I'll start a merge vote next
>>> week.
>>> Thanks.
>>>
>>> Jonathan Hung
>>>
>>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Reviving this thread: we tested YARN RU starting with a cluster running
2.7.4, to running branch-2 + YARN-8200. Ran some simple MR/Spark jobs
concurrently with the RM/NM upgrades and did not see any issues.
If no other concerns I'll continue with a vote.
Jonathan Hung
On Thu, Apr 18, 2019 at 5:12 PM Jonathan Hung <jy...@gmail.com> wrote:
> Sorry for the delay, had to deprioritize this. Hoping to get to this next
> week.
>
> Jonathan
>
> ------------------------------
> *From:* Jim Brennan <ja...@verizonmedia.com>
> *Sent:* Thursday, April 18, 2019 7:28 AM
> *To:* Jonathan Hung
> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>
> Hi Jonathan,
>
> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>> issue, but we’ll try it out and report back.
>
>
> Any update on this?
> Jim
>
>
> On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com> wrote:
>
>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>> issue, but we’ll try it out and report back.
>>
>> Jonathan
>>
>> ------------------------------
>> *From:* Jim Brennan <ja...@verizonmedia.com>
>> *Sent:* Tuesday, April 2, 2019 9:17 AM
>> *To:* Jonathan Hung
>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>
>> Thanks for working on this!
>> One concern for us is support for a rolling upgrade. If we are running a
>> cluster based on branch-2.8, will we be able to do a rolling upgrade (no
>> cluster down-time) to a branch containing these changes? Have you tested
>> rolling upgrades?
>>
>> Thanks.
>> Jim
>>
>> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>>> Hello devs,
>>>
>>> Starting a discuss thread to merge resource types/native GPU scheduling
>>> support to branch-3.0 and branch-2. The resource types work was done in
>>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the
>>> proposal
>>> is to merge GPU support into branch-3.0 and both resource types/GPU
>>> support
>>> to branch-2.
>>>
>>> Internally we've been running resource types/GPU support off a fork of
>>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has
>>> worked
>>> well. Also for completeness we verified that everything going into
>>> branch-2
>>> also exists in branch-3.0.
>>>
>>> The specific list of patches to merge is in feature branch
>>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>>> diff
>>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>>
>>> If there's no issues from the community I'll start a merge vote next
>>> week.
>>> Thanks.
>>>
>>> Jonathan Hung
>>>
>>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Sorry for the delay, had to deprioritize this. Hoping to get to this next week.
Jonathan
________________________________
From: Jim Brennan <ja...@verizonmedia.com>
Sent: Thursday, April 18, 2019 7:28 AM
To: Jonathan Hung
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Hi Jonathan,
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an issue, but we’ll try it out and report back.
Any update on this?
Jim
On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com>> wrote:
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an issue, but we’ll try it out and report back.
Jonathan
________________________________
From: Jim Brennan <ja...@verizonmedia.com>>
Sent: Tuesday, April 2, 2019 9:17 AM
To: Jonathan Hung
Cc: yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>; mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Thanks for working on this!
One concern for us is support for a rolling upgrade. If we are running a cluster based on branch-2.8, will we be able to do a rolling upgrade (no cluster down-time) to a branch containing these changes? Have you tested rolling upgrades?
Thanks.
Jim
On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>> wrote:
Hello devs,
Starting a discuss thread to merge resource types/native GPU scheduling
support to branch-3.0 and branch-2. The resource types work was done in
trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
is to merge GPU support into branch-3.0 and both resource types/GPU support
to branch-2.
Internally we've been running resource types/GPU support off a fork of
branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
well. Also for completeness we verified that everything going into branch-2
also exists in branch-3.0.
The specific list of patches to merge is in feature branch
YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
If there's no issues from the community I'll start a merge vote next week.
Thanks.
Jonathan Hung
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Sorry for the delay, had to deprioritize this. Hoping to get to this next week.
Jonathan
________________________________
From: Jim Brennan <ja...@verizonmedia.com>
Sent: Thursday, April 18, 2019 7:28 AM
To: Jonathan Hung
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Hi Jonathan,
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an issue, but we’ll try it out and report back.
Any update on this?
Jim
On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com>> wrote:
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an issue, but we’ll try it out and report back.
Jonathan
________________________________
From: Jim Brennan <ja...@verizonmedia.com>>
Sent: Tuesday, April 2, 2019 9:17 AM
To: Jonathan Hung
Cc: yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>; mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Thanks for working on this!
One concern for us is support for a rolling upgrade. If we are running a cluster based on branch-2.8, will we be able to do a rolling upgrade (no cluster down-time) to a branch containing these changes? Have you tested rolling upgrades?
Thanks.
Jim
On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>> wrote:
Hello devs,
Starting a discuss thread to merge resource types/native GPU scheduling
support to branch-3.0 and branch-2. The resource types work was done in
trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
is to merge GPU support into branch-3.0 and both resource types/GPU support
to branch-2.
Internally we've been running resource types/GPU support off a fork of
branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
well. Also for completeness we verified that everything going into branch-2
also exists in branch-3.0.
The specific list of patches to merge is in feature branch
YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
If there's no issues from the community I'll start a merge vote next week.
Thanks.
Jonathan Hung
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Hi Jonathan,
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
> issue, but we’ll try it out and report back.
Any update on this?
Jim
On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com> wrote:
> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
> issue, but we’ll try it out and report back.
>
> Jonathan
>
> ------------------------------
> *From:* Jim Brennan <ja...@verizonmedia.com>
> *Sent:* Tuesday, April 2, 2019 9:17 AM
> *To:* Jonathan Hung
> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>
> Thanks for working on this!
> One concern for us is support for a rolling upgrade. If we are running a
> cluster based on branch-2.8, will we be able to do a rolling upgrade (no
> cluster down-time) to a branch containing these changes? Have you tested
> rolling upgrades?
>
> Thanks.
> Jim
>
> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
> wrote:
>
>> Hello devs,
>>
>> Starting a discuss thread to merge resource types/native GPU scheduling
>> support to branch-3.0 and branch-2. The resource types work was done in
>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
>> is to merge GPU support into branch-3.0 and both resource types/GPU
>> support
>> to branch-2.
>>
>> Internally we've been running resource types/GPU support off a fork of
>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
>> well. Also for completeness we verified that everything going into
>> branch-2
>> also exists in branch-3.0.
>>
>> The specific list of patches to merge is in feature branch
>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>> diff
>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>
>> If there's no issues from the community I'll start a merge vote next week.
>> Thanks.
>>
>> Jonathan Hung
>>
>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Hi Jonathan,
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
> issue, but we’ll try it out and report back.
Any update on this?
Jim
On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com> wrote:
> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
> issue, but we’ll try it out and report back.
>
> Jonathan
>
> ------------------------------
> *From:* Jim Brennan <ja...@verizonmedia.com>
> *Sent:* Tuesday, April 2, 2019 9:17 AM
> *To:* Jonathan Hung
> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>
> Thanks for working on this!
> One concern for us is support for a rolling upgrade. If we are running a
> cluster based on branch-2.8, will we be able to do a rolling upgrade (no
> cluster down-time) to a branch containing these changes? Have you tested
> rolling upgrades?
>
> Thanks.
> Jim
>
> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
> wrote:
>
>> Hello devs,
>>
>> Starting a discuss thread to merge resource types/native GPU scheduling
>> support to branch-3.0 and branch-2. The resource types work was done in
>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
>> is to merge GPU support into branch-3.0 and both resource types/GPU
>> support
>> to branch-2.
>>
>> Internally we've been running resource types/GPU support off a fork of
>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
>> well. Also for completeness we verified that everything going into
>> branch-2
>> also exists in branch-3.0.
>>
>> The specific list of patches to merge is in feature branch
>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>> diff
>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>
>> If there's no issues from the community I'll start a merge vote next week.
>> Thanks.
>>
>> Jonathan Hung
>>
>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an issue, but we’ll try it out and report back.
Jonathan
________________________________
From: Jim Brennan <ja...@verizonmedia.com>
Sent: Tuesday, April 2, 2019 9:17 AM
To: Jonathan Hung
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Thanks for working on this!
One concern for us is support for a rolling upgrade. If we are running a cluster based on branch-2.8, will we be able to do a rolling upgrade (no cluster down-time) to a branch containing these changes? Have you tested rolling upgrades?
Thanks.
Jim
On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>> wrote:
Hello devs,
Starting a discuss thread to merge resource types/native GPU scheduling
support to branch-3.0 and branch-2. The resource types work was done in
trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
is to merge GPU support into branch-3.0 and both resource types/GPU support
to branch-2.
Internally we've been running resource types/GPU support off a fork of
branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
well. Also for completeness we verified that everything going into branch-2
also exists in branch-3.0.
The specific list of patches to merge is in feature branch
YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
If there's no issues from the community I'll start a merge vote next week.
Thanks.
Jonathan Hung
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an issue, but we’ll try it out and report back.
Jonathan
________________________________
From: Jim Brennan <ja...@verizonmedia.com>
Sent: Tuesday, April 2, 2019 9:17 AM
To: Jonathan Hung
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Thanks for working on this!
One concern for us is support for a rolling upgrade. If we are running a cluster based on branch-2.8, will we be able to do a rolling upgrade (no cluster down-time) to a branch containing these changes? Have you tested rolling upgrades?
Thanks.
Jim
On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>> wrote:
Hello devs,
Starting a discuss thread to merge resource types/native GPU scheduling
support to branch-3.0 and branch-2. The resource types work was done in
trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
is to merge GPU support into branch-3.0 and both resource types/GPU support
to branch-2.
Internally we've been running resource types/GPU support off a fork of
branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
well. Also for completeness we verified that everything going into branch-2
also exists in branch-3.0.
The specific list of patches to merge is in feature branch
YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
If there's no issues from the community I'll start a merge vote next week.
Thanks.
Jonathan Hung
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Thanks for working on this!
One concern for us is support for a rolling upgrade. If we are running a
cluster based on branch-2.8, will we be able to do a rolling upgrade (no
cluster down-time) to a branch containing these changes? Have you tested
rolling upgrades?
Thanks.
Jim
On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com> wrote:
> Hello devs,
>
> Starting a discuss thread to merge resource types/native GPU scheduling
> support to branch-3.0 and branch-2. The resource types work was done in
> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
> is to merge GPU support into branch-3.0 and both resource types/GPU support
> to branch-2.
>
> Internally we've been running resource types/GPU support off a fork of
> branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
> well. Also for completeness we verified that everything going into branch-2
> also exists in branch-3.0.
>
> The specific list of patches to merge is in feature branch
> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>
> If there's no issues from the community I'll start a merge vote next week.
> Thanks.
>
> Jonathan Hung
>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Thanks for working on this!
One concern for us is support for a rolling upgrade. If we are running a
cluster based on branch-2.8, will we be able to do a rolling upgrade (no
cluster down-time) to a branch containing these changes? Have you tested
rolling upgrades?
Thanks.
Jim
On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com> wrote:
> Hello devs,
>
> Starting a discuss thread to merge resource types/native GPU scheduling
> support to branch-3.0 and branch-2. The resource types work was done in
> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
> is to merge GPU support into branch-3.0 and both resource types/GPU support
> to branch-2.
>
> Internally we've been running resource types/GPU support off a fork of
> branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
> well. Also for completeness we verified that everything going into branch-2
> also exists in branch-3.0.
>
> The specific list of patches to merge is in feature branch
> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>
> If there's no issues from the community I'll start a merge vote next week.
> Thanks.
>
> Jonathan Hung
>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Konstantin Shvachko <sh...@gmail.com>.
Sounds like a good idea to me.
This has been tested on branch-2 in production for some time and benchmarks
look good.
Thanks,
--Konstantin
On Fri, Mar 29, 2019 at 12:14 PM Jonathan Hung <jy...@gmail.com> wrote:
> Hello devs,
>
> Starting a discuss thread to merge resource types/native GPU scheduling
> support to branch-3.0 and branch-2. The resource types work was done in
> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
> is to merge GPU support into branch-3.0 and both resource types/GPU support
> to branch-2.
>
> Internally we've been running resource types/GPU support off a fork of
> branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
> well. Also for completeness we verified that everything going into branch-2
> also exists in branch-3.0.
>
> The specific list of patches to merge is in feature branch
> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>
> If there's no issues from the community I'll start a merge vote next week.
> Thanks.
>
> Jonathan Hung
>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Konstantin Shvachko <sh...@gmail.com>.
Sounds like a good idea to me.
This has been tested on branch-2 in production for some time and benchmarks
look good.
Thanks,
--Konstantin
On Fri, Mar 29, 2019 at 12:14 PM Jonathan Hung <jy...@gmail.com> wrote:
> Hello devs,
>
> Starting a discuss thread to merge resource types/native GPU scheduling
> support to branch-3.0 and branch-2. The resource types work was done in
> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the proposal
> is to merge GPU support into branch-3.0 and both resource types/GPU support
> to branch-2.
>
> Internally we've been running resource types/GPU support off a fork of
> branch-2.9.0 in a > 300 node GPU cluster for a few months which has worked
> well. Also for completeness we verified that everything going into branch-2
> also exists in branch-3.0.
>
> The specific list of patches to merge is in feature branch
> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0 diff
> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>
> If there's no issues from the community I'll start a merge vote next week.
> Thanks.
>
> Jonathan Hung
>