You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Jonathan Hung <jy...@gmail.com> on 2019/08/21 19:36:52 UTC
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Reviving this thread: we tested YARN RU starting with a cluster running
2.7.4, to running branch-2 + YARN-8200. Ran some simple MR/Spark jobs
concurrently with the RM/NM upgrades and did not see any issues.
If no other concerns I'll continue with a vote.
Jonathan Hung
On Thu, Apr 18, 2019 at 5:12 PM Jonathan Hung <jy...@gmail.com> wrote:
> Sorry for the delay, had to deprioritize this. Hoping to get to this next
> week.
>
> Jonathan
>
> ------------------------------
> *From:* Jim Brennan <ja...@verizonmedia.com>
> *Sent:* Thursday, April 18, 2019 7:28 AM
> *To:* Jonathan Hung
> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>
> Hi Jonathan,
>
> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>> issue, but we’ll try it out and report back.
>
>
> Any update on this?
> Jim
>
>
> On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com> wrote:
>
>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>> issue, but we’ll try it out and report back.
>>
>> Jonathan
>>
>> ------------------------------
>> *From:* Jim Brennan <ja...@verizonmedia.com>
>> *Sent:* Tuesday, April 2, 2019 9:17 AM
>> *To:* Jonathan Hung
>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>
>> Thanks for working on this!
>> One concern for us is support for a rolling upgrade. If we are running a
>> cluster based on branch-2.8, will we be able to do a rolling upgrade (no
>> cluster down-time) to a branch containing these changes? Have you tested
>> rolling upgrades?
>>
>> Thanks.
>> Jim
>>
>> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>>> Hello devs,
>>>
>>> Starting a discuss thread to merge resource types/native GPU scheduling
>>> support to branch-3.0 and branch-2. The resource types work was done in
>>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the
>>> proposal
>>> is to merge GPU support into branch-3.0 and both resource types/GPU
>>> support
>>> to branch-2.
>>>
>>> Internally we've been running resource types/GPU support off a fork of
>>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has
>>> worked
>>> well. Also for completeness we verified that everything going into
>>> branch-2
>>> also exists in branch-3.0.
>>>
>>> The specific list of patches to merge is in feature branch
>>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>>> diff
>>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>>
>>> If there's no issues from the community I'll start a merge vote next
>>> week.
>>> Thanks.
>>>
>>> Jonathan Hung
>>>
>>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Thanks for following up on the rolling upgrades!
We are definitely interested in seeing this move forward.
On Wed, Aug 21, 2019 at 2:37 PM Jonathan Hung <jy...@gmail.com> wrote:
> Reviving this thread: we tested YARN RU starting with a cluster running
> 2.7.4, to running branch-2 + YARN-8200. Ran some simple MR/Spark jobs
> concurrently with the RM/NM upgrades and did not see any issues.
>
> If no other concerns I'll continue with a vote.
>
> Jonathan Hung
>
>
> On Thu, Apr 18, 2019 at 5:12 PM Jonathan Hung <jy...@gmail.com>
> wrote:
>
>> Sorry for the delay, had to deprioritize this. Hoping to get to this next
>> week.
>>
>> Jonathan
>>
>> ------------------------------
>> *From:* Jim Brennan <ja...@verizonmedia.com>
>> *Sent:* Thursday, April 18, 2019 7:28 AM
>> *To:* Jonathan Hung
>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>
>> Hi Jonathan,
>>
>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>>> issue, but we’ll try it out and report back.
>>
>>
>> Any update on this?
>> Jim
>>
>>
>> On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being
>>> an issue, but we’ll try it out and report back.
>>>
>>> Jonathan
>>>
>>> ------------------------------
>>> *From:* Jim Brennan <ja...@verizonmedia.com>
>>> *Sent:* Tuesday, April 2, 2019 9:17 AM
>>> *To:* Jonathan Hung
>>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>>
>>> Thanks for working on this!
>>> One concern for us is support for a rolling upgrade. If we are running
>>> a cluster based on branch-2.8, will we be able to do a rolling upgrade (no
>>> cluster down-time) to a branch containing these changes? Have you tested
>>> rolling upgrades?
>>>
>>> Thanks.
>>> Jim
>>>
>>> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
>>> wrote:
>>>
>>>> Hello devs,
>>>>
>>>> Starting a discuss thread to merge resource types/native GPU scheduling
>>>> support to branch-3.0 and branch-2. The resource types work was done in
>>>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the
>>>> proposal
>>>> is to merge GPU support into branch-3.0 and both resource types/GPU
>>>> support
>>>> to branch-2.
>>>>
>>>> Internally we've been running resource types/GPU support off a fork of
>>>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has
>>>> worked
>>>> well. Also for completeness we verified that everything going into
>>>> branch-2
>>>> also exists in branch-3.0.
>>>>
>>>> The specific list of patches to merge is in feature branch
>>>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>>>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>>>> diff
>>>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>>>
>>>> If there's no issues from the community I'll start a merge vote next
>>>> week.
>>>> Thanks.
>>>>
>>>> Jonathan Hung
>>>>
>>>
Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
Posted by Jim Brennan <ja...@verizonmedia.com.INVALID>.
Thanks for following up on the rolling upgrades!
We are definitely interested in seeing this move forward.
On Wed, Aug 21, 2019 at 2:37 PM Jonathan Hung <jy...@gmail.com> wrote:
> Reviving this thread: we tested YARN RU starting with a cluster running
> 2.7.4, to running branch-2 + YARN-8200. Ran some simple MR/Spark jobs
> concurrently with the RM/NM upgrades and did not see any issues.
>
> If no other concerns I'll continue with a vote.
>
> Jonathan Hung
>
>
> On Thu, Apr 18, 2019 at 5:12 PM Jonathan Hung <jy...@gmail.com>
> wrote:
>
>> Sorry for the delay, had to deprioritize this. Hoping to get to this next
>> week.
>>
>> Jonathan
>>
>> ------------------------------
>> *From:* Jim Brennan <ja...@verizonmedia.com>
>> *Sent:* Thursday, April 18, 2019 7:28 AM
>> *To:* Jonathan Hung
>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>
>> Hi Jonathan,
>>
>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being an
>>> issue, but we’ll try it out and report back.
>>
>>
>> Any update on this?
>> Jim
>>
>>
>> On Wed, Apr 3, 2019 at 2:16 AM Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>>> Hi Jim, we have not tested rolling upgrade. I don’t foresee this being
>>> an issue, but we’ll try it out and report back.
>>>
>>> Jonathan
>>>
>>> ------------------------------
>>> *From:* Jim Brennan <ja...@verizonmedia.com>
>>> *Sent:* Tuesday, April 2, 2019 9:17 AM
>>> *To:* Jonathan Hung
>>> *Cc:* yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> *Subject:* Re: [DISCUSS] Merging YARN-8200 to branch-3.0 and branch-2
>>>
>>> Thanks for working on this!
>>> One concern for us is support for a rolling upgrade. If we are running
>>> a cluster based on branch-2.8, will we be able to do a rolling upgrade (no
>>> cluster down-time) to a branch containing these changes? Have you tested
>>> rolling upgrades?
>>>
>>> Thanks.
>>> Jim
>>>
>>> On Fri, Mar 29, 2019 at 2:14 PM Jonathan Hung <jy...@gmail.com>
>>> wrote:
>>>
>>>> Hello devs,
>>>>
>>>> Starting a discuss thread to merge resource types/native GPU scheduling
>>>> support to branch-3.0 and branch-2. The resource types work was done in
>>>> trunk~branch-3.0 and GPU support done in trunk~branch-3.1, so the
>>>> proposal
>>>> is to merge GPU support into branch-3.0 and both resource types/GPU
>>>> support
>>>> to branch-2.
>>>>
>>>> Internally we've been running resource types/GPU support off a fork of
>>>> branch-2.9.0 in a > 300 node GPU cluster for a few months which has
>>>> worked
>>>> well. Also for completeness we verified that everything going into
>>>> branch-2
>>>> also exists in branch-3.0.
>>>>
>>>> The specific list of patches to merge is in feature branch
>>>> YARN-8200.branch3 (for branch-3.0) and feature branch YARN-8200 (for
>>>> branch-2). Full patches containing the YARN-8200.branch3 -> branch-3.0
>>>> diff
>>>> and YARN-8200 -> branch-2 diff have been posted to YARN-8200 jira.
>>>>
>>>> If there's no issues from the community I'll start a merge vote next
>>>> week.
>>>> Thanks.
>>>>
>>>> Jonathan Hung
>>>>
>>>