You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by Jian He <jh...@hortonworks.com> on 2017/08/15 20:56:12 UTC

[DISCUSS] Merge yarn-native-services branch into trunk

Hi All,
I would like to bring up the discussion of merging yarn-native-services branch into trunk in a few weeks. There are a few issues left under YARN-5079 that are being actively worked upon. As soon as they are resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to get this in for hadoop3 beta.

The major work in this branch include below umbrella jiras:
 - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and orchestrate existing services to YARN either docker or non-docker based.
 - YARN-4793. A Rest API server for user to deploy a service via a simple JSON spec
 - YARN-4757. Extending today's service registry with a simple DNS service to enable users to discover services deployed on YARN
 - YARN-6419. UI support for native-services on the new YARN UI
All these new services are optional and have to be explicitly enabled. 

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, Akhil PB. This effort could not be possible without their ideas and hard work.

Please share your thoughts. Thanks.

Jian


---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

Hi Arun

Sorry for late reply.
* Is there a branch-2 merge planned for this ?
Branch-2 is not planned for this merge.

* I understand YARN-7126 has some introductory documentation, But I think we need to flesh it up a bit more before release, I would also like to see steps to deploy a sample service.
We have added more documentations, QuickStart.md, Overview.md and others in the same folder.
YarnCommands.md is also updated to document the new shell commands.

I encourage everyone to try and share suggestions.

As said in another email thread, we decided to drop this for beta and re-target it for GA.

Thanks,
Jian

On Sep 5, 2017, at 1:37 PM, Arun Suresh <as...@apache.org>> wrote:

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

But I did have a couple of questions:
* Is there a branch-2 merge planned for this ?
* I understand YARN-7126 has some introductory documentation, But I think we need to flesh it up a bit more before release, I would also like to see steps to deploy a sample service.

Cheers
-Arun

On Thu, Aug 31, 2017 at 12:40 AM, Jian He <jh...@hortonworks.com>> wrote:
Update:
I’ve chatted with Andrew offline, we’ll proceed with merging yarn-native-services into trunk for beta.
We’ll advertise this feature as “alpha"
Currently, we have completed all the jiras for this merge - I’ve also moved out the subtasks that are not blocking this merge.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang <an...@cloudera.com>>> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com>>> wrote:
Thanks Andrew for the comments. Answers below:

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are unstable, I'd propose calling it "alpha" in the release notes, like we do the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

Hi Arun

Sorry for late reply.
* Is there a branch-2 merge planned for this ?
Branch-2 is not planned for this merge.

I encourage everyone to try and share suggestions.

As said in another email thread, we decided to drop this for beta and re-target it for GA.

Thanks,
Jian

On Sep 5, 2017, at 1:37 PM, Arun Suresh <as...@apache.org>> wrote:

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

Cheers
-Arun

I’ve created YARN-7127 to run the entire patch against trunk, once that goes green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang <an...@cloudera.com>>> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com>>> wrote:
Thanks Andrew for the comments. Answers below:

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are unstable, I'd propose calling it "alpha" in the release notes, like we do the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

Hi Arun

Sorry for late reply.
* Is there a branch-2 merge planned for this ?
Branch-2 is not planned for this merge.

I encourage everyone to try and share suggestions.

As said in another email thread, we decided to drop this for beta and re-target it for GA.

Thanks,
Jian

On Sep 5, 2017, at 1:37 PM, Arun Suresh <as...@apache.org>> wrote:

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

Cheers
-Arun

I’ve created YARN-7127 to run the entire patch against trunk, once that goes green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang <an...@cloudera.com>>> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com>>> wrote:
Thanks Andrew for the comments. Answers below:

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are unstable, I'd propose calling it "alpha" in the release notes, like we do the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Arun Suresh <as...@apache.org>.

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

But I did have a couple of questions:
* Is there a branch-2 merge planned for this ?
* I understand YARN-7126 has some introductory documentation, But I think
we need to flesh it up a bit more before release, I would also like to see
steps to deploy a sample service.

Cheers
-Arun

On Thu, Aug 31, 2017 at 12:40 AM, Jian He <jh...@hortonworks.com> wrote:

> Update:
> I’ve chatted with Andrew offline, we’ll proceed with merging
> yarn-native-services into trunk for beta.
> We’ll advertise this feature as “alpha"
> Currently, we have completed all the jiras for this merge - I’ve also
> moved out the subtasks that are not blocking this merge.
>
> I’ve created YARN-7127 to run the entire patch against trunk, once that
> goes green, I plan to start a formal vote.
>
> Thanks,
> Jian
>
> On Aug 18, 2017, at 2:48 PM, Andrew Wang <andrew.wang@cloudera.com<mailto:
> andrew.wang@cloudera.com>> wrote:
>
> Hi Jian, thanks for the reply,
>
> On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jhe@hortonworks.com<mailto:jh
> e@hortonworks.com>> wrote:
> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>
> Great! That adds a lot more confidence that this is safe to merge.
>
> Are these new APIs listed in user documentation, and described as unstable?
>
> - We have been running/testing a version of the entire system internally
> for quite a while.
>
> Do you mind elaborating on the level of testing? Number of nodes, types of
> applications, production or test workload, etc. It'd help us build
> confidence.
>
> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>
> Few more questions:
>
> How should we advertise this feature in the release? Since the APIs are
> unstable, I'd propose calling it "alpha" in the release notes, like we do
> the TSv2.
>
> Could you move out subtasks from YARN-5079 that are not blocking the
> merge? This would make it easier to understand what's remaining.
>
> Thanks,
> Andrew
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Arun Suresh <as...@apache.org>.

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

But I did have a couple of questions:
* Is there a branch-2 merge planned for this ?
* I understand YARN-7126 has some introductory documentation, But I think
we need to flesh it up a bit more before release, I would also like to see
steps to deploy a sample service.

Cheers
-Arun

On Thu, Aug 31, 2017 at 12:40 AM, Jian He <jh...@hortonworks.com> wrote:

> Update:
> I’ve chatted with Andrew offline, we’ll proceed with merging
> yarn-native-services into trunk for beta.
> We’ll advertise this feature as “alpha"
> Currently, we have completed all the jiras for this merge - I’ve also
> moved out the subtasks that are not blocking this merge.
>
> I’ve created YARN-7127 to run the entire patch against trunk, once that
> goes green, I plan to start a formal vote.
>
> Thanks,
> Jian
>
> On Aug 18, 2017, at 2:48 PM, Andrew Wang <andrew.wang@cloudera.com<mailto:
> andrew.wang@cloudera.com>> wrote:
>
> Hi Jian, thanks for the reply,
>
> On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jhe@hortonworks.com<mailto:jh
> e@hortonworks.com>> wrote:
> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>
> Great! That adds a lot more confidence that this is safe to merge.
>
> Are these new APIs listed in user documentation, and described as unstable?
>
> - We have been running/testing a version of the entire system internally
> for quite a while.
>
> Do you mind elaborating on the level of testing? Number of nodes, types of
> applications, production or test workload, etc. It'd help us build
> confidence.
>
> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>
> Few more questions:
>
> How should we advertise this feature in the release? Since the APIs are
> unstable, I'd propose calling it "alpha" in the release notes, like we do
> the TSv2.
>
> Could you move out subtasks from YARN-5079 that are not blocking the
> merge? This would make it easier to understand what's remaining.
>
> Thanks,
> Andrew
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Arun Suresh <as...@apache.org>.

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

But I did have a couple of questions:
* Is there a branch-2 merge planned for this ?
* I understand YARN-7126 has some introductory documentation, But I think
we need to flesh it up a bit more before release, I would also like to see
steps to deploy a sample service.

Cheers
-Arun

On Thu, Aug 31, 2017 at 12:40 AM, Jian He <jh...@hortonworks.com> wrote:

> Update:
> I’ve chatted with Andrew offline, we’ll proceed with merging
> yarn-native-services into trunk for beta.
> We’ll advertise this feature as “alpha"
> Currently, we have completed all the jiras for this merge - I’ve also
> moved out the subtasks that are not blocking this merge.
>
> I’ve created YARN-7127 to run the entire patch against trunk, once that
> goes green, I plan to start a formal vote.
>
> Thanks,
> Jian
>
> On Aug 18, 2017, at 2:48 PM, Andrew Wang <andrew.wang@cloudera.com<mailto:
> andrew.wang@cloudera.com>> wrote:
>
> Hi Jian, thanks for the reply,
>
> On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jhe@hortonworks.com<mailto:jh
> e@hortonworks.com>> wrote:
> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>
> Great! That adds a lot more confidence that this is safe to merge.
>
> Are these new APIs listed in user documentation, and described as unstable?
>
> - We have been running/testing a version of the entire system internally
> for quite a while.
>
> Do you mind elaborating on the level of testing? Number of nodes, types of
> applications, production or test workload, etc. It'd help us build
> confidence.
>
> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>
> Few more questions:
>
> How should we advertise this feature in the release? Since the APIs are
> unstable, I'd propose calling it "alpha" in the release notes, like we do
> the TSv2.
>
> Could you move out subtasks from YARN-5079 that are not blocking the
> merge? This would make it easier to understand what's remaining.
>
> Thanks,
> Andrew
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Arun Suresh <as...@apache.org>.

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

But I did have a couple of questions:
* Is there a branch-2 merge planned for this ?
* I understand YARN-7126 has some introductory documentation, But I think
we need to flesh it up a bit more before release, I would also like to see
steps to deploy a sample service.

Cheers
-Arun

On Thu, Aug 31, 2017 at 12:40 AM, Jian He <jh...@hortonworks.com> wrote:

> Update:
> I’ve chatted with Andrew offline, we’ll proceed with merging
> yarn-native-services into trunk for beta.
> We’ll advertise this feature as “alpha"
> Currently, we have completed all the jiras for this merge - I’ve also
> moved out the subtasks that are not blocking this merge.
>
> I’ve created YARN-7127 to run the entire patch against trunk, once that
> goes green, I plan to start a formal vote.
>
> Thanks,
> Jian
>
> On Aug 18, 2017, at 2:48 PM, Andrew Wang <andrew.wang@cloudera.com<mailto:
> andrew.wang@cloudera.com>> wrote:
>
> Hi Jian, thanks for the reply,
>
> On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jhe@hortonworks.com<mailto:jh
> e@hortonworks.com>> wrote:
> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>
> Great! That adds a lot more confidence that this is safe to merge.
>
> Are these new APIs listed in user documentation, and described as unstable?
>
> - We have been running/testing a version of the entire system internally
> for quite a while.
>
> Do you mind elaborating on the level of testing? Number of nodes, types of
> applications, production or test workload, etc. It'd help us build
> confidence.
>
> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>
> Few more questions:
>
> How should we advertise this feature in the release? Since the APIs are
> unstable, I'd propose calling it "alpha" in the release notes, like we do
> the TSv2.
>
> Could you move out subtasks from YARN-5079 that are not blocking the
> merge? This would make it easier to understand what's remaining.
>
> Thanks,
> Andrew
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

Update:
I’ve chatted with Andrew offline, we’ll proceed with merging yarn-native-services into trunk for beta.
We’ll advertise this feature as “alpha"
Currently, we have completed all the jiras for this merge - I’ve also moved out the subtasks that are not blocking this merge.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang <an...@cloudera.com>> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com>> wrote:
Thanks Andrew for the comments. Answers below:

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are unstable, I'd propose calling it "alpha" in the release notes, like we do the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang <an...@cloudera.com>> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com>> wrote:
Thanks Andrew for the comments. Answers below:

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are unstable, I'd propose calling it "alpha" in the release notes, like we do the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang <an...@cloudera.com>> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com>> wrote:
Thanks Andrew for the comments. Answers below:

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are unstable, I'd propose calling it "alpha" in the release notes, like we do the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com> wrote:

> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

> - We have been running/testing a version of the entire system internally
> for quite a while.
>

Do you mind elaborating on the level of testing? Number of nodes, types of
applications, production or test workload, etc. It'd help us build
confidence.

> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>

Few more questions:

How should we advertise this feature in the release? Since the APIs are
unstable, I'd propose calling it "alpha" in the release notes, like we do
the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge?
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com> wrote:

> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

> - We have been running/testing a version of the entire system internally
> for quite a while.
>

Do you mind elaborating on the level of testing? Number of nodes, types of
applications, production or test workload, etc. It'd help us build
confidence.

> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>

Few more questions:

How should we advertise this feature in the release? Since the APIs are
unstable, I'd propose calling it "alpha" in the release notes, like we do
the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge?
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com> wrote:

> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

> - We have been running/testing a version of the entire system internally
> for quite a while.
>

Do you mind elaborating on the level of testing? Number of nodes, types of
applications, production or test workload, etc. It'd help us build
confidence.

> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>

Few more questions:

How should we advertise this feature in the release? Since the APIs are
unstable, I'd propose calling it "alpha" in the release notes, like we do
the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge?
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He <jh...@hortonworks.com> wrote:

> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

> - We have been running/testing a version of the entire system internally
> for quite a while.
>

Do you mind elaborating on the level of testing? Number of nodes, types of
applications, production or test workload, etc. It'd help us build
confidence.

> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>

Few more questions:

How should we advertise this feature in the release? Since the APIs are
unstable, I'd propose calling it "alpha" in the release notes, like we do
the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge?
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

Thanks Andrew for the comments. Answers below:

- There are no new APIs added in YARN/Hadoop core. In fact, all the new code are running outside of existing system and they are optional and require users to explicitly opt in. The new system’s own rest API is not stable and will be evolving.
- We have been running/testing a version of the entire system internally for quite a while.
- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility of moving fast and not block the potential timeline.

Jian

On Aug 16, 2017, at 10:29 AM, Andrew Wang <an...@cloudera.com>> wrote:

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in hopefully the next two weeks, that's very, very close to the goal release date. We've already got a pile of blockers and criticals to resolve before then.

Could you comment on testing and API stability for this branch? YARN Federation was run at high scale and did not add new APIs, which provided a lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0 efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He <jh...@hortonworks.com>> wrote:
Hi All,
I would like to bring up the discussion of merging yarn-native-services branch into trunk in a few weeks. There are a few issues left under YARN-5079 that are being actively worked upon. As soon as they are resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to get this in for hadoop3 beta.

The major work in this branch include below umbrella jiras:
- YARN-5079. A native YARN framework (ApplicationMaster) to migrate and orchestrate existing services to YARN either docker or non-docker based.
- YARN-4793. A Rest API server for user to deploy a service via a simple JSON spec
- YARN-4757. Extending today's service registry with a simple DNS service to enable users to discover services deployed on YARN
- YARN-6419. UI support for native-services on the new YARN UI
All these new services are optional and have to be explicitly enabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, Akhil PB. This effort could not be possible without their ideas and hard work.

Please share your thoughts. Thanks.

Jian

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org<ma...@hadoop.apache.org>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

Thanks Andrew for the comments. Answers below:

Jian

On Aug 16, 2017, at 10:29 AM, Andrew Wang <an...@cloudera.com>> wrote:

Hi Jian,

Could you comment on testing and API stability for this branch? YARN Federation was run at high scale and did not add new APIs, which provided a lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0 efforts, and targeting this for 3.1.0.

Best,
Andrew

Please share your thoughts. Thanks.

Jian

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Jian He <jh...@hortonworks.com>.

Thanks Andrew for the comments. Answers below:

Jian

On Aug 16, 2017, at 10:29 AM, Andrew Wang <an...@cloudera.com>> wrote:

Hi Jian,

Could you comment on testing and API stability for this branch? YARN Federation was run at high scale and did not add new APIs, which provided a lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0 efforts, and targeting this for 3.1.0.

Best,
Andrew

Please share your thoughts. Thanks.

Jian

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in
hopefully the next two weeks, that's very, very close to the goal release
date. We've already got a pile of blockers and criticals to resolve before
then.

Could you comment on testing and API stability for this branch? YARN
Federation was run at high scale and did not add new APIs, which provided a
lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0
efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He <jh...@hortonworks.com> wrote:

> Hi All,
> I would like to bring up the discussion of merging yarn-native-services
> branch into trunk in a few weeks. There are a few issues left under
> YARN-5079 that are being actively worked upon. As soon as they are
> resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to
> get this in for hadoop3 beta.
>
> The major work in this branch include below umbrella jiras:
>  - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
>  - YARN-4793. A Rest API server for user to deploy a service via a simple
> JSON spec
>  - YARN-4757. Extending today's service registry with a simple DNS service
> to enable users to discover services deployed on YARN
>  - YARN-6419. UI support for native-services on the new YARN UI
> All these new services are optional and have to be explicitly enabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Please share your thoughts. Thanks.
>
> Jian
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in
hopefully the next two weeks, that's very, very close to the goal release
date. We've already got a pile of blockers and criticals to resolve before
then.

Could you comment on testing and API stability for this branch? YARN
Federation was run at high scale and did not add new APIs, which provided a
lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0
efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He <jh...@hortonworks.com> wrote:

> Hi All,
> I would like to bring up the discussion of merging yarn-native-services
> branch into trunk in a few weeks. There are a few issues left under
> YARN-5079 that are being actively worked upon. As soon as they are
> resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to
> get this in for hadoop3 beta.
>
> The major work in this branch include below umbrella jiras:
>  - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
>  - YARN-4793. A Rest API server for user to deploy a service via a simple
> JSON spec
>  - YARN-4757. Extending today's service registry with a simple DNS service
> to enable users to discover services deployed on YARN
>  - YARN-6419. UI support for native-services on the new YARN UI
> All these new services are optional and have to be explicitly enabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Please share your thoughts. Thanks.
>
> Jian
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in
hopefully the next two weeks, that's very, very close to the goal release
date. We've already got a pile of blockers and criticals to resolve before
then.

Could you comment on testing and API stability for this branch? YARN
Federation was run at high scale and did not add new APIs, which provided a
lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0
efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He <jh...@hortonworks.com> wrote:

> Hi All,
> I would like to bring up the discussion of merging yarn-native-services
> branch into trunk in a few weeks. There are a few issues left under
> YARN-5079 that are being actively worked upon. As soon as they are
> resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to
> get this in for hadoop3 beta.
>
> The major work in this branch include below umbrella jiras:
>  - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
>  - YARN-4793. A Rest API server for user to deploy a service via a simple
> JSON spec
>  - YARN-4757. Extending today's service registry with a simple DNS service
> to enable users to discover services deployed on YARN
>  - YARN-6419. UI support for native-services on the new YARN UI
> All these new services are optional and have to be explicitly enabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Please share your thoughts. Thanks.
>
> Jian
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Posted by Andrew Wang <an...@cloudera.com>.

Hi Jian,

Hadoop 3.0.0-beta1 is planned for mid-September. If the plan is to merge in
hopefully the next two weeks, that's very, very close to the goal release
date. We've already got a pile of blockers and criticals to resolve before
then.

Could you comment on testing and API stability for this branch? YARN
Federation was run at high scale and did not add new APIs, which provided a
lot of confidence in the merge.

I'll also raise the option of cutting branch-3 or branch-3.0 for the 3.0.0
efforts, and targeting this for 3.1.0.

Best,
Andrew

On Tue, Aug 15, 2017 at 1:56 PM, Jian He <jh...@hortonworks.com> wrote:

> Hi All,
> I would like to bring up the discussion of merging yarn-native-services
> branch into trunk in a few weeks. There are a few issues left under
> YARN-5079 that are being actively worked upon. As soon as they are
> resolved, we plan on start a vote hopefully in next 2 weeks. The goal is to
> get this in for hadoop3 beta.
>
> The major work in this branch include below umbrella jiras:
>  - YARN-5079. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
>  - YARN-4793. A Rest API server for user to deploy a service via a simple
> JSON spec
>  - YARN-4757. Extending today's service registry with a simple DNS service
> to enable users to discover services deployed on YARN
>  - YARN-6419. UI support for native-services on the new YARN UI
> All these new services are optional and have to be explicitly enabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Please share your thoughts. Thanks.
>
> Jian
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>