You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Jian He <jh...@hortonworks.com> on 2017/09/01 03:33:25 UTC
[VOTE] Merge yarn-native-services branch into trunk
Hi All,
I would like to call a vote for merging yarn-native-services to trunk. The vote will run for 7 days as usual.
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate and orchestrate existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API server for user to deploy a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to enable users to discover services deployed on YARN
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing system, and have no impact on existing system if disabled.
Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, Akhil PB. This effort could not be possible without their ideas and hard work.
Thanks,
Jian
[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
Thanks Allen. You are right, the github renderer does have trouble
rendering the headers. I was only looking at the html generated by mvn
site, which did not have trouble rendering them. Anyway I added a space
after all the hashes and it looks ok through github now.
-Gour
On 9/5/17, 3:20 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>>
>> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
>> This includes things like Œyarnsite.xml¹ (missing a dash.)
>>
>> The md patch uploaded to YARN-5244 had some special chars. I fixed those
>> in YARN-7161.
>
>
> It’s a lot more than just special chars I think. Even github (which has
>a way better markdown processor than what we’re using for the site docs)
>is having trouble rendering it:
>
>https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c64
>06ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/na
>tive-services/NativeServicesDiscovery.md
>
>e.g., all of those ‘###’ are likely missing a space.
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
Thanks Allen. You are right, the github renderer does have trouble
rendering the headers. I was only looking at the html generated by mvn
site, which did not have trouble rendering them. Anyway I added a space
after all the hashes and it looks ok through github now.
-Gour
On 9/5/17, 3:20 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>>
>> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
>> This includes things like Œyarnsite.xml¹ (missing a dash.)
>>
>> The md patch uploaded to YARN-5244 had some special chars. I fixed those
>> in YARN-7161.
>
>
> It’s a lot more than just special chars I think. Even github (which has
>a way better markdown processor than what we’re using for the site docs)
>is having trouble rendering it:
>
>https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c64
>06ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/na
>tive-services/NativeServicesDiscovery.md
>
>e.g., all of those ‘###’ are likely missing a space.
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
Thanks Allen. You are right, the github renderer does have trouble
rendering the headers. I was only looking at the html generated by mvn
site, which did not have trouble rendering them. Anyway I added a space
after all the hashes and it looks ok through github now.
-Gour
On 9/5/17, 3:20 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>>
>> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
>> This includes things like Œyarnsite.xml¹ (missing a dash.)
>>
>> The md patch uploaded to YARN-5244 had some special chars. I fixed those
>> in YARN-7161.
>
>
> It’s a lot more than just special chars I think. Even github (which has
>a way better markdown processor than what we’re using for the site docs)
>is having trouble rendering it:
>
>https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c64
>06ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/na
>tive-services/NativeServicesDiscovery.md
>
>e.g., all of those ‘###’ are likely missing a space.
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
Thanks Allen. You are right, the github renderer does have trouble
rendering the headers. I was only looking at the html generated by mvn
site, which did not have trouble rendering them. Anyway I added a space
after all the hashes and it looks ok through github now.
-Gour
On 9/5/17, 3:20 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>>
>> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
>> This includes things like Œyarnsite.xml¹ (missing a dash.)
>>
>> The md patch uploaded to YARN-5244 had some special chars. I fixed those
>> in YARN-7161.
>
>
> It’s a lot more than just special chars I think. Even github (which has
>a way better markdown processor than what we’re using for the site docs)
>is having trouble rendering it:
>
>https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c64
>06ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/na
>tive-services/NativeServicesDiscovery.md
>
>e.g., all of those ‘###’ are likely missing a space.
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
> This includes things like Œyarnsite.xml¹ (missing a dash.)
>
> The md patch uploaded to YARN-5244 had some special chars. I fixed those
> in YARN-7161.
It’s a lot more than just special chars I think. Even github (which has a way better markdown processor than what we’re using for the site docs) is having trouble rendering it:
https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c6406ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md
e.g., all of those ‘###’ are likely missing a space.
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
> This includes things like Œyarnsite.xml¹ (missing a dash.)
>
> The md patch uploaded to YARN-5244 had some special chars. I fixed those
> in YARN-7161.
It’s a lot more than just special chars I think. Even github (which has a way better markdown processor than what we’re using for the site docs) is having trouble rendering it:
https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c6406ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md
e.g., all of those ‘###’ are likely missing a space.
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
> This includes things like Œyarnsite.xml¹ (missing a dash.)
>
> The md patch uploaded to YARN-5244 had some special chars. I fixed those
> in YARN-7161.
It’s a lot more than just special chars I think. Even github (which has a way better markdown processor than what we’re using for the site docs) is having trouble rendering it:
https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c6406ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md
e.g., all of those ‘###’ are likely missing a space.
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 3:12 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
> This includes things like Œyarnsite.xml¹ (missing a dash.)
>
> The md patch uploaded to YARN-5244 had some special chars. I fixed those
> in YARN-7161.
It’s a lot more than just special chars I think. Even github (which has a way better markdown processor than what we’re using for the site docs) is having trouble rendering it:
https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c6406ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md
e.g., all of those ‘###’ are likely missing a space.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
2) Lots of markdown problems in the NativeServicesDiscovery.md document.
This includes things like Œyarnsite.xml¹ (missing a dash.)
The md patch uploaded to YARN-5244 had some special chars. I fixed those
in YARN-7161.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
2) Lots of markdown problems in the NativeServicesDiscovery.md document.
This includes things like Œyarnsite.xml¹ (missing a dash.)
The md patch uploaded to YARN-5244 had some special chars. I fixed those
in YARN-7161.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
2) Lots of markdown problems in the NativeServicesDiscovery.md document.
This includes things like Œyarnsite.xml¹ (missing a dash.)
The md patch uploaded to YARN-5244 had some special chars. I fixed those
in YARN-7161.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Thanks for your consideration Jian, let's track this for GA then.
Best,
Andrew
On Fri, Sep 8, 2017 at 3:02 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi Andrew,
>
> At this point, there are no more release blockers including documentations
> from our side - all work done.
> But I agree it is too close to the release, after talking with other team
> members, we are fine to drop this from beta,
>
> And we want to target this for GA.
> I’m withdrawing this vote and will start afresh vote later for GA.
> Thanks all who voted this effort !
>
> Thanks,
> Jian
>
>
> > On Sep 7, 2017, at 3:59 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> > Hi folks,
> >
> > This vote closes today. I see a -1 from Allen on inclusion in beta1. I
> see
> > there's active fixing going on, but given that we're one week out from
> RC0,
> > I think we should drop this from beta1.
> >
> > Allen, Jian, others, is this reasonable? What release should we retarget
> > this for? I don't have a sense for how much work there is left to do, but
> > as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> >
> > Best,
> > Andrew
> >
> > On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> >
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >> Sorry for confusion, I was actively writing the formal documentation for
> >> how to use/how it works etc. and will post soon in a few hours.
> >>
> >>
> >>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <
> aw@effectivemachines.com>
> >> wrote:
> >>>
> >>>
> >>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>>>
> >>>>> If it doesn’t have all the bells and whistles, then it shouldn’t
> >> be on port 53 by default.
> >>>> Sure, I’ll change the default port to not use 53 and document it.
> >>>>> *how* is it getting launched on a privileged port? It sounds like
> >> the expectation is to run “command” as root. *ALL* of the previous
> >> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t
> this
> >> one? These questions matter from a security standpoint.
> >>>> Yes, it is running as “root” to be able to use the privileged port.
> The
> >> DNS server is not yet integrated with the hadoop script.
> >>>>
> >>>>> Check the output. It’s pretty obviously borked:
> >>>> Thanks for pointing out. Missed this when rebasing onto trunk.
> >>>
> >>>
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >>>
> >>> * A bunch of mostly new Java code that may or may not have
> >> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> >>> * ~1/3 of the docs are roadmap/TBD
> >>> * ~1/3 of the docs are for an optional DNS daemon that has
> >> no end user hook to start it
> >>> * ~1/3 of the docs are for a REST API that comes from some
> >> undefined daemon (apiserver?)
> >>> * Two new, but undocumented, subcommands to yarn
> >>> * There are no docs for admins or users on how to actually
> >> start or use this completely new/separate/optional feature
> >>>
> >>> How are outside people (e.g., non-branch committers) supposed to
> >> test this new feature under these conditions?
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Thanks for your consideration Jian, let's track this for GA then.
Best,
Andrew
On Fri, Sep 8, 2017 at 3:02 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi Andrew,
>
> At this point, there are no more release blockers including documentations
> from our side - all work done.
> But I agree it is too close to the release, after talking with other team
> members, we are fine to drop this from beta,
>
> And we want to target this for GA.
> I’m withdrawing this vote and will start afresh vote later for GA.
> Thanks all who voted this effort !
>
> Thanks,
> Jian
>
>
> > On Sep 7, 2017, at 3:59 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> > Hi folks,
> >
> > This vote closes today. I see a -1 from Allen on inclusion in beta1. I
> see
> > there's active fixing going on, but given that we're one week out from
> RC0,
> > I think we should drop this from beta1.
> >
> > Allen, Jian, others, is this reasonable? What release should we retarget
> > this for? I don't have a sense for how much work there is left to do, but
> > as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> >
> > Best,
> > Andrew
> >
> > On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> >
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >> Sorry for confusion, I was actively writing the formal documentation for
> >> how to use/how it works etc. and will post soon in a few hours.
> >>
> >>
> >>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <
> aw@effectivemachines.com>
> >> wrote:
> >>>
> >>>
> >>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>>>
> >>>>> If it doesn’t have all the bells and whistles, then it shouldn’t
> >> be on port 53 by default.
> >>>> Sure, I’ll change the default port to not use 53 and document it.
> >>>>> *how* is it getting launched on a privileged port? It sounds like
> >> the expectation is to run “command” as root. *ALL* of the previous
> >> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t
> this
> >> one? These questions matter from a security standpoint.
> >>>> Yes, it is running as “root” to be able to use the privileged port.
> The
> >> DNS server is not yet integrated with the hadoop script.
> >>>>
> >>>>> Check the output. It’s pretty obviously borked:
> >>>> Thanks for pointing out. Missed this when rebasing onto trunk.
> >>>
> >>>
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >>>
> >>> * A bunch of mostly new Java code that may or may not have
> >> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> >>> * ~1/3 of the docs are roadmap/TBD
> >>> * ~1/3 of the docs are for an optional DNS daemon that has
> >> no end user hook to start it
> >>> * ~1/3 of the docs are for a REST API that comes from some
> >> undefined daemon (apiserver?)
> >>> * Two new, but undocumented, subcommands to yarn
> >>> * There are no docs for admins or users on how to actually
> >> start or use this completely new/separate/optional feature
> >>>
> >>> How are outside people (e.g., non-branch committers) supposed to
> >> test this new feature under these conditions?
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Thanks for your consideration Jian, let's track this for GA then.
Best,
Andrew
On Fri, Sep 8, 2017 at 3:02 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi Andrew,
>
> At this point, there are no more release blockers including documentations
> from our side - all work done.
> But I agree it is too close to the release, after talking with other team
> members, we are fine to drop this from beta,
>
> And we want to target this for GA.
> I’m withdrawing this vote and will start afresh vote later for GA.
> Thanks all who voted this effort !
>
> Thanks,
> Jian
>
>
> > On Sep 7, 2017, at 3:59 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> > Hi folks,
> >
> > This vote closes today. I see a -1 from Allen on inclusion in beta1. I
> see
> > there's active fixing going on, but given that we're one week out from
> RC0,
> > I think we should drop this from beta1.
> >
> > Allen, Jian, others, is this reasonable? What release should we retarget
> > this for? I don't have a sense for how much work there is left to do, but
> > as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> >
> > Best,
> > Andrew
> >
> > On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> >
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >> Sorry for confusion, I was actively writing the formal documentation for
> >> how to use/how it works etc. and will post soon in a few hours.
> >>
> >>
> >>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <
> aw@effectivemachines.com>
> >> wrote:
> >>>
> >>>
> >>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>>>
> >>>>> If it doesn’t have all the bells and whistles, then it shouldn’t
> >> be on port 53 by default.
> >>>> Sure, I’ll change the default port to not use 53 and document it.
> >>>>> *how* is it getting launched on a privileged port? It sounds like
> >> the expectation is to run “command” as root. *ALL* of the previous
> >> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t
> this
> >> one? These questions matter from a security standpoint.
> >>>> Yes, it is running as “root” to be able to use the privileged port.
> The
> >> DNS server is not yet integrated with the hadoop script.
> >>>>
> >>>>> Check the output. It’s pretty obviously borked:
> >>>> Thanks for pointing out. Missed this when rebasing onto trunk.
> >>>
> >>>
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >>>
> >>> * A bunch of mostly new Java code that may or may not have
> >> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> >>> * ~1/3 of the docs are roadmap/TBD
> >>> * ~1/3 of the docs are for an optional DNS daemon that has
> >> no end user hook to start it
> >>> * ~1/3 of the docs are for a REST API that comes from some
> >> undefined daemon (apiserver?)
> >>> * Two new, but undocumented, subcommands to yarn
> >>> * There are no docs for admins or users on how to actually
> >> start or use this completely new/separate/optional feature
> >>>
> >>> How are outside people (e.g., non-branch committers) supposed to
> >> test this new feature under these conditions?
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Thanks for your consideration Jian, let's track this for GA then.
Best,
Andrew
On Fri, Sep 8, 2017 at 3:02 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi Andrew,
>
> At this point, there are no more release blockers including documentations
> from our side - all work done.
> But I agree it is too close to the release, after talking with other team
> members, we are fine to drop this from beta,
>
> And we want to target this for GA.
> I’m withdrawing this vote and will start afresh vote later for GA.
> Thanks all who voted this effort !
>
> Thanks,
> Jian
>
>
> > On Sep 7, 2017, at 3:59 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> > Hi folks,
> >
> > This vote closes today. I see a -1 from Allen on inclusion in beta1. I
> see
> > there's active fixing going on, but given that we're one week out from
> RC0,
> > I think we should drop this from beta1.
> >
> > Allen, Jian, others, is this reasonable? What release should we retarget
> > this for? I don't have a sense for how much work there is left to do, but
> > as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> >
> > Best,
> > Andrew
> >
> > On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> >
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >> Sorry for confusion, I was actively writing the formal documentation for
> >> how to use/how it works etc. and will post soon in a few hours.
> >>
> >>
> >>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <
> aw@effectivemachines.com>
> >> wrote:
> >>>
> >>>
> >>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>>>
> >>>>> If it doesn’t have all the bells and whistles, then it shouldn’t
> >> be on port 53 by default.
> >>>> Sure, I’ll change the default port to not use 53 and document it.
> >>>>> *how* is it getting launched on a privileged port? It sounds like
> >> the expectation is to run “command” as root. *ALL* of the previous
> >> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t
> this
> >> one? These questions matter from a security standpoint.
> >>>> Yes, it is running as “root” to be able to use the privileged port.
> The
> >> DNS server is not yet integrated with the hadoop script.
> >>>>
> >>>>> Check the output. It’s pretty obviously borked:
> >>>> Thanks for pointing out. Missed this when rebasing onto trunk.
> >>>
> >>>
> >>> Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >>>
> >>> * A bunch of mostly new Java code that may or may not have
> >> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> >>> * ~1/3 of the docs are roadmap/TBD
> >>> * ~1/3 of the docs are for an optional DNS daemon that has
> >> no end user hook to start it
> >>> * ~1/3 of the docs are for a REST API that comes from some
> >> undefined daemon (apiserver?)
> >>> * Two new, but undocumented, subcommands to yarn
> >>> * There are no docs for admins or users on how to actually
> >> start or use this completely new/separate/optional feature
> >>>
> >>> How are outside people (e.g., non-branch committers) supposed to
> >> test this new feature under these conditions?
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>
> >>
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Andrew,
At this point, there are no more release blockers including documentations from our side - all work done.
But I agree it is too close to the release, after talking with other team members, we are fine to drop this from beta,
And we want to target this for GA.
I’m withdrawing this vote and will start afresh vote later for GA.
Thanks all who voted this effort !
Thanks,
Jian
> On Sep 7, 2017, at 3:59 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> Hi folks,
>
> This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
> there's active fixing going on, but given that we're one week out from RC0,
> I think we should drop this from beta1.
>
> Allen, Jian, others, is this reasonable? What release should we retarget
> this for? I don't have a sense for how much work there is left to do, but
> as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
>
> Best,
> Andrew
>
> On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
>
>>> Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>> Sorry for confusion, I was actively writing the formal documentation for
>> how to use/how it works etc. and will post soon in a few hours.
>>
>>
>>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com>
>> wrote:
>>>
>>>
>>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>>>
>>>>> If it doesn’t have all the bells and whistles, then it shouldn’t
>> be on port 53 by default.
>>>> Sure, I’ll change the default port to not use 53 and document it.
>>>>> *how* is it getting launched on a privileged port? It sounds like
>> the expectation is to run “command” as root. *ALL* of the previous
>> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this
>> one? These questions matter from a security standpoint.
>>>> Yes, it is running as “root” to be able to use the privileged port. The
>> DNS server is not yet integrated with the hadoop script.
>>>>
>>>>> Check the output. It’s pretty obviously borked:
>>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>>>
>>>
>>> Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>>>
>>> * A bunch of mostly new Java code that may or may not have
>> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>>> * ~1/3 of the docs are roadmap/TBD
>>> * ~1/3 of the docs are for an optional DNS daemon that has
>> no end user hook to start it
>>> * ~1/3 of the docs are for a REST API that comes from some
>> undefined daemon (apiserver?)
>>> * Two new, but undocumented, subcommands to yarn
>>> * There are no docs for admins or users on how to actually
>> start or use this completely new/separate/optional feature
>>>
>>> How are outside people (e.g., non-branch committers) supposed to
>> test this new feature under these conditions?
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Andrew,
At this point, there are no more release blockers including documentations from our side - all work done.
But I agree it is too close to the release, after talking with other team members, we are fine to drop this from beta,
And we want to target this for GA.
I’m withdrawing this vote and will start afresh vote later for GA.
Thanks all who voted this effort !
Thanks,
Jian
> On Sep 7, 2017, at 3:59 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> Hi folks,
>
> This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
> there's active fixing going on, but given that we're one week out from RC0,
> I think we should drop this from beta1.
>
> Allen, Jian, others, is this reasonable? What release should we retarget
> this for? I don't have a sense for how much work there is left to do, but
> as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
>
> Best,
> Andrew
>
> On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
>
>>> Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>> Sorry for confusion, I was actively writing the formal documentation for
>> how to use/how it works etc. and will post soon in a few hours.
>>
>>
>>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com>
>> wrote:
>>>
>>>
>>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>>>
>>>>> If it doesn’t have all the bells and whistles, then it shouldn’t
>> be on port 53 by default.
>>>> Sure, I’ll change the default port to not use 53 and document it.
>>>>> *how* is it getting launched on a privileged port? It sounds like
>> the expectation is to run “command” as root. *ALL* of the previous
>> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this
>> one? These questions matter from a security standpoint.
>>>> Yes, it is running as “root” to be able to use the privileged port. The
>> DNS server is not yet integrated with the hadoop script.
>>>>
>>>>> Check the output. It’s pretty obviously borked:
>>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>>>
>>>
>>> Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>>>
>>> * A bunch of mostly new Java code that may or may not have
>> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>>> * ~1/3 of the docs are roadmap/TBD
>>> * ~1/3 of the docs are for an optional DNS daemon that has
>> no end user hook to start it
>>> * ~1/3 of the docs are for a REST API that comes from some
>> undefined daemon (apiserver?)
>>> * Two new, but undocumented, subcommands to yarn
>>> * There are no docs for admins or users on how to actually
>> start or use this completely new/separate/optional feature
>>>
>>> How are outside people (e.g., non-branch committers) supposed to
>> test this new feature under these conditions?
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Andrew,
At this point, there are no more release blockers including documentations from our side - all work done.
But I agree it is too close to the release, after talking with other team members, we are fine to drop this from beta,
And we want to target this for GA.
I’m withdrawing this vote and will start afresh vote later for GA.
Thanks all who voted this effort !
Thanks,
Jian
> On Sep 7, 2017, at 3:59 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> Hi folks,
>
> This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
> there's active fixing going on, but given that we're one week out from RC0,
> I think we should drop this from beta1.
>
> Allen, Jian, others, is this reasonable? What release should we retarget
> this for? I don't have a sense for how much work there is left to do, but
> as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
>
> Best,
> Andrew
>
> On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
>
>>> Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>> Sorry for confusion, I was actively writing the formal documentation for
>> how to use/how it works etc. and will post soon in a few hours.
>>
>>
>>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com>
>> wrote:
>>>
>>>
>>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>>>
>>>>> If it doesn’t have all the bells and whistles, then it shouldn’t
>> be on port 53 by default.
>>>> Sure, I’ll change the default port to not use 53 and document it.
>>>>> *how* is it getting launched on a privileged port? It sounds like
>> the expectation is to run “command” as root. *ALL* of the previous
>> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this
>> one? These questions matter from a security standpoint.
>>>> Yes, it is running as “root” to be able to use the privileged port. The
>> DNS server is not yet integrated with the hadoop script.
>>>>
>>>>> Check the output. It’s pretty obviously borked:
>>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>>>
>>>
>>> Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>>>
>>> * A bunch of mostly new Java code that may or may not have
>> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>>> * ~1/3 of the docs are roadmap/TBD
>>> * ~1/3 of the docs are for an optional DNS daemon that has
>> no end user hook to start it
>>> * ~1/3 of the docs are for a REST API that comes from some
>> undefined daemon (apiserver?)
>>> * Two new, but undocumented, subcommands to yarn
>>> * There are no docs for admins or users on how to actually
>> start or use this completely new/separate/optional feature
>>>
>>> How are outside people (e.g., non-branch committers) supposed to
>> test this new feature under these conditions?
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi folks,
This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
there's active fixing going on, but given that we're one week out from RC0,
I think we should drop this from beta1.
Allen, Jian, others, is this reasonable? What release should we retarget
this for? I don't have a sense for how much work there is left to do, but
as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
Best,
Andrew
On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for
> how to use/how it works etc. and will post soon in a few hours.
>
>
> > On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com>
> wrote:
> >
> >
> >> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>
> >>> If it doesn’t have all the bells and whistles, then it shouldn’t
> be on port 53 by default.
> >> Sure, I’ll change the default port to not use 53 and document it.
> >>> *how* is it getting launched on a privileged port? It sounds like
> the expectation is to run “command” as root. *ALL* of the previous
> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this
> one? These questions matter from a security standpoint.
> >> Yes, it is running as “root” to be able to use the privileged port. The
> DNS server is not yet integrated with the hadoop script.
> >>
> >>> Check the output. It’s pretty obviously borked:
> >> Thanks for pointing out. Missed this when rebasing onto trunk.
> >
> >
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> >
> > * A bunch of mostly new Java code that may or may not have
> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> > * ~1/3 of the docs are roadmap/TBD
> > * ~1/3 of the docs are for an optional DNS daemon that has
> no end user hook to start it
> > * ~1/3 of the docs are for a REST API that comes from some
> undefined daemon (apiserver?)
> > * Two new, but undocumented, subcommands to yarn
> > * There are no docs for admins or users on how to actually
> start or use this completely new/separate/optional feature
> >
> > How are outside people (e.g., non-branch committers) supposed to
> test this new feature under these conditions?
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Allen,
Thanks for sharing the feedback. I opened YARN-7191 for addressing the feedback.
We can move the discussions there.
Thanks,
Jian
> On Sep 13, 2017, at 10:10 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 8, 2017, at 9:25 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>> Hi Allen,
>> The documentations are committed. Please check QuickStart.md and others in the same folder.
>> YarnCommands.md doc is updated to include new commands.
>> DNS default port is also documented.
>> Would you like to give a look and see if it address your concerns ?
>
> Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on.
>
> Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions:
>
> "A host of scheduling features are being developed to support long running services.”
>
> Yeah, ok? How is this a concept?
>
> or
>
> "[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.”
>
>
> The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA?
>
> If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries.
>
> etc, etc.
>
> ...
>
> The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here.
>
> Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.)
>
> …
>
> Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents?
>
> "Start the DNS server”
>
> Just… yikes.
>
> a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here.
> b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root.
> c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free.
> d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually….
>
>
> Given:
>
> "3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.”
>
> then:
>
> "It should not be used as a fully-functional corporate DNS.”
>
> Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...)
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Allen,
Thanks for sharing the feedback. I opened YARN-7191 for addressing the feedback.
We can move the discussions there.
Thanks,
Jian
> On Sep 13, 2017, at 10:10 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 8, 2017, at 9:25 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>> Hi Allen,
>> The documentations are committed. Please check QuickStart.md and others in the same folder.
>> YarnCommands.md doc is updated to include new commands.
>> DNS default port is also documented.
>> Would you like to give a look and see if it address your concerns ?
>
> Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on.
>
> Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions:
>
> "A host of scheduling features are being developed to support long running services.”
>
> Yeah, ok? How is this a concept?
>
> or
>
> "[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.”
>
>
> The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA?
>
> If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries.
>
> etc, etc.
>
> ...
>
> The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here.
>
> Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.)
>
> …
>
> Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents?
>
> "Start the DNS server”
>
> Just… yikes.
>
> a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here.
> b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root.
> c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free.
> d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually….
>
>
> Given:
>
> "3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.”
>
> then:
>
> "It should not be used as a fully-functional corporate DNS.”
>
> Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...)
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Allen,
Thanks for sharing the feedback. I opened YARN-7191 for addressing the feedback.
We can move the discussions there.
Thanks,
Jian
> On Sep 13, 2017, at 10:10 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 8, 2017, at 9:25 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>> Hi Allen,
>> The documentations are committed. Please check QuickStart.md and others in the same folder.
>> YarnCommands.md doc is updated to include new commands.
>> DNS default port is also documented.
>> Would you like to give a look and see if it address your concerns ?
>
> Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on.
>
> Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions:
>
> "A host of scheduling features are being developed to support long running services.”
>
> Yeah, ok? How is this a concept?
>
> or
>
> "[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.”
>
>
> The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA?
>
> If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries.
>
> etc, etc.
>
> ...
>
> The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here.
>
> Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.)
>
> …
>
> Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents?
>
> "Start the DNS server”
>
> Just… yikes.
>
> a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here.
> b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root.
> c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free.
> d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually….
>
>
> Given:
>
> "3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.”
>
> then:
>
> "It should not be used as a fully-functional corporate DNS.”
>
> Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...)
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2017, at 9:25 AM, Jian He <jh...@hortonworks.com> wrote:
>
> Hi Allen,
> The documentations are committed. Please check QuickStart.md and others in the same folder.
> YarnCommands.md doc is updated to include new commands.
> DNS default port is also documented.
> Would you like to give a look and see if it address your concerns ?
Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on.
Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions:
"A host of scheduling features are being developed to support long running services.”
Yeah, ok? How is this a concept?
or
"[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.”
The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA?
If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries.
etc, etc.
...
The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here.
Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.)
…
Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents?
"Start the DNS server”
Just… yikes.
a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here.
b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root.
c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free.
d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually….
Given:
"3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.”
then:
"It should not be used as a fully-functional corporate DNS.”
Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2017, at 9:25 AM, Jian He <jh...@hortonworks.com> wrote:
>
> Hi Allen,
> The documentations are committed. Please check QuickStart.md and others in the same folder.
> YarnCommands.md doc is updated to include new commands.
> DNS default port is also documented.
> Would you like to give a look and see if it address your concerns ?
Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on.
Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions:
"A host of scheduling features are being developed to support long running services.”
Yeah, ok? How is this a concept?
or
"[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.”
The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA?
If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries.
etc, etc.
...
The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here.
Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.)
…
Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents?
"Start the DNS server”
Just… yikes.
a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here.
b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root.
c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free.
d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually….
Given:
"3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.”
then:
"It should not be used as a fully-functional corporate DNS.”
Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2017, at 9:25 AM, Jian He <jh...@hortonworks.com> wrote:
>
> Hi Allen,
> The documentations are committed. Please check QuickStart.md and others in the same folder.
> YarnCommands.md doc is updated to include new commands.
> DNS default port is also documented.
> Would you like to give a look and see if it address your concerns ?
Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on.
Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions:
"A host of scheduling features are being developed to support long running services.”
Yeah, ok? How is this a concept?
or
"[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.”
The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA?
If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries.
etc, etc.
...
The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here.
Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.)
…
Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents?
"Start the DNS server”
Just… yikes.
a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here.
b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root.
c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free.
d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually….
Given:
"3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.”
then:
"It should not be used as a fully-functional corporate DNS.”
Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2017, at 9:25 AM, Jian He <jh...@hortonworks.com> wrote:
>
> Hi Allen,
> The documentations are committed. Please check QuickStart.md and others in the same folder.
> YarnCommands.md doc is updated to include new commands.
> DNS default port is also documented.
> Would you like to give a look and see if it address your concerns ?
Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on.
Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions:
"A host of scheduling features are being developed to support long running services.”
Yeah, ok? How is this a concept?
or
"[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.”
The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA?
If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries.
etc, etc.
...
The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here.
Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.)
…
Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents?
"Start the DNS server”
Just… yikes.
a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here.
b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root.
c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free.
d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually….
Given:
"3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.”
then:
"It should not be used as a fully-functional corporate DNS.”
Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Allen,
The documentations are committed. Please check QuickStart.md and others in the same folder.
YarnCommands.md doc is updated to include new commands.
DNS default port is also documented.
Would you like to give a look and see if it address your concerns ?
Jian
> On Sep 6, 2017, at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for how to use/how it works etc. and will post soon in a few hours.
>
>
>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>>
>>>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
>>> Sure, I’ll change the default port to not use 53 and document it.
>>>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>>> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>>>
>>>> Check the output. It’s pretty obviously borked:
>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>>
>>
>> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
>>
>> * A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>> * ~1/3 of the docs are roadmap/TBD
>> * ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
>> * ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
>> * Two new, but undocumented, subcommands to yarn
>> * There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
>>
>> How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
>>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi folks,
This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
there's active fixing going on, but given that we're one week out from RC0,
I think we should drop this from beta1.
Allen, Jian, others, is this reasonable? What release should we retarget
this for? I don't have a sense for how much work there is left to do, but
as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
Best,
Andrew
On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for
> how to use/how it works etc. and will post soon in a few hours.
>
>
> > On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com>
> wrote:
> >
> >
> >> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>
> >>> If it doesn’t have all the bells and whistles, then it shouldn’t
> be on port 53 by default.
> >> Sure, I’ll change the default port to not use 53 and document it.
> >>> *how* is it getting launched on a privileged port? It sounds like
> the expectation is to run “command” as root. *ALL* of the previous
> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this
> one? These questions matter from a security standpoint.
> >> Yes, it is running as “root” to be able to use the privileged port. The
> DNS server is not yet integrated with the hadoop script.
> >>
> >>> Check the output. It’s pretty obviously borked:
> >> Thanks for pointing out. Missed this when rebasing onto trunk.
> >
> >
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> >
> > * A bunch of mostly new Java code that may or may not have
> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> > * ~1/3 of the docs are roadmap/TBD
> > * ~1/3 of the docs are for an optional DNS daemon that has
> no end user hook to start it
> > * ~1/3 of the docs are for a REST API that comes from some
> undefined daemon (apiserver?)
> > * Two new, but undocumented, subcommands to yarn
> > * There are no docs for admins or users on how to actually
> start or use this completely new/separate/optional feature
> >
> > How are outside people (e.g., non-branch committers) supposed to
> test this new feature under these conditions?
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Allen,
The documentations are committed. Please check QuickStart.md and others in the same folder.
YarnCommands.md doc is updated to include new commands.
DNS default port is also documented.
Would you like to give a look and see if it address your concerns ?
Jian
> On Sep 6, 2017, at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for how to use/how it works etc. and will post soon in a few hours.
>
>
>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>>
>>>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
>>> Sure, I’ll change the default port to not use 53 and document it.
>>>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>>> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>>>
>>>> Check the output. It’s pretty obviously borked:
>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>>
>>
>> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
>>
>> * A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>> * ~1/3 of the docs are roadmap/TBD
>> * ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
>> * ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
>> * Two new, but undocumented, subcommands to yarn
>> * There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
>>
>> How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
>>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi folks,
This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
there's active fixing going on, but given that we're one week out from RC0,
I think we should drop this from beta1.
Allen, Jian, others, is this reasonable? What release should we retarget
this for? I don't have a sense for how much work there is left to do, but
as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
Best,
Andrew
On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for
> how to use/how it works etc. and will post soon in a few hours.
>
>
> > On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com>
> wrote:
> >
> >
> >> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>
> >>> If it doesn’t have all the bells and whistles, then it shouldn’t
> be on port 53 by default.
> >> Sure, I’ll change the default port to not use 53 and document it.
> >>> *how* is it getting launched on a privileged port? It sounds like
> the expectation is to run “command” as root. *ALL* of the previous
> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this
> one? These questions matter from a security standpoint.
> >> Yes, it is running as “root” to be able to use the privileged port. The
> DNS server is not yet integrated with the hadoop script.
> >>
> >>> Check the output. It’s pretty obviously borked:
> >> Thanks for pointing out. Missed this when rebasing onto trunk.
> >
> >
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> >
> > * A bunch of mostly new Java code that may or may not have
> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> > * ~1/3 of the docs are roadmap/TBD
> > * ~1/3 of the docs are for an optional DNS daemon that has
> no end user hook to start it
> > * ~1/3 of the docs are for a REST API that comes from some
> undefined daemon (apiserver?)
> > * Two new, but undocumented, subcommands to yarn
> > * There are no docs for admins or users on how to actually
> start or use this completely new/separate/optional feature
> >
> > How are outside people (e.g., non-branch committers) supposed to
> test this new feature under these conditions?
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
Hi Allen,
The documentations are committed. Please check QuickStart.md and others in the same folder.
YarnCommands.md doc is updated to include new commands.
DNS default port is also documented.
Would you like to give a look and see if it address your concerns ?
Jian
> On Sep 6, 2017, at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for how to use/how it works etc. and will post soon in a few hours.
>
>
>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>>
>>>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
>>> Sure, I’ll change the default port to not use 53 and document it.
>>>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>>> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>>>
>>>> Check the output. It’s pretty obviously borked:
>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>>
>>
>> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
>>
>> * A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>> * ~1/3 of the docs are roadmap/TBD
>> * ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
>> * ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
>> * Two new, but undocumented, subcommands to yarn
>> * There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
>>
>> How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
>>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi folks,
This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
there's active fixing going on, but given that we're one week out from RC0,
I think we should drop this from beta1.
Allen, Jian, others, is this reasonable? What release should we retarget
this for? I don't have a sense for how much work there is left to do, but
as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
Best,
Andrew
On Wed, Sep 6, 2017 at 10:19 AM, Jian He <jh...@hortonworks.com> wrote:
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for
> how to use/how it works etc. and will post soon in a few hours.
>
>
> > On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com>
> wrote:
> >
> >
> >> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
> >>
> >>> If it doesn’t have all the bells and whistles, then it shouldn’t
> be on port 53 by default.
> >> Sure, I’ll change the default port to not use 53 and document it.
> >>> *how* is it getting launched on a privileged port? It sounds like
> the expectation is to run “command” as root. *ALL* of the previous
> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this
> one? These questions matter from a security standpoint.
> >> Yes, it is running as “root” to be able to use the privileged port. The
> DNS server is not yet integrated with the hadoop script.
> >>
> >>> Check the output. It’s pretty obviously borked:
> >> Thanks for pointing out. Missed this when rebasing onto trunk.
> >
> >
> > Please correct me if I’m wrong, but the current summary of the
> branch, post these changes, looks like:
> >
> > * A bunch of mostly new Java code that may or may not have
> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> > * ~1/3 of the docs are roadmap/TBD
> > * ~1/3 of the docs are for an optional DNS daemon that has
> no end user hook to start it
> > * ~1/3 of the docs are for a REST API that comes from some
> undefined daemon (apiserver?)
> > * Two new, but undocumented, subcommands to yarn
> > * There are no docs for admins or users on how to actually
> start or use this completely new/separate/optional feature
> >
> > How are outside people (e.g., non-branch committers) supposed to
> test this new feature under these conditions?
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
Sorry for confusion, I was actively writing the formal documentation for how to use/how it works etc. and will post soon in a few hours.
> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
>> Sure, I’ll change the default port to not use 53 and document it.
>>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>>
>>> Check the output. It’s pretty obviously borked:
>> Thanks for pointing out. Missed this when rebasing onto trunk.
>
>
> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
>
> * A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> * ~1/3 of the docs are roadmap/TBD
> * ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
> * ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
> * Two new, but undocumented, subcommands to yarn
> * There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
>
> How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
Sorry for confusion, I was actively writing the formal documentation for how to use/how it works etc. and will post soon in a few hours.
> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
>> Sure, I’ll change the default port to not use 53 and document it.
>>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>>
>>> Check the output. It’s pretty obviously borked:
>> Thanks for pointing out. Missed this when rebasing onto trunk.
>
>
> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
>
> * A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> * ~1/3 of the docs are roadmap/TBD
> * ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
> * ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
> * Two new, but undocumented, subcommands to yarn
> * There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
>
> How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
Sorry for confusion, I was actively writing the formal documentation for how to use/how it works etc. and will post soon in a few hours.
> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
>> Sure, I’ll change the default port to not use 53 and document it.
>>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>>
>>> Check the output. It’s pretty obviously borked:
>> Thanks for pointing out. Missed this when rebasing onto trunk.
>
>
> Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
>
> * A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> * ~1/3 of the docs are roadmap/TBD
> * ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
> * ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
> * Two new, but undocumented, subcommands to yarn
> * There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
>
> How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
> Sure, I’ll change the default port to not use 53 and document it.
>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>
>> Check the output. It’s pretty obviously borked:
> Thanks for pointing out. Missed this when rebasing onto trunk.
Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
* A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
* ~1/3 of the docs are roadmap/TBD
* ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
* ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
* Two new, but undocumented, subcommands to yarn
* There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
> Sure, I’ll change the default port to not use 53 and document it.
>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>
>> Check the output. It’s pretty obviously borked:
> Thanks for pointing out. Missed this when rebasing onto trunk.
Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
* A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
* ~1/3 of the docs are roadmap/TBD
* ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
* ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
* Two new, but undocumented, subcommands to yarn
* There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
> Sure, I’ll change the default port to not use 53 and document it.
>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>
>> Check the output. It’s pretty obviously borked:
> Thanks for pointing out. Missed this when rebasing onto trunk.
Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
* A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
* ~1/3 of the docs are roadmap/TBD
* ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
* ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
* Two new, but undocumented, subcommands to yarn
* There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 6:23 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
> Sure, I’ll change the default port to not use 53 and document it.
>> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
> Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
>
>> Check the output. It’s pretty obviously borked:
> Thanks for pointing out. Missed this when rebasing onto trunk.
Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like:
* A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835)
* ~1/3 of the docs are roadmap/TBD
* ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it
* ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?)
* Two new, but undocumented, subcommands to yarn
* There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature
How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions?
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
Sure, I’ll change the default port to not use 53 and document it.
> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
> Check the output. It’s pretty obviously borked:
Thanks for pointing out. Missed this when rebasing onto trunk.
> On Sep 5, 2017, at 3:11 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 5, 2017, at 2:53 PM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>>
>> It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
>
> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port.
>
>>> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
>>
>> Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
>
> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>
>>> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
>>
>> The “yarn” usage command is working for me. what do you mean ?
>
> Check the output. It’s pretty obviously borked:
>
> ===snip====
>
> Daemon Commands:
>
> nodemanager run a nodemanager on each worker
> proxyserver run the web app proxy server
> resourcemanager run the ResourceManager
> router run the Router daemon
> timelineserver run the timeline server
>
> Run a service Commands:
>
> service run a service
>
> Run yarn-native-service rest server Commands:
>
> apiserver run yarn-native-service rest server
>
>
> ===snip===
>
>> Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
>
> Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation….
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
Sure, I’ll change the default port to not use 53 and document it.
> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
> Check the output. It’s pretty obviously borked:
Thanks for pointing out. Missed this when rebasing onto trunk.
> On Sep 5, 2017, at 3:11 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 5, 2017, at 2:53 PM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>>
>> It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
>
> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port.
>
>>> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
>>
>> Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
>
> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>
>>> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
>>
>> The “yarn” usage command is working for me. what do you mean ?
>
> Check the output. It’s pretty obviously borked:
>
> ===snip====
>
> Daemon Commands:
>
> nodemanager run a nodemanager on each worker
> proxyserver run the web app proxy server
> resourcemanager run the ResourceManager
> router run the Router daemon
> timelineserver run the timeline server
>
> Run a service Commands:
>
> service run a service
>
> Run yarn-native-service rest server Commands:
>
> apiserver run yarn-native-service rest server
>
>
> ===snip===
>
>> Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
>
> Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation….
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default.
Sure, I’ll change the default port to not use 53 and document it.
> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
Yes, it is running as “root” to be able to use the privileged port. The DNS server is not yet integrated with the hadoop script.
> Check the output. It’s pretty obviously borked:
Thanks for pointing out. Missed this when rebasing onto trunk.
> On Sep 5, 2017, at 3:11 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 5, 2017, at 2:53 PM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>>
>> It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
>
> If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port.
>
>>> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
>>
>> Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
>
> *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>
>>> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
>>
>> The “yarn” usage command is working for me. what do you mean ?
>
> Check the output. It’s pretty obviously borked:
>
> ===snip====
>
> Daemon Commands:
>
> nodemanager run a nodemanager on each worker
> proxyserver run the web app proxy server
> resourcemanager run the ResourceManager
> router run the Router daemon
> timelineserver run the timeline server
>
> Run a service Commands:
>
> service run a service
>
> Run yarn-native-service rest server Commands:
>
> apiserver run yarn-native-service rest server
>
>
> ===snip===
>
>> Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
>
> Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation….
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 2:53 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>
> It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port.
>> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
>
> Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
*how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
>
> The “yarn” usage command is working for me. what do you mean ?
Check the output. It’s pretty obviously borked:
===snip====
Daemon Commands:
nodemanager run a nodemanager on each worker
proxyserver run the web app proxy server
resourcemanager run the ResourceManager
router run the Router daemon
timelineserver run the timeline server
Run a service Commands:
service run a service
Run yarn-native-service rest server Commands:
apiserver run yarn-native-service rest server
===snip===
> Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation….
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 2:53 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>
> It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port.
>> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
>
> Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
*how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
>
> The “yarn” usage command is working for me. what do you mean ?
Check the output. It’s pretty obviously borked:
===snip====
Daemon Commands:
nodemanager run a nodemanager on each worker
proxyserver run the web app proxy server
resourcemanager run the ResourceManager
router run the Router daemon
timelineserver run the timeline server
Run a service Commands:
service run a service
Run yarn-native-service rest server Commands:
apiserver run yarn-native-service rest server
===snip===
> Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation….
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 2:53 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>
> It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port.
>> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
>
> Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
*how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
>
> The “yarn” usage command is working for me. what do you mean ?
Check the output. It’s pretty obviously borked:
===snip====
Daemon Commands:
nodemanager run a nodemanager on each worker
proxyserver run the web app proxy server
resourcemanager run the ResourceManager
router run the Router daemon
timelineserver run the timeline server
Run a service Commands:
service run a service
Run yarn-native-service rest server Commands:
apiserver run yarn-native-service rest server
===snip===
> Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation….
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 5, 2017, at 2:53 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
>
> It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port.
>> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
>
> Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
*how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint.
>> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
>
> The “yarn” usage command is working for me. what do you mean ?
Check the output. It’s pretty obviously borked:
===snip====
Daemon Commands:
nodemanager run a nodemanager on each worker
proxyserver run the web app proxy server
resourcemanager run the ResourceManager
router run the Router daemon
timelineserver run the timeline server
Run a service Commands:
service run a service
Run yarn-native-service rest server Commands:
apiserver run yarn-native-service rest server
===snip===
> Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation….
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> 1) Did I miss it or is there no actual end-user documentation on how to use this?
Yes, we are in the process of finishing up the the doc and posting it. We considered this a release blocker for 3.0.0-beta1 and so working on it in parallel while the branch merge happens.
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever.
Thanks for pointing out. We will fix this.
> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
The “yarn” usage command is working for me. what do you mean ?
Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> 1) Did I miss it or is there no actual end-user documentation on how to use this?
Yes, we are in the process of finishing up the the doc and posting it. We considered this a release blocker for 3.0.0-beta1 and so working on it in parallel while the branch merge happens.
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever.
Thanks for pointing out. We will fix this.
> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
The “yarn” usage command is working for me. what do you mean ?
Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Jian He <jh...@hortonworks.com>.
> 1) Did I miss it or is there no actual end-user documentation on how to use this?
Yes, we are in the process of finishing up the the doc and posting it. We considered this a release blocker for 3.0.0-beta1 and so working on it in parallel while the branch merge happens.
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever.
Thanks for pointing out. We will fix this.
> Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc.
It seems like this is a rehash of some of the discussion you and others had on the JIRA. The DNS here is a thin layer backed by service registry. My understanding from the JIRA is that there are no claims that this is already a DNS with all the bells and whistles - its goal is mainly to expose dynamic services running on YARN as end-points. Clearly, this is an optional daemon, if the provided feature set is deemed insufficient, an alternative solution can be plugged in by specific admins because the DNS piece is completely decoupled from the rest of native-services.
> In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
Yes, we have tested this DNS server on port 53 on a cluster by running the DNS server as root user. The port is clearly configurable, so the admin has two options. Run as root + port 53. Run as non-root + non-privileged port. We tested and left it as port 53 to keep it on a standard DNS port. It is already documented as such though I can see that part can be improved a little.
> 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
The “yarn” usage command is working for me. what do you mean ?
Yeah, looks like some previous features also forgot to update YarnCommands.md for the new sub commands
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
2) Lots of markdown problems in the NativeServicesDiscovery.md document.
This includes things like Œyarnsite.xml¹ (missing a dash.)
The md patch uploaded to YARN-5244 had some special chars. I fixed those
in YARN-7161.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 31, 2017, at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> I would like to call a vote for merging yarn-native-services to trunk.
1) Did I miss it or is there no actual end-user documentation on how to use this? I see hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md, but that’s not particularly useful. It looks like there are daemons that need to get started, based on other documentation? How? What do I configure? Is there a command to use to say “go do native for this job”? I honestly have no idea how to make this do anything because most of the docs appear to be either TBD or expect me to read through a ton of JIRAs.
2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever.
3) The default port for the DNS server should NOT be 53 if typical deployments need to specify an alternate port. Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc. In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
At this point in time:
-1 on 3.0.0-beta1
-0 on trunk
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Arun Suresh <as...@apache.org>.
+1 (binding).
Cheers
-Arun
On Fri, Sep 1, 2017 at 5:24 PM, Wangda Tan <wh...@gmail.com> wrote:
> +1 (Binding), I tried to use YARN service assembly before to run different
> kinds of jobs (for example, distributed Tensorflow), it is really easy for
> end user to run jobs on YARN.
>
> Thanks to the whole team for the great job!
>
> Best,
> Wangda
>
>
> On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 (non-binding)
> >
> > On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
> >
> > >+1 (non-binding)
> > >
> > >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> I would like to call a vote for merging yarn-native-services to trunk.
> > >>The
> > >> vote will run for 7 days as usual.
> > >>
> > >> At a high level, the following are the key feautres implemented.
> > >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> > >>and
> > >> orchestrate existing services to YARN either docker or non-docker
> based.
> > >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> > >> simple JSON spec
> > >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> > >> service to enable users to discover services deployed on YARN
> > >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> > >> All these new services are optional and are sitting outside of the
> > >> existing system, and have no impact on existing system if disabled.
> > >>
> > >> Special thanks to a team of folks who worked hard towards this: Billie
> > >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> > >>Sharma
> > >> K S, Sunil G, Akhil PB. This effort could not be possible without
> their
> > >> ideas and hard work.
> > >>
> > >> Thanks,
> > >> Jian
> > >>
> > >> [1] https://issues.apache.org/jira/browse/YARN-5079
> > >> [2] https://issues.apache.org/jira/browse/YARN-4793
> > >> [3] https://issues.apache.org/jira/browse/YARN-4757
> > >> [4] https://issues.apache.org/jira/browse/YARN-6419
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Arun Suresh <as...@apache.org>.
+1 (binding).
Cheers
-Arun
On Fri, Sep 1, 2017 at 5:24 PM, Wangda Tan <wh...@gmail.com> wrote:
> +1 (Binding), I tried to use YARN service assembly before to run different
> kinds of jobs (for example, distributed Tensorflow), it is really easy for
> end user to run jobs on YARN.
>
> Thanks to the whole team for the great job!
>
> Best,
> Wangda
>
>
> On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 (non-binding)
> >
> > On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
> >
> > >+1 (non-binding)
> > >
> > >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> I would like to call a vote for merging yarn-native-services to trunk.
> > >>The
> > >> vote will run for 7 days as usual.
> > >>
> > >> At a high level, the following are the key feautres implemented.
> > >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> > >>and
> > >> orchestrate existing services to YARN either docker or non-docker
> based.
> > >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> > >> simple JSON spec
> > >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> > >> service to enable users to discover services deployed on YARN
> > >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> > >> All these new services are optional and are sitting outside of the
> > >> existing system, and have no impact on existing system if disabled.
> > >>
> > >> Special thanks to a team of folks who worked hard towards this: Billie
> > >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> > >>Sharma
> > >> K S, Sunil G, Akhil PB. This effort could not be possible without
> their
> > >> ideas and hard work.
> > >>
> > >> Thanks,
> > >> Jian
> > >>
> > >> [1] https://issues.apache.org/jira/browse/YARN-5079
> > >> [2] https://issues.apache.org/jira/browse/YARN-4793
> > >> [3] https://issues.apache.org/jira/browse/YARN-4757
> > >> [4] https://issues.apache.org/jira/browse/YARN-6419
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Arun Suresh <as...@apache.org>.
+1 (binding).
Cheers
-Arun
On Fri, Sep 1, 2017 at 5:24 PM, Wangda Tan <wh...@gmail.com> wrote:
> +1 (Binding), I tried to use YARN service assembly before to run different
> kinds of jobs (for example, distributed Tensorflow), it is really easy for
> end user to run jobs on YARN.
>
> Thanks to the whole team for the great job!
>
> Best,
> Wangda
>
>
> On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 (non-binding)
> >
> > On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
> >
> > >+1 (non-binding)
> > >
> > >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> I would like to call a vote for merging yarn-native-services to trunk.
> > >>The
> > >> vote will run for 7 days as usual.
> > >>
> > >> At a high level, the following are the key feautres implemented.
> > >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> > >>and
> > >> orchestrate existing services to YARN either docker or non-docker
> based.
> > >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> > >> simple JSON spec
> > >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> > >> service to enable users to discover services deployed on YARN
> > >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> > >> All these new services are optional and are sitting outside of the
> > >> existing system, and have no impact on existing system if disabled.
> > >>
> > >> Special thanks to a team of folks who worked hard towards this: Billie
> > >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> > >>Sharma
> > >> K S, Sunil G, Akhil PB. This effort could not be possible without
> their
> > >> ideas and hard work.
> > >>
> > >> Thanks,
> > >> Jian
> > >>
> > >> [1] https://issues.apache.org/jira/browse/YARN-5079
> > >> [2] https://issues.apache.org/jira/browse/YARN-4793
> > >> [3] https://issues.apache.org/jira/browse/YARN-4757
> > >> [4] https://issues.apache.org/jira/browse/YARN-6419
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Arun Suresh <as...@apache.org>.
+1 (binding).
Cheers
-Arun
On Fri, Sep 1, 2017 at 5:24 PM, Wangda Tan <wh...@gmail.com> wrote:
> +1 (Binding), I tried to use YARN service assembly before to run different
> kinds of jobs (for example, distributed Tensorflow), it is really easy for
> end user to run jobs on YARN.
>
> Thanks to the whole team for the great job!
>
> Best,
> Wangda
>
>
> On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 (non-binding)
> >
> > On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
> >
> > >+1 (non-binding)
> > >
> > >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> I would like to call a vote for merging yarn-native-services to trunk.
> > >>The
> > >> vote will run for 7 days as usual.
> > >>
> > >> At a high level, the following are the key feautres implemented.
> > >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> > >>and
> > >> orchestrate existing services to YARN either docker or non-docker
> based.
> > >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> > >> simple JSON spec
> > >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> > >> service to enable users to discover services deployed on YARN
> > >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> > >> All these new services are optional and are sitting outside of the
> > >> existing system, and have no impact on existing system if disabled.
> > >>
> > >> Special thanks to a team of folks who worked hard towards this: Billie
> > >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> > >>Sharma
> > >> K S, Sunil G, Akhil PB. This effort could not be possible without
> their
> > >> ideas and hard work.
> > >>
> > >> Thanks,
> > >> Jian
> > >>
> > >> [1] https://issues.apache.org/jira/browse/YARN-5079
> > >> [2] https://issues.apache.org/jira/browse/YARN-4793
> > >> [3] https://issues.apache.org/jira/browse/YARN-4757
> > >> [4] https://issues.apache.org/jira/browse/YARN-6419
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >
> >
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), I tried to use YARN service assembly before to run different
kinds of jobs (for example, distributed Tensorflow), it is really easy for
end user to run jobs on YARN.
Thanks to the whole team for the great job!
Best,
Wangda
On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
> +1 (non-binding)
>
> On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>
> >+1 (non-binding)
> >
> >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> >
> >> Hi All,
> >>
> >> I would like to call a vote for merging yarn-native-services to trunk.
> >>The
> >> vote will run for 7 days as usual.
> >>
> >> At a high level, the following are the key feautres implemented.
> >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> >>and
> >> orchestrate existing services to YARN either docker or non-docker based.
> >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> >> simple JSON spec
> >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> >> service to enable users to discover services deployed on YARN
> >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> >> All these new services are optional and are sitting outside of the
> >> existing system, and have no impact on existing system if disabled.
> >>
> >> Special thanks to a team of folks who worked hard towards this: Billie
> >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> >>Sharma
> >> K S, Sunil G, Akhil PB. This effort could not be possible without their
> >> ideas and hard work.
> >>
> >> Thanks,
> >> Jian
> >>
> >> [1] https://issues.apache.org/jira/browse/YARN-5079
> >> [2] https://issues.apache.org/jira/browse/YARN-4793
> >> [3] https://issues.apache.org/jira/browse/YARN-4757
> >> [4] https://issues.apache.org/jira/browse/YARN-6419
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), I tried to use YARN service assembly before to run different
kinds of jobs (for example, distributed Tensorflow), it is really easy for
end user to run jobs on YARN.
Thanks to the whole team for the great job!
Best,
Wangda
On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
> +1 (non-binding)
>
> On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>
> >+1 (non-binding)
> >
> >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> >
> >> Hi All,
> >>
> >> I would like to call a vote for merging yarn-native-services to trunk.
> >>The
> >> vote will run for 7 days as usual.
> >>
> >> At a high level, the following are the key feautres implemented.
> >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> >>and
> >> orchestrate existing services to YARN either docker or non-docker based.
> >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> >> simple JSON spec
> >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> >> service to enable users to discover services deployed on YARN
> >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> >> All these new services are optional and are sitting outside of the
> >> existing system, and have no impact on existing system if disabled.
> >>
> >> Special thanks to a team of folks who worked hard towards this: Billie
> >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> >>Sharma
> >> K S, Sunil G, Akhil PB. This effort could not be possible without their
> >> ideas and hard work.
> >>
> >> Thanks,
> >> Jian
> >>
> >> [1] https://issues.apache.org/jira/browse/YARN-5079
> >> [2] https://issues.apache.org/jira/browse/YARN-4793
> >> [3] https://issues.apache.org/jira/browse/YARN-4757
> >> [4] https://issues.apache.org/jira/browse/YARN-6419
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), I tried to use YARN service assembly before to run different
kinds of jobs (for example, distributed Tensorflow), it is really easy for
end user to run jobs on YARN.
Thanks to the whole team for the great job!
Best,
Wangda
On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
> +1 (non-binding)
>
> On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>
> >+1 (non-binding)
> >
> >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> >
> >> Hi All,
> >>
> >> I would like to call a vote for merging yarn-native-services to trunk.
> >>The
> >> vote will run for 7 days as usual.
> >>
> >> At a high level, the following are the key feautres implemented.
> >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> >>and
> >> orchestrate existing services to YARN either docker or non-docker based.
> >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> >> simple JSON spec
> >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> >> service to enable users to discover services deployed on YARN
> >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> >> All these new services are optional and are sitting outside of the
> >> existing system, and have no impact on existing system if disabled.
> >>
> >> Special thanks to a team of folks who worked hard towards this: Billie
> >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> >>Sharma
> >> K S, Sunil G, Akhil PB. This effort could not be possible without their
> >> ideas and hard work.
> >>
> >> Thanks,
> >> Jian
> >>
> >> [1] https://issues.apache.org/jira/browse/YARN-5079
> >> [2] https://issues.apache.org/jira/browse/YARN-4793
> >> [3] https://issues.apache.org/jira/browse/YARN-4757
> >> [4] https://issues.apache.org/jira/browse/YARN-6419
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), I tried to use YARN service assembly before to run different
kinds of jobs (for example, distributed Tensorflow), it is really easy for
end user to run jobs on YARN.
Thanks to the whole team for the great job!
Best,
Wangda
On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha <gs...@hortonworks.com> wrote:
> +1 (non-binding)
>
> On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>
> >+1 (non-binding)
> >
> >On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> >
> >> Hi All,
> >>
> >> I would like to call a vote for merging yarn-native-services to trunk.
> >>The
> >> vote will run for 7 days as usual.
> >>
> >> At a high level, the following are the key feautres implemented.
> >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
> >>and
> >> orchestrate existing services to YARN either docker or non-docker based.
> >> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> >> simple JSON spec
> >> - YARN-4757[3]. Extending today's service registry with a simple DNS
> >> service to enable users to discover services deployed on YARN
> >> - YARN-6419[4]. UI support for native-services on the new YARN UI
> >> All these new services are optional and are sitting outside of the
> >> existing system, and have no impact on existing system if disabled.
> >>
> >> Special thanks to a team of folks who worked hard towards this: Billie
> >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> >>Sharma
> >> K S, Sunil G, Akhil PB. This effort could not be possible without their
> >> ideas and hard work.
> >>
> >> Thanks,
> >> Jian
> >>
> >> [1] https://issues.apache.org/jira/browse/YARN-5079
> >> [2] https://issues.apache.org/jira/browse/YARN-4793
> >> [3] https://issues.apache.org/jira/browse/YARN-4757
> >> [4] https://issues.apache.org/jira/browse/YARN-6419
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
+1 (non-binding)
On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>+1 (non-binding)
>
>On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Hi All,
>>
>> I would like to call a vote for merging yarn-native-services to trunk.
>>The
>> vote will run for 7 days as usual.
>>
>> At a high level, the following are the key feautres implemented.
>> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
>>and
>> orchestrate existing services to YARN either docker or non-docker based.
>> - YARN-4793[2]. A Rest API server for user to deploy a service via a
>> simple JSON spec
>> - YARN-4757[3]. Extending today's service registry with a simple DNS
>> service to enable users to discover services deployed on YARN
>> - YARN-6419[4]. UI support for native-services on the new YARN UI
>> All these new services are optional and are sitting outside of the
>> existing system, and have no impact on existing system if disabled.
>>
>> Special thanks to a team of folks who worked hard towards this: Billie
>> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
>>Sharma
>> K S, Sunil G, Akhil PB. This effort could not be possible without their
>> ideas and hard work.
>>
>> Thanks,
>> Jian
>>
>> [1] https://issues.apache.org/jira/browse/YARN-5079
>> [2] https://issues.apache.org/jira/browse/YARN-4793
>> [3] https://issues.apache.org/jira/browse/YARN-4757
>> [4] https://issues.apache.org/jira/browse/YARN-6419
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
+1 (non-binding)
On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>+1 (non-binding)
>
>On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Hi All,
>>
>> I would like to call a vote for merging yarn-native-services to trunk.
>>The
>> vote will run for 7 days as usual.
>>
>> At a high level, the following are the key feautres implemented.
>> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
>>and
>> orchestrate existing services to YARN either docker or non-docker based.
>> - YARN-4793[2]. A Rest API server for user to deploy a service via a
>> simple JSON spec
>> - YARN-4757[3]. Extending today's service registry with a simple DNS
>> service to enable users to discover services deployed on YARN
>> - YARN-6419[4]. UI support for native-services on the new YARN UI
>> All these new services are optional and are sitting outside of the
>> existing system, and have no impact on existing system if disabled.
>>
>> Special thanks to a team of folks who worked hard towards this: Billie
>> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
>>Sharma
>> K S, Sunil G, Akhil PB. This effort could not be possible without their
>> ideas and hard work.
>>
>> Thanks,
>> Jian
>>
>> [1] https://issues.apache.org/jira/browse/YARN-5079
>> [2] https://issues.apache.org/jira/browse/YARN-4793
>> [3] https://issues.apache.org/jira/browse/YARN-4757
>> [4] https://issues.apache.org/jira/browse/YARN-6419
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
+1 (non-binding)
On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>+1 (non-binding)
>
>On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Hi All,
>>
>> I would like to call a vote for merging yarn-native-services to trunk.
>>The
>> vote will run for 7 days as usual.
>>
>> At a high level, the following are the key feautres implemented.
>> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
>>and
>> orchestrate existing services to YARN either docker or non-docker based.
>> - YARN-4793[2]. A Rest API server for user to deploy a service via a
>> simple JSON spec
>> - YARN-4757[3]. Extending today's service registry with a simple DNS
>> service to enable users to discover services deployed on YARN
>> - YARN-6419[4]. UI support for native-services on the new YARN UI
>> All these new services are optional and are sitting outside of the
>> existing system, and have no impact on existing system if disabled.
>>
>> Special thanks to a team of folks who worked hard towards this: Billie
>> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
>>Sharma
>> K S, Sunil G, Akhil PB. This effort could not be possible without their
>> ideas and hard work.
>>
>> Thanks,
>> Jian
>>
>> [1] https://issues.apache.org/jira/browse/YARN-5079
>> [2] https://issues.apache.org/jira/browse/YARN-4793
>> [3] https://issues.apache.org/jira/browse/YARN-4757
>> [4] https://issues.apache.org/jira/browse/YARN-6419
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Gour Saha <gs...@hortonworks.com>.
+1 (non-binding)
On 9/1/17, 11:58 AM, "Billie Rinaldi" <bi...@gmail.com> wrote:
>+1 (non-binding)
>
>On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
>
>> Hi All,
>>
>> I would like to call a vote for merging yarn-native-services to trunk.
>>The
>> vote will run for 7 days as usual.
>>
>> At a high level, the following are the key feautres implemented.
>> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
>>and
>> orchestrate existing services to YARN either docker or non-docker based.
>> - YARN-4793[2]. A Rest API server for user to deploy a service via a
>> simple JSON spec
>> - YARN-4757[3]. Extending today's service registry with a simple DNS
>> service to enable users to discover services deployed on YARN
>> - YARN-6419[4]. UI support for native-services on the new YARN UI
>> All these new services are optional and are sitting outside of the
>> existing system, and have no impact on existing system if disabled.
>>
>> Special thanks to a team of folks who worked hard towards this: Billie
>> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
>>Sharma
>> K S, Sunil G, Akhil PB. This effort could not be possible without their
>> ideas and hard work.
>>
>> Thanks,
>> Jian
>>
>> [1] https://issues.apache.org/jira/browse/YARN-5079
>> [2] https://issues.apache.org/jira/browse/YARN-4793
>> [3] https://issues.apache.org/jira/browse/YARN-4757
>> [4] https://issues.apache.org/jira/browse/YARN-6419
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Billie Rinaldi <bi...@gmail.com>.
+1 (non-binding)
On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi All,
>
> I would like to call a vote for merging yarn-native-services to trunk. The
> vote will run for 7 days as usual.
>
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Billie Rinaldi <bi...@gmail.com>.
+1 (non-binding)
On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi All,
>
> I would like to call a vote for merging yarn-native-services to trunk. The
> vote will run for 7 days as usual.
>
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 31, 2017, at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> I would like to call a vote for merging yarn-native-services to trunk.
1) Did I miss it or is there no actual end-user documentation on how to use this? I see hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md, but that’s not particularly useful. It looks like there are daemons that need to get started, based on other documentation? How? What do I configure? Is there a command to use to say “go do native for this job”? I honestly have no idea how to make this do anything because most of the docs appear to be either TBD or expect me to read through a ton of JIRAs.
2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever.
3) The default port for the DNS server should NOT be 53 if typical deployments need to specify an alternate port. Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc. In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
At this point in time:
-1 on 3.0.0-beta1
-0 on trunk
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 31, 2017, at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> I would like to call a vote for merging yarn-native-services to trunk.
1) Did I miss it or is there no actual end-user documentation on how to use this? I see hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md, but that’s not particularly useful. It looks like there are daemons that need to get started, based on other documentation? How? What do I configure? Is there a command to use to say “go do native for this job”? I honestly have no idea how to make this do anything because most of the docs appear to be either TBD or expect me to read through a ton of JIRAs.
2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever.
3) The default port for the DNS server should NOT be 53 if typical deployments need to specify an alternate port. Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc. In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
At this point in time:
-1 on 3.0.0-beta1
-0 on trunk
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 31, 2017, at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> I would like to call a vote for merging yarn-native-services to trunk.
1) Did I miss it or is there no actual end-user documentation on how to use this? I see hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md, but that’s not particularly useful. It looks like there are daemons that need to get started, based on other documentation? How? What do I configure? Is there a command to use to say “go do native for this job”? I honestly have no idea how to make this do anything because most of the docs appear to be either TBD or expect me to read through a ton of JIRAs.
2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever.
3) The default port for the DNS server should NOT be 53 if typical deployments need to specify an alternate port. Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc. In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”?
4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands.
At this point in time:
-1 on 3.0.0-beta1
-0 on trunk
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Billie Rinaldi <bi...@gmail.com>.
+1 (non-binding)
On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi All,
>
> I would like to call a vote for merging yarn-native-services to trunk. The
> vote will run for 7 days as usual.
>
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge yarn-native-services branch into trunk
Posted by Billie Rinaldi <bi...@gmail.com>.
+1 (non-binding)
On Thu, Aug 31, 2017 at 8:33 PM, Jian He <jh...@hortonworks.com> wrote:
> Hi All,
>
> I would like to call a vote for merging yarn-native-services to trunk. The
> vote will run for 7 days as usual.
>
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate and
> orchestrate existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API server for user to deploy a service via a
> simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB. This effort could not be possible without their
> ideas and hard work.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>