[DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Hi, Hadoop YARN community!

(Cross-posting across Hadoop and Slider communities)

I opened a JIRA a little while ago to pursue a native YARN framework for services: https://issues.apache.org/jira/browse/YARN-5079.

It is part of a bigger effort that a bunch of us YARN community members are interested in making progress on: YARN-4692 - [Umbrella] Simplified and first-class support for services in YARN.

The idea is that with our current attention on making services first­-class, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. I’ve been looking at various possibilities - ranging from a custom new framework room scratch to using one of the existing projects - and stopped at Apache Slider (http://slider.incubator.apache.org) given its association with some of the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy, myself etc.).

Slider client & AM already handles a great deal of the functionality that we need. I posit that assimilating the client, ApplicationMaster etc of an existing framework like Apache Slider can serve our purpose really well. My early informal discussions about this with few Hadoop and Slider community members yielded generally favourable feedback.

The Apache Slider incubator community also discussed this and expressed generally positive interest in YARN taking up Slider’s key pieces, you can see that discussion here: https://s.apache.org/0hoh.

So in summary, we are looking to the following

- Code
   — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into Apache Hadoop for providing a native services experiences in YARN
   — Leave for now some of the pieces behind in Apache Slider - (a) Slider agent as we won’t need it, (b) Slider packages that need more deliberation in terms of where they will live in the long term.
   — Create a branch in YARN, copy this code over into a new module(s), and work towards completing a functioning app running on YARN.

- Communities & releases
   — Good thing is that many of Apache Slider community members are already seasoned folk in the Apache Hadoop ecosystem projects. For those committers & PMC in Slider that are not yet Hadoop committers / PMC, without complicating things much, a proposed path forward is active participation in the branch (as branch committers?) and eventually in mainline YARN and thus go through a natural progression to committership / PMC. Given that most of the members are stalwarts in the Apache communities, this should be a cinch IMO.
   — The work on this new code can start, and depending on its state, and assuming that the experiment succeeds, can be merged into trunk and later picked up in the next nearest & feasible Apache Hadoop release.
   — While the work on forked-over-code goes on till we have a functioning app, the current Apache Slider project continues to live, with supporting releases etc. At some point in the future, when things become clearer, more decisions can be taken on the (parts left behind in the) incubating project’s future.

Thoughts?

Thanks
+Vinod
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

If you see comments from Gour (of Apache Slider), that seems like the general direction: "All that would be expected is to migrate the app-packages to the agent-less version. Of course the Slider community will do the migration for the current app-packages in the Slider codebase.”

Whether this means taking an existing package and run it as is on the newer framework remains to be seen. One possible solution is to just support today’s Slider framework on newer YARN. That way if you have an existing app-package, it continues to run on the newer cluster, but for getting some of the newer functionality, you may have to move to the newer package.

My personal take is that it will take a bit of effort to make all of this happen, but is a very important goal to support existing users and their applications.

Thanks
+Vinod

> On Jul 19, 2016, at 3:29 PM, Siddharth Seth <ss...@apache.org> wrote:
> 
> In terms of compatibility and Slider fixes - is the intent to allow
> existing slider packages to work without migrating over to the new
> agent-less, etc implementation in Hadoop. e.g. Assuming the slider
> integration work is released in Hadoop 2.xx - would an existing slider
> package which works on Hadoop-2.6 + Slider 0.91.0 continue to work without
> changes?

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

If you see comments from Gour (of Apache Slider), that seems like the general direction: "All that would be expected is to migrate the app-packages to the agent-less version. Of course the Slider community will do the migration for the current app-packages in the Slider codebase.”

Whether this means taking an existing package and run it as is on the newer framework remains to be seen. One possible solution is to just support today’s Slider framework on newer YARN. That way if you have an existing app-package, it continues to run on the newer cluster, but for getting some of the newer functionality, you may have to move to the newer package.

My personal take is that it will take a bit of effort to make all of this happen, but is a very important goal to support existing users and their applications.

Thanks
+Vinod

> On Jul 19, 2016, at 3:29 PM, Siddharth Seth <ss...@apache.org> wrote:
> 
> In terms of compatibility and Slider fixes - is the intent to allow
> existing slider packages to work without migrating over to the new
> agent-less, etc implementation in Hadoop. e.g. Assuming the slider
> integration work is released in Hadoop 2.xx - would an existing slider
> package which works on Hadoop-2.6 + Slider 0.91.0 continue to work without
> changes?

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Siddharth Seth <ss...@apache.org>]

+1 as a step towards better service support in YARN.

In terms of compatibility and Slider fixes - is the intent to allow
existing slider packages to work without migrating over to the new
agent-less, etc implementation in Hadoop. e.g. Assuming the slider
integration work is released in Hadoop 2.xx - would an existing slider
package which works on Hadoop-2.6 + Slider 0.91.0 continue to work without
changes?

On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gs...@hortonworks.com> wrote:

> +1 forwarding from the discussion in Slider DL
>
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.
>
> -Gour
>
> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
> >Hi, Hadoop YARN community!
> >
> >(Cross-posting across Hadoop and Slider communities)
> >
> >I opened a JIRA a little while ago to pursue a native YARN framework for
> >services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> >It is part of a bigger effort that a bunch of us YARN community members
> >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> >and first-class support for services in YARN.
> >
> >The idea is that with our current attention on making services
> >first­-class, it's time to take a fresh look at how we can make Apache
> >Hadoop YARN support services well out of the box. I’ve been looking at
> >various possibilities - ranging from a custom new framework room scratch
> >to using one of the existing projects - and stopped at Apache Slider
> >(http://slider.incubator.apache.org) given its association with some of
> >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> >myself etc.).
> >
> >Slider client & AM already handles a great deal of the functionality that
> >we need. I posit that assimilating the client, ApplicationMaster etc of
> >an existing framework like Apache Slider can serve our purpose really
> >well. My early informal discussions about this with few Hadoop and Slider
> >community members yielded generally favourable feedback.
> >
> >The Apache Slider incubator community also discussed this and expressed
> >generally positive interest in YARN taking up Slider’s key pieces, you
> >can see that discussion here: https://s.apache.org/0hoh.
> >
> >So in summary, we are looking to the following
> >
> >- Code
> >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> >Apache Hadoop for providing a native services experiences in YARN
> >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> >Slider agent as we won’t need it, (b) Slider packages that need more
> >deliberation in terms of where they will live in the long term.
> >   ― Create a branch in YARN, copy this code over into a new module(s),
> >and work towards completing a functioning app running on YARN.
> >
> >- Communities & releases
> >   ― Good thing is that many of Apache Slider community members are
> >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> >without complicating things much, a proposed path forward is active
> >participation in the branch (as branch committers?) and eventually in
> >mainline YARN and thus go through a natural progression to committership
> >/ PMC. Given that most of the members are stalwarts in the Apache
> >communities, this should be a cinch IMO.
> >   ― The work on this new code can start, and depending on its state, and
> >assuming that the experiment succeeds, can be merged into trunk and later
> >picked up in the next nearest & feasible Apache Hadoop release.
> >   ― While the work on forked-over-code goes on till we have a
> >functioning app, the current Apache Slider project continues to live,
> >with supporting releases etc. At some point in the future, when things
> >become clearer, more decisions can be taken on the (parts left behind in
> >the) incubating project’s future.
> >
> >Thoughts?
> >
> >Thanks
> >+Vinod
>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Varun Vasudev <vv...@apache.org>]

+1. This should go a long way to help running services on YARN.

-Varun

On 7/19/16, 12:53 AM, "Gour Saha" <gs...@hortonworks.com> wrote:

>+1 forwarding from the discussion in Slider DL
>
>Note: On the Slider side, we plan to create a branch corresponding to this
>YARN branch. In this, we intend to remove all the pieces which will move
>to the YARN branch (Slider core, AM, client). We will retain the Agent and
>app-packages which will then depend on the new hadoop-slider module (in
>addition to the existing hadoop modules that it already depends on). This
>will create a single view of Slider codebase, exactly as it stands today,
>fully supporting the current app-packages. Slider can even release its
>future versions from this new branch, once the hadoop-sldier module
>reaches a matured state. It will be an easier path for existing Slider
>users/app-owners to move to the future stable state of Slider completely
>off of Hadoop YARN codebase. All that would be expected is to migrate the
>app-packages to the agent-less version. Of course the Slider community
>will do the migration for the current app-packages in the Slider codebase.
>
>-Gour
>
>On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
>>Hi, Hadoop YARN community!
>>
>>(Cross-posting across Hadoop and Slider communities)
>>
>>I opened a JIRA a little while ago to pursue a native YARN framework for
>>services: https://issues.apache.org/jira/browse/YARN-5079.
>>
>>It is part of a bigger effort that a bunch of us YARN community members
>>are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>and first-class support for services in YARN.
>>
>>The idea is that with our current attention on making services
>>first–-class, it's time to take a fresh look at how we can make Apache
>>Hadoop YARN support services well out of the box. I’ve been looking at
>>various possibilities - ranging from a custom new framework room scratch
>>to using one of the existing projects - and stopped at Apache Slider
>>(http://slider.incubator.apache.org) given its association with some of
>>the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>myself etc.).
>>
>>Slider client & AM already handles a great deal of the functionality that
>>we need. I posit that assimilating the client, ApplicationMaster etc of
>>an existing framework like Apache Slider can serve our purpose really
>>well. My early informal discussions about this with few Hadoop and Slider
>>community members yielded generally favourable feedback.
>>
>>The Apache Slider incubator community also discussed this and expressed
>>generally positive interest in YARN taking up Slider’s key pieces, you
>>can see that discussion here: https://s.apache.org/0hoh.
>>
>>So in summary, we are looking to the following
>>
>>- Code
>>   — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>Apache Hadoop for providing a native services experiences in YARN
>>   — Leave for now some of the pieces behind in Apache Slider - (a)
>>Slider agent as we won’t need it, (b) Slider packages that need more
>>deliberation in terms of where they will live in the long term.
>>   — Create a branch in YARN, copy this code over into a new module(s),
>>and work towards completing a functioning app running on YARN.
>>
>>- Communities & releases
>>   — Good thing is that many of Apache Slider community members are
>>already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>without complicating things much, a proposed path forward is active
>>participation in the branch (as branch committers?) and eventually in
>>mainline YARN and thus go through a natural progression to committership
>>/ PMC. Given that most of the members are stalwarts in the Apache
>>communities, this should be a cinch IMO.
>>   — The work on this new code can start, and depending on its state, and
>>assuming that the experiment succeeds, can be merged into trunk and later
>>picked up in the next nearest & feasible Apache Hadoop release.
>>   — While the work on forked-over-code goes on till we have a
>>functioning app, the current Apache Slider project continues to live,
>>with supporting releases etc. At some point in the future, when things
>>become clearer, more decisions can be taken on the (parts left behind in
>>the) incubating project’s future.
>>
>>Thoughts?
>>
>>Thanks
>>+Vinod
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

An interesting proposal, we can continue this discussion on the Slider dev lists.

Thanks
+Vinod

> On Jul 19, 2016, at 4:51 PM, jonmaron@gmail.com wrote:
> 
> +1 - this approach definitely addresses an important need.
> 
> The project has gone through at least a couple of development/repo
> approaches:
> 
> 1)  git-flow with master/develop branches
> 2)  abandonment of the master branch and work/releases based on the develop
> branch.
> 
> I have no strong objections to the approach above, but given the existence
> of the two branches above, and some developer familiarity with the
> 'develop' branch, perhaps we could:
> 
> 1)  Synch the master branch to the develop branch (not sure which git
> mechanism would best achieve this)
> 2)  Leverage the develop branch to create the agent/app-package based
> Slider distribution
> 
> In this way there'a a legacy branch available while develop continues to
> proceed on the existing and familiar development branch.
> 
> Just a thought.  Like I said - no real strong feelings :)
> 
> On Mon, Jul 18, 2016 at 3:23 PM, Gour Saha <gs...@hortonworks.com> wrote:
> 
>> +1 forwarding from the discussion in Slider DL
>> 
>> Note: On the Slider side, we plan to create a branch corresponding to this
>> YARN branch. In this, we intend to remove all the pieces which will move
>> to the YARN branch (Slider core, AM, client). We will retain the Agent and
>> app-packages which will then depend on the new hadoop-slider module (in
>> addition to the existing hadoop modules that it already depends on). This
>> will create a single view of Slider codebase, exactly as it stands today,
>> fully supporting the current app-packages. Slider can even release its
>> future versions from this new branch, once the hadoop-sldier module
>> reaches a matured state. It will be an easier path for existing Slider
>> users/app-owners to move to the future stable state of Slider completely
>> off of Hadoop YARN codebase. All that would be expected is to migrate the
>> app-packages to the agent-less version. Of course the Slider community
>> will do the migration for the current app-packages in the Slider codebase.
>> 
>> -Gour
>> 
>> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>> 
>>> Hi, Hadoop YARN community!
>>> 
>>> (Cross-posting across Hadoop and Slider communities)
>>> 
>>> I opened a JIRA a little while ago to pursue a native YARN framework for
>>> services: https://issues.apache.org/jira/browse/YARN-5079.
>>> 
>>> It is part of a bigger effort that a bunch of us YARN community members
>>> are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>> and first-class support for services in YARN.
>>> 
>>> The idea is that with our current attention on making services
>>> first­-class, it's time to take a fresh look at how we can make Apache
>>> Hadoop YARN support services well out of the box. I’ve been looking at
>>> various possibilities - ranging from a custom new framework room scratch
>>> to using one of the existing projects - and stopped at Apache Slider
>>> (http://slider.incubator.apache.org) given its association with some of
>>> the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>> myself etc.).
>>> 
>>> Slider client & AM already handles a great deal of the functionality that
>>> we need. I posit that assimilating the client, ApplicationMaster etc of
>>> an existing framework like Apache Slider can serve our purpose really
>>> well. My early informal discussions about this with few Hadoop and Slider
>>> community members yielded generally favourable feedback.
>>> 
>>> The Apache Slider incubator community also discussed this and expressed
>>> generally positive interest in YARN taking up Slider’s key pieces, you
>>> can see that discussion here: https://s.apache.org/0hoh.
>>> 
>>> So in summary, we are looking to the following
>>> 
>>> - Code
>>>  ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>> Apache Hadoop for providing a native services experiences in YARN
>>>  ― Leave for now some of the pieces behind in Apache Slider - (a)
>>> Slider agent as we won’t need it, (b) Slider packages that need more
>>> deliberation in terms of where they will live in the long term.
>>>  ― Create a branch in YARN, copy this code over into a new module(s),
>>> and work towards completing a functioning app running on YARN.
>>> 
>>> - Communities & releases
>>>  ― Good thing is that many of Apache Slider community members are
>>> already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>> committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>> without complicating things much, a proposed path forward is active
>>> participation in the branch (as branch committers?) and eventually in
>>> mainline YARN and thus go through a natural progression to committership
>>> / PMC. Given that most of the members are stalwarts in the Apache
>>> communities, this should be a cinch IMO.
>>>  ― The work on this new code can start, and depending on its state, and
>>> assuming that the experiment succeeds, can be merged into trunk and later
>>> picked up in the next nearest & feasible Apache Hadoop release.
>>>  ― While the work on forked-over-code goes on till we have a
>>> functioning app, the current Apache Slider project continues to live,
>>> with supporting releases etc. At some point in the future, when things
>>> become clearer, more decisions can be taken on the (parts left behind in
>>> the) incubating project’s future.
>>> 
>>> Thoughts?
>>> 
>>> Thanks
>>> +Vinod
>> 
>> 
> 
> 
> -- 
> Sent from some random computer....


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Subramaniam V K <su...@gmail.com>]

+1 on the proposal as:

1) We definitely need first class support for long running services on YARN.
2) We need to surface the support we are building in an easy to consume
format to app developers as YARN APIs IMHO is too low-level for direct
end-developer consumption.
3) Slider is the obvious choice for (2) (echoing Vinod/Gour).
3) At the same time we need to make sure that other *frameworks *like
Apache REEF can also build directly on the native services support in YARN
as there might be custom scenarios (in our case ML services) that cannot
all be satisfied by Slider alone.

I would like to think of Slider integration as a production grade reference
implementation of services on YARN akin to Nexus/Surface lines of devices
for Android/Windows :).


On Tue, Jul 19, 2016 at 4:51 PM, jonmaron@gmail.com <jo...@gmail.com>
wrote:

> +1 - this approach definitely addresses an important need.
>
> The project has gone through at least a couple of development/repo
> approaches:
>
> 1)  git-flow with master/develop branches
> 2)  abandonment of the master branch and work/releases based on the develop
> branch.
>
> I have no strong objections to the approach above, but given the existence
> of the two branches above, and some developer familiarity with the
> 'develop' branch, perhaps we could:
>
> 1)  Synch the master branch to the develop branch (not sure which git
> mechanism would best achieve this)
> 2)  Leverage the develop branch to create the agent/app-package based
> Slider distribution
>
> In this way there'a a legacy branch available while develop continues to
> proceed on the existing and familiar development branch.
>
> Just a thought.  Like I said - no real strong feelings :)
>
> On Mon, Jul 18, 2016 at 3:23 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 forwarding from the discussion in Slider DL
> >
> > Note: On the Slider side, we plan to create a branch corresponding to
> this
> > YARN branch. In this, we intend to remove all the pieces which will move
> > to the YARN branch (Slider core, AM, client). We will retain the Agent
> and
> > app-packages which will then depend on the new hadoop-slider module (in
> > addition to the existing hadoop modules that it already depends on). This
> > will create a single view of Slider codebase, exactly as it stands today,
> > fully supporting the current app-packages. Slider can even release its
> > future versions from this new branch, once the hadoop-sldier module
> > reaches a matured state. It will be an easier path for existing Slider
> > users/app-owners to move to the future stable state of Slider completely
> > off of Hadoop YARN codebase. All that would be expected is to migrate the
> > app-packages to the agent-less version. Of course the Slider community
> > will do the migration for the current app-packages in the Slider
> codebase.
> >
> > -Gour
> >
> > On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org>
> wrote:
> >
> > >Hi, Hadoop YARN community!
> > >
> > >(Cross-posting across Hadoop and Slider communities)
> > >
> > >I opened a JIRA a little while ago to pursue a native YARN framework for
> > >services: https://issues.apache.org/jira/browse/YARN-5079.
> > >
> > >It is part of a bigger effort that a bunch of us YARN community members
> > >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> > >and first-class support for services in YARN.
> > >
> > >The idea is that with our current attention on making services
> > >first­-class, it's time to take a fresh look at how we can make Apache
> > >Hadoop YARN support services well out of the box. I’ve been looking at
> > >various possibilities - ranging from a custom new framework room scratch
> > >to using one of the existing projects - and stopped at Apache Slider
> > >(http://slider.incubator.apache.org) given its association with some of
> > >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > >myself etc.).
> > >
> > >Slider client & AM already handles a great deal of the functionality
> that
> > >we need. I posit that assimilating the client, ApplicationMaster etc of
> > >an existing framework like Apache Slider can serve our purpose really
> > >well. My early informal discussions about this with few Hadoop and
> Slider
> > >community members yielded generally favourable feedback.
> > >
> > >The Apache Slider incubator community also discussed this and expressed
> > >generally positive interest in YARN taking up Slider’s key pieces, you
> > >can see that discussion here: https://s.apache.org/0hoh.
> > >
> > >So in summary, we are looking to the following
> > >
> > >- Code
> > >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > >Apache Hadoop for providing a native services experiences in YARN
> > >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> > >Slider agent as we won’t need it, (b) Slider packages that need more
> > >deliberation in terms of where they will live in the long term.
> > >   ― Create a branch in YARN, copy this code over into a new module(s),
> > >and work towards completing a functioning app running on YARN.
> > >
> > >- Communities & releases
> > >   ― Good thing is that many of Apache Slider community members are
> > >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > >without complicating things much, a proposed path forward is active
> > >participation in the branch (as branch committers?) and eventually in
> > >mainline YARN and thus go through a natural progression to committership
> > >/ PMC. Given that most of the members are stalwarts in the Apache
> > >communities, this should be a cinch IMO.
> > >   ― The work on this new code can start, and depending on its state,
> and
> > >assuming that the experiment succeeds, can be merged into trunk and
> later
> > >picked up in the next nearest & feasible Apache Hadoop release.
> > >   ― While the work on forked-over-code goes on till we have a
> > >functioning app, the current Apache Slider project continues to live,
> > >with supporting releases etc. At some point in the future, when things
> > >become clearer, more decisions can be taken on the (parts left behind in
> > >the) incubating project’s future.
> > >
> > >Thoughts?
> > >
> > >Thanks
> > >+Vinod
> >
> >
>
>
> --
> Sent from some random computer....
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Subramaniam V K <su...@gmail.com>]

+1 on the proposal as:

1) We definitely need first class support for long running services on YARN.
2) We need to surface the support we are building in an easy to consume
format to app developers as YARN APIs IMHO is too low-level for direct
end-developer consumption.
3) Slider is the obvious choice for (2) (echoing Vinod/Gour).
3) At the same time we need to make sure that other *frameworks *like
Apache REEF can also build directly on the native services support in YARN
as there might be custom scenarios (in our case ML services) that cannot
all be satisfied by Slider alone.

I would like to think of Slider integration as a production grade reference
implementation of services on YARN akin to Nexus/Surface lines of devices
for Android/Windows :).


On Tue, Jul 19, 2016 at 4:51 PM, jonmaron@gmail.com <jo...@gmail.com>
wrote:

> +1 - this approach definitely addresses an important need.
>
> The project has gone through at least a couple of development/repo
> approaches:
>
> 1)  git-flow with master/develop branches
> 2)  abandonment of the master branch and work/releases based on the develop
> branch.
>
> I have no strong objections to the approach above, but given the existence
> of the two branches above, and some developer familiarity with the
> 'develop' branch, perhaps we could:
>
> 1)  Synch the master branch to the develop branch (not sure which git
> mechanism would best achieve this)
> 2)  Leverage the develop branch to create the agent/app-package based
> Slider distribution
>
> In this way there'a a legacy branch available while develop continues to
> proceed on the existing and familiar development branch.
>
> Just a thought.  Like I said - no real strong feelings :)
>
> On Mon, Jul 18, 2016 at 3:23 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 forwarding from the discussion in Slider DL
> >
> > Note: On the Slider side, we plan to create a branch corresponding to
> this
> > YARN branch. In this, we intend to remove all the pieces which will move
> > to the YARN branch (Slider core, AM, client). We will retain the Agent
> and
> > app-packages which will then depend on the new hadoop-slider module (in
> > addition to the existing hadoop modules that it already depends on). This
> > will create a single view of Slider codebase, exactly as it stands today,
> > fully supporting the current app-packages. Slider can even release its
> > future versions from this new branch, once the hadoop-sldier module
> > reaches a matured state. It will be an easier path for existing Slider
> > users/app-owners to move to the future stable state of Slider completely
> > off of Hadoop YARN codebase. All that would be expected is to migrate the
> > app-packages to the agent-less version. Of course the Slider community
> > will do the migration for the current app-packages in the Slider
> codebase.
> >
> > -Gour
> >
> > On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org>
> wrote:
> >
> > >Hi, Hadoop YARN community!
> > >
> > >(Cross-posting across Hadoop and Slider communities)
> > >
> > >I opened a JIRA a little while ago to pursue a native YARN framework for
> > >services: https://issues.apache.org/jira/browse/YARN-5079.
> > >
> > >It is part of a bigger effort that a bunch of us YARN community members
> > >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> > >and first-class support for services in YARN.
> > >
> > >The idea is that with our current attention on making services
> > >first­-class, it's time to take a fresh look at how we can make Apache
> > >Hadoop YARN support services well out of the box. I’ve been looking at
> > >various possibilities - ranging from a custom new framework room scratch
> > >to using one of the existing projects - and stopped at Apache Slider
> > >(http://slider.incubator.apache.org) given its association with some of
> > >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > >myself etc.).
> > >
> > >Slider client & AM already handles a great deal of the functionality
> that
> > >we need. I posit that assimilating the client, ApplicationMaster etc of
> > >an existing framework like Apache Slider can serve our purpose really
> > >well. My early informal discussions about this with few Hadoop and
> Slider
> > >community members yielded generally favourable feedback.
> > >
> > >The Apache Slider incubator community also discussed this and expressed
> > >generally positive interest in YARN taking up Slider’s key pieces, you
> > >can see that discussion here: https://s.apache.org/0hoh.
> > >
> > >So in summary, we are looking to the following
> > >
> > >- Code
> > >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > >Apache Hadoop for providing a native services experiences in YARN
> > >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> > >Slider agent as we won’t need it, (b) Slider packages that need more
> > >deliberation in terms of where they will live in the long term.
> > >   ― Create a branch in YARN, copy this code over into a new module(s),
> > >and work towards completing a functioning app running on YARN.
> > >
> > >- Communities & releases
> > >   ― Good thing is that many of Apache Slider community members are
> > >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > >without complicating things much, a proposed path forward is active
> > >participation in the branch (as branch committers?) and eventually in
> > >mainline YARN and thus go through a natural progression to committership
> > >/ PMC. Given that most of the members are stalwarts in the Apache
> > >communities, this should be a cinch IMO.
> > >   ― The work on this new code can start, and depending on its state,
> and
> > >assuming that the experiment succeeds, can be merged into trunk and
> later
> > >picked up in the next nearest & feasible Apache Hadoop release.
> > >   ― While the work on forked-over-code goes on till we have a
> > >functioning app, the current Apache Slider project continues to live,
> > >with supporting releases etc. At some point in the future, when things
> > >become clearer, more decisions can be taken on the (parts left behind in
> > >the) incubating project’s future.
> > >
> > >Thoughts?
> > >
> > >Thanks
> > >+Vinod
> >
> >
>
>
> --
> Sent from some random computer....
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

An interesting proposal, we can continue this discussion on the Slider dev lists.

Thanks
+Vinod

> On Jul 19, 2016, at 4:51 PM, jonmaron@gmail.com wrote:
> 
> +1 - this approach definitely addresses an important need.
> 
> The project has gone through at least a couple of development/repo
> approaches:
> 
> 1)  git-flow with master/develop branches
> 2)  abandonment of the master branch and work/releases based on the develop
> branch.
> 
> I have no strong objections to the approach above, but given the existence
> of the two branches above, and some developer familiarity with the
> 'develop' branch, perhaps we could:
> 
> 1)  Synch the master branch to the develop branch (not sure which git
> mechanism would best achieve this)
> 2)  Leverage the develop branch to create the agent/app-package based
> Slider distribution
> 
> In this way there'a a legacy branch available while develop continues to
> proceed on the existing and familiar development branch.
> 
> Just a thought.  Like I said - no real strong feelings :)
> 
> On Mon, Jul 18, 2016 at 3:23 PM, Gour Saha <gs...@hortonworks.com> wrote:
> 
>> +1 forwarding from the discussion in Slider DL
>> 
>> Note: On the Slider side, we plan to create a branch corresponding to this
>> YARN branch. In this, we intend to remove all the pieces which will move
>> to the YARN branch (Slider core, AM, client). We will retain the Agent and
>> app-packages which will then depend on the new hadoop-slider module (in
>> addition to the existing hadoop modules that it already depends on). This
>> will create a single view of Slider codebase, exactly as it stands today,
>> fully supporting the current app-packages. Slider can even release its
>> future versions from this new branch, once the hadoop-sldier module
>> reaches a matured state. It will be an easier path for existing Slider
>> users/app-owners to move to the future stable state of Slider completely
>> off of Hadoop YARN codebase. All that would be expected is to migrate the
>> app-packages to the agent-less version. Of course the Slider community
>> will do the migration for the current app-packages in the Slider codebase.
>> 
>> -Gour
>> 
>> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>> 
>>> Hi, Hadoop YARN community!
>>> 
>>> (Cross-posting across Hadoop and Slider communities)
>>> 
>>> I opened a JIRA a little while ago to pursue a native YARN framework for
>>> services: https://issues.apache.org/jira/browse/YARN-5079.
>>> 
>>> It is part of a bigger effort that a bunch of us YARN community members
>>> are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>> and first-class support for services in YARN.
>>> 
>>> The idea is that with our current attention on making services
>>> first­-class, it's time to take a fresh look at how we can make Apache
>>> Hadoop YARN support services well out of the box. I’ve been looking at
>>> various possibilities - ranging from a custom new framework room scratch
>>> to using one of the existing projects - and stopped at Apache Slider
>>> (http://slider.incubator.apache.org) given its association with some of
>>> the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>> myself etc.).
>>> 
>>> Slider client & AM already handles a great deal of the functionality that
>>> we need. I posit that assimilating the client, ApplicationMaster etc of
>>> an existing framework like Apache Slider can serve our purpose really
>>> well. My early informal discussions about this with few Hadoop and Slider
>>> community members yielded generally favourable feedback.
>>> 
>>> The Apache Slider incubator community also discussed this and expressed
>>> generally positive interest in YARN taking up Slider’s key pieces, you
>>> can see that discussion here: https://s.apache.org/0hoh.
>>> 
>>> So in summary, we are looking to the following
>>> 
>>> - Code
>>>  ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>> Apache Hadoop for providing a native services experiences in YARN
>>>  ― Leave for now some of the pieces behind in Apache Slider - (a)
>>> Slider agent as we won’t need it, (b) Slider packages that need more
>>> deliberation in terms of where they will live in the long term.
>>>  ― Create a branch in YARN, copy this code over into a new module(s),
>>> and work towards completing a functioning app running on YARN.
>>> 
>>> - Communities & releases
>>>  ― Good thing is that many of Apache Slider community members are
>>> already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>> committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>> without complicating things much, a proposed path forward is active
>>> participation in the branch (as branch committers?) and eventually in
>>> mainline YARN and thus go through a natural progression to committership
>>> / PMC. Given that most of the members are stalwarts in the Apache
>>> communities, this should be a cinch IMO.
>>>  ― The work on this new code can start, and depending on its state, and
>>> assuming that the experiment succeeds, can be merged into trunk and later
>>> picked up in the next nearest & feasible Apache Hadoop release.
>>>  ― While the work on forked-over-code goes on till we have a
>>> functioning app, the current Apache Slider project continues to live,
>>> with supporting releases etc. At some point in the future, when things
>>> become clearer, more decisions can be taken on the (parts left behind in
>>> the) incubating project’s future.
>>> 
>>> Thoughts?
>>> 
>>> Thanks
>>> +Vinod
>> 
>> 
> 
> 
> -- 
> Sent from some random computer....

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

["jonmaron@gmail.com" <jo...@gmail.com>]

+1 - this approach definitely addresses an important need.

The project has gone through at least a couple of development/repo
approaches:

1)  git-flow with master/develop branches
2)  abandonment of the master branch and work/releases based on the develop
branch.

I have no strong objections to the approach above, but given the existence
of the two branches above, and some developer familiarity with the
'develop' branch, perhaps we could:

1)  Synch the master branch to the develop branch (not sure which git
mechanism would best achieve this)
2)  Leverage the develop branch to create the agent/app-package based
Slider distribution

In this way there'a a legacy branch available while develop continues to
proceed on the existing and familiar development branch.

Just a thought.  Like I said - no real strong feelings :)

On Mon, Jul 18, 2016 at 3:23 PM, Gour Saha <gs...@hortonworks.com> wrote:

> +1 forwarding from the discussion in Slider DL
>
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.
>
> -Gour
>
> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
> >Hi, Hadoop YARN community!
> >
> >(Cross-posting across Hadoop and Slider communities)
> >
> >I opened a JIRA a little while ago to pursue a native YARN framework for
> >services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> >It is part of a bigger effort that a bunch of us YARN community members
> >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> >and first-class support for services in YARN.
> >
> >The idea is that with our current attention on making services
> >first­-class, it's time to take a fresh look at how we can make Apache
> >Hadoop YARN support services well out of the box. I’ve been looking at
> >various possibilities - ranging from a custom new framework room scratch
> >to using one of the existing projects - and stopped at Apache Slider
> >(http://slider.incubator.apache.org) given its association with some of
> >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> >myself etc.).
> >
> >Slider client & AM already handles a great deal of the functionality that
> >we need. I posit that assimilating the client, ApplicationMaster etc of
> >an existing framework like Apache Slider can serve our purpose really
> >well. My early informal discussions about this with few Hadoop and Slider
> >community members yielded generally favourable feedback.
> >
> >The Apache Slider incubator community also discussed this and expressed
> >generally positive interest in YARN taking up Slider’s key pieces, you
> >can see that discussion here: https://s.apache.org/0hoh.
> >
> >So in summary, we are looking to the following
> >
> >- Code
> >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> >Apache Hadoop for providing a native services experiences in YARN
> >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> >Slider agent as we won’t need it, (b) Slider packages that need more
> >deliberation in terms of where they will live in the long term.
> >   ― Create a branch in YARN, copy this code over into a new module(s),
> >and work towards completing a functioning app running on YARN.
> >
> >- Communities & releases
> >   ― Good thing is that many of Apache Slider community members are
> >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> >without complicating things much, a proposed path forward is active
> >participation in the branch (as branch committers?) and eventually in
> >mainline YARN and thus go through a natural progression to committership
> >/ PMC. Given that most of the members are stalwarts in the Apache
> >communities, this should be a cinch IMO.
> >   ― The work on this new code can start, and depending on its state, and
> >assuming that the experiment succeeds, can be merged into trunk and later
> >picked up in the next nearest & feasible Apache Hadoop release.
> >   ― While the work on forked-over-code goes on till we have a
> >functioning app, the current Apache Slider project continues to live,
> >with supporting releases etc. At some point in the future, when things
> >become clearer, more decisions can be taken on the (parts left behind in
> >the) incubating project’s future.
> >
> >Thoughts?
> >
> >Thanks
> >+Vinod
>
>


-- 
Sent from some random computer....

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

["jonmaron@gmail.com" <jo...@gmail.com>]

+1 - this approach definitely addresses an important need.

The project has gone through at least a couple of development/repo
approaches:

1)  git-flow with master/develop branches
2)  abandonment of the master branch and work/releases based on the develop
branch.

I have no strong objections to the approach above, but given the existence
of the two branches above, and some developer familiarity with the
'develop' branch, perhaps we could:

1)  Synch the master branch to the develop branch (not sure which git
mechanism would best achieve this)
2)  Leverage the develop branch to create the agent/app-package based
Slider distribution

In this way there'a a legacy branch available while develop continues to
proceed on the existing and familiar development branch.

Just a thought.  Like I said - no real strong feelings :)

On Mon, Jul 18, 2016 at 3:23 PM, Gour Saha <gs...@hortonworks.com> wrote:

> +1 forwarding from the discussion in Slider DL
>
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.
>
> -Gour
>
> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
> >Hi, Hadoop YARN community!
> >
> >(Cross-posting across Hadoop and Slider communities)
> >
> >I opened a JIRA a little while ago to pursue a native YARN framework for
> >services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> >It is part of a bigger effort that a bunch of us YARN community members
> >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> >and first-class support for services in YARN.
> >
> >The idea is that with our current attention on making services
> >first­-class, it's time to take a fresh look at how we can make Apache
> >Hadoop YARN support services well out of the box. I’ve been looking at
> >various possibilities - ranging from a custom new framework room scratch
> >to using one of the existing projects - and stopped at Apache Slider
> >(http://slider.incubator.apache.org) given its association with some of
> >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> >myself etc.).
> >
> >Slider client & AM already handles a great deal of the functionality that
> >we need. I posit that assimilating the client, ApplicationMaster etc of
> >an existing framework like Apache Slider can serve our purpose really
> >well. My early informal discussions about this with few Hadoop and Slider
> >community members yielded generally favourable feedback.
> >
> >The Apache Slider incubator community also discussed this and expressed
> >generally positive interest in YARN taking up Slider’s key pieces, you
> >can see that discussion here: https://s.apache.org/0hoh.
> >
> >So in summary, we are looking to the following
> >
> >- Code
> >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> >Apache Hadoop for providing a native services experiences in YARN
> >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> >Slider agent as we won’t need it, (b) Slider packages that need more
> >deliberation in terms of where they will live in the long term.
> >   ― Create a branch in YARN, copy this code over into a new module(s),
> >and work towards completing a functioning app running on YARN.
> >
> >- Communities & releases
> >   ― Good thing is that many of Apache Slider community members are
> >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> >without complicating things much, a proposed path forward is active
> >participation in the branch (as branch committers?) and eventually in
> >mainline YARN and thus go through a natural progression to committership
> >/ PMC. Given that most of the members are stalwarts in the Apache
> >communities, this should be a cinch IMO.
> >   ― The work on this new code can start, and depending on its state, and
> >assuming that the experiment succeeds, can be merged into trunk and later
> >picked up in the next nearest & feasible Apache Hadoop release.
> >   ― While the work on forked-over-code goes on till we have a
> >functioning app, the current Apache Slider project continues to live,
> >with supporting releases etc. At some point in the future, when things
> >become clearer, more decisions can be taken on the (parts left behind in
> >the) incubating project’s future.
> >
> >Thoughts?
> >
> >Thanks
> >+Vinod
>
>


-- 
Sent from some random computer....

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Yes, that’s the general goal for YARN-4692 also!

Thanks
+Vinod

> On Jul 18, 2016, at 8:14 PM, Arun Suresh <as...@apache.org> wrote:
> 
> I would also like to see this effort encompass not just services but also
> general long-lived applications

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Yes, that’s the general goal for YARN-4692 also!

Thanks
+Vinod

> On Jul 18, 2016, at 8:14 PM, Arun Suresh <as...@apache.org> wrote:
> 
> I would also like to see this effort encompass not just services but also
> general long-lived applications

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Arun Suresh <as...@apache.org>]

+1
It is definitely a step in the right direction and it's about time YARN
recognizes services as first-class citizens.
I would also like to see this effort encompass not just services but also
general long-lived applications such as say storm/heron topologies, samza
and possibly support for frameworks like REEF.



On Mon, Jul 18, 2016 at 6:32 PM, Sangjin Lee <sj...@apache.org> wrote:

> +1. It would be great if it becomes the default way for services to run on
> YARN. That would also make it easier to make YARN features support services
> better. As you mentioned, there would be a lot of logistics to work out,
> and we'll need to talk about them in more detail as they come up.
>
> Regards,
> Sangjin
>
> On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 forwarding from the discussion in Slider DL
> >
> > Note: On the Slider side, we plan to create a branch corresponding to
> this
> > YARN branch. In this, we intend to remove all the pieces which will move
> > to the YARN branch (Slider core, AM, client). We will retain the Agent
> and
> > app-packages which will then depend on the new hadoop-slider module (in
> > addition to the existing hadoop modules that it already depends on). This
> > will create a single view of Slider codebase, exactly as it stands today,
> > fully supporting the current app-packages. Slider can even release its
> > future versions from this new branch, once the hadoop-sldier module
> > reaches a matured state. It will be an easier path for existing Slider
> > users/app-owners to move to the future stable state of Slider completely
> > off of Hadoop YARN codebase. All that would be expected is to migrate the
> > app-packages to the agent-less version. Of course the Slider community
> > will do the migration for the current app-packages in the Slider
> codebase.
> >
> > -Gour
> >
> > On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org>
> wrote:
> >
> > >Hi, Hadoop YARN community!
> > >
> > >(Cross-posting across Hadoop and Slider communities)
> > >
> > >I opened a JIRA a little while ago to pursue a native YARN framework for
> > >services: https://issues.apache.org/jira/browse/YARN-5079.
> > >
> > >It is part of a bigger effort that a bunch of us YARN community members
> > >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> > >and first-class support for services in YARN.
> > >
> > >The idea is that with our current attention on making services
> > >first­-class, it's time to take a fresh look at how we can make Apache
> > >Hadoop YARN support services well out of the box. I’ve been looking at
> > >various possibilities - ranging from a custom new framework room scratch
> > >to using one of the existing projects - and stopped at Apache Slider
> > >(http://slider.incubator.apache.org) given its association with some of
> > >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > >myself etc.).
> > >
> > >Slider client & AM already handles a great deal of the functionality
> that
> > >we need. I posit that assimilating the client, ApplicationMaster etc of
> > >an existing framework like Apache Slider can serve our purpose really
> > >well. My early informal discussions about this with few Hadoop and
> Slider
> > >community members yielded generally favourable feedback.
> > >
> > >The Apache Slider incubator community also discussed this and expressed
> > >generally positive interest in YARN taking up Slider’s key pieces, you
> > >can see that discussion here: https://s.apache.org/0hoh.
> > >
> > >So in summary, we are looking to the following
> > >
> > >- Code
> > >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > >Apache Hadoop for providing a native services experiences in YARN
> > >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> > >Slider agent as we won’t need it, (b) Slider packages that need more
> > >deliberation in terms of where they will live in the long term.
> > >   ― Create a branch in YARN, copy this code over into a new module(s),
> > >and work towards completing a functioning app running on YARN.
> > >
> > >- Communities & releases
> > >   ― Good thing is that many of Apache Slider community members are
> > >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > >without complicating things much, a proposed path forward is active
> > >participation in the branch (as branch committers?) and eventually in
> > >mainline YARN and thus go through a natural progression to committership
> > >/ PMC. Given that most of the members are stalwarts in the Apache
> > >communities, this should be a cinch IMO.
> > >   ― The work on this new code can start, and depending on its state,
> and
> > >assuming that the experiment succeeds, can be merged into trunk and
> later
> > >picked up in the next nearest & feasible Apache Hadoop release.
> > >   ― While the work on forked-over-code goes on till we have a
> > >functioning app, the current Apache Slider project continues to live,
> > >with supporting releases etc. At some point in the future, when things
> > >become clearer, more decisions can be taken on the (parts left behind in
> > >the) incubating project’s future.
> > >
> > >Thoughts?
> > >
> > >Thanks
> > >+Vinod
> >
> >
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Arun Suresh <as...@apache.org>]

+1
It is definitely a step in the right direction and it's about time YARN
recognizes services as first-class citizens.
I would also like to see this effort encompass not just services but also
general long-lived applications such as say storm/heron topologies, samza
and possibly support for frameworks like REEF.



On Mon, Jul 18, 2016 at 6:32 PM, Sangjin Lee <sj...@apache.org> wrote:

> +1. It would be great if it becomes the default way for services to run on
> YARN. That would also make it easier to make YARN features support services
> better. As you mentioned, there would be a lot of logistics to work out,
> and we'll need to talk about them in more detail as they come up.
>
> Regards,
> Sangjin
>
> On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> > +1 forwarding from the discussion in Slider DL
> >
> > Note: On the Slider side, we plan to create a branch corresponding to
> this
> > YARN branch. In this, we intend to remove all the pieces which will move
> > to the YARN branch (Slider core, AM, client). We will retain the Agent
> and
> > app-packages which will then depend on the new hadoop-slider module (in
> > addition to the existing hadoop modules that it already depends on). This
> > will create a single view of Slider codebase, exactly as it stands today,
> > fully supporting the current app-packages. Slider can even release its
> > future versions from this new branch, once the hadoop-sldier module
> > reaches a matured state. It will be an easier path for existing Slider
> > users/app-owners to move to the future stable state of Slider completely
> > off of Hadoop YARN codebase. All that would be expected is to migrate the
> > app-packages to the agent-less version. Of course the Slider community
> > will do the migration for the current app-packages in the Slider
> codebase.
> >
> > -Gour
> >
> > On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org>
> wrote:
> >
> > >Hi, Hadoop YARN community!
> > >
> > >(Cross-posting across Hadoop and Slider communities)
> > >
> > >I opened a JIRA a little while ago to pursue a native YARN framework for
> > >services: https://issues.apache.org/jira/browse/YARN-5079.
> > >
> > >It is part of a bigger effort that a bunch of us YARN community members
> > >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> > >and first-class support for services in YARN.
> > >
> > >The idea is that with our current attention on making services
> > >first­-class, it's time to take a fresh look at how we can make Apache
> > >Hadoop YARN support services well out of the box. I’ve been looking at
> > >various possibilities - ranging from a custom new framework room scratch
> > >to using one of the existing projects - and stopped at Apache Slider
> > >(http://slider.incubator.apache.org) given its association with some of
> > >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > >myself etc.).
> > >
> > >Slider client & AM already handles a great deal of the functionality
> that
> > >we need. I posit that assimilating the client, ApplicationMaster etc of
> > >an existing framework like Apache Slider can serve our purpose really
> > >well. My early informal discussions about this with few Hadoop and
> Slider
> > >community members yielded generally favourable feedback.
> > >
> > >The Apache Slider incubator community also discussed this and expressed
> > >generally positive interest in YARN taking up Slider’s key pieces, you
> > >can see that discussion here: https://s.apache.org/0hoh.
> > >
> > >So in summary, we are looking to the following
> > >
> > >- Code
> > >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > >Apache Hadoop for providing a native services experiences in YARN
> > >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> > >Slider agent as we won’t need it, (b) Slider packages that need more
> > >deliberation in terms of where they will live in the long term.
> > >   ― Create a branch in YARN, copy this code over into a new module(s),
> > >and work towards completing a functioning app running on YARN.
> > >
> > >- Communities & releases
> > >   ― Good thing is that many of Apache Slider community members are
> > >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > >without complicating things much, a proposed path forward is active
> > >participation in the branch (as branch committers?) and eventually in
> > >mainline YARN and thus go through a natural progression to committership
> > >/ PMC. Given that most of the members are stalwarts in the Apache
> > >communities, this should be a cinch IMO.
> > >   ― The work on this new code can start, and depending on its state,
> and
> > >assuming that the experiment succeeds, can be merged into trunk and
> later
> > >picked up in the next nearest & feasible Apache Hadoop release.
> > >   ― While the work on forked-over-code goes on till we have a
> > >functioning app, the current Apache Slider project continues to live,
> > >with supporting releases etc. At some point in the future, when things
> > >become clearer, more decisions can be taken on the (parts left behind in
> > >the) incubating project’s future.
> > >
> > >Thoughts?
> > >
> > >Thanks
> > >+Vinod
> >
> >
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Sangjin Lee <sj...@apache.org>]

+1. It would be great if it becomes the default way for services to run on
YARN. That would also make it easier to make YARN features support services
better. As you mentioned, there would be a lot of logistics to work out,
and we'll need to talk about them in more detail as they come up.

Regards,
Sangjin

On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gs...@hortonworks.com> wrote:

> +1 forwarding from the discussion in Slider DL
>
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.
>
> -Gour
>
> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
> >Hi, Hadoop YARN community!
> >
> >(Cross-posting across Hadoop and Slider communities)
> >
> >I opened a JIRA a little while ago to pursue a native YARN framework for
> >services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> >It is part of a bigger effort that a bunch of us YARN community members
> >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> >and first-class support for services in YARN.
> >
> >The idea is that with our current attention on making services
> >first­-class, it's time to take a fresh look at how we can make Apache
> >Hadoop YARN support services well out of the box. I’ve been looking at
> >various possibilities - ranging from a custom new framework room scratch
> >to using one of the existing projects - and stopped at Apache Slider
> >(http://slider.incubator.apache.org) given its association with some of
> >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> >myself etc.).
> >
> >Slider client & AM already handles a great deal of the functionality that
> >we need. I posit that assimilating the client, ApplicationMaster etc of
> >an existing framework like Apache Slider can serve our purpose really
> >well. My early informal discussions about this with few Hadoop and Slider
> >community members yielded generally favourable feedback.
> >
> >The Apache Slider incubator community also discussed this and expressed
> >generally positive interest in YARN taking up Slider’s key pieces, you
> >can see that discussion here: https://s.apache.org/0hoh.
> >
> >So in summary, we are looking to the following
> >
> >- Code
> >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> >Apache Hadoop for providing a native services experiences in YARN
> >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> >Slider agent as we won’t need it, (b) Slider packages that need more
> >deliberation in terms of where they will live in the long term.
> >   ― Create a branch in YARN, copy this code over into a new module(s),
> >and work towards completing a functioning app running on YARN.
> >
> >- Communities & releases
> >   ― Good thing is that many of Apache Slider community members are
> >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> >without complicating things much, a proposed path forward is active
> >participation in the branch (as branch committers?) and eventually in
> >mainline YARN and thus go through a natural progression to committership
> >/ PMC. Given that most of the members are stalwarts in the Apache
> >communities, this should be a cinch IMO.
> >   ― The work on this new code can start, and depending on its state, and
> >assuming that the experiment succeeds, can be merged into trunk and later
> >picked up in the next nearest & feasible Apache Hadoop release.
> >   ― While the work on forked-over-code goes on till we have a
> >functioning app, the current Apache Slider project continues to live,
> >with supporting releases etc. At some point in the future, when things
> >become clearer, more decisions can be taken on the (parts left behind in
> >the) incubating project’s future.
> >
> >Thoughts?
> >
> >Thanks
> >+Vinod
>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

This sounds like a good plan for ongoing existence of the code in both the places, Gour!

Thanks
+Vinod

> On Jul 18, 2016, at 12:23 PM, Gour Saha <gsaha@hortonworks.com <ma...@hortonworks.com>> wrote:
> 
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Varun Vasudev <vv...@apache.org>]

+1. This should go a long way to help running services on YARN.

-Varun

On 7/19/16, 12:53 AM, "Gour Saha" <gs...@hortonworks.com> wrote:

>+1 forwarding from the discussion in Slider DL
>
>Note: On the Slider side, we plan to create a branch corresponding to this
>YARN branch. In this, we intend to remove all the pieces which will move
>to the YARN branch (Slider core, AM, client). We will retain the Agent and
>app-packages which will then depend on the new hadoop-slider module (in
>addition to the existing hadoop modules that it already depends on). This
>will create a single view of Slider codebase, exactly as it stands today,
>fully supporting the current app-packages. Slider can even release its
>future versions from this new branch, once the hadoop-sldier module
>reaches a matured state. It will be an easier path for existing Slider
>users/app-owners to move to the future stable state of Slider completely
>off of Hadoop YARN codebase. All that would be expected is to migrate the
>app-packages to the agent-less version. Of course the Slider community
>will do the migration for the current app-packages in the Slider codebase.
>
>-Gour
>
>On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
>>Hi, Hadoop YARN community!
>>
>>(Cross-posting across Hadoop and Slider communities)
>>
>>I opened a JIRA a little while ago to pursue a native YARN framework for
>>services: https://issues.apache.org/jira/browse/YARN-5079.
>>
>>It is part of a bigger effort that a bunch of us YARN community members
>>are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>and first-class support for services in YARN.
>>
>>The idea is that with our current attention on making services
>>first–-class, it's time to take a fresh look at how we can make Apache
>>Hadoop YARN support services well out of the box. I’ve been looking at
>>various possibilities - ranging from a custom new framework room scratch
>>to using one of the existing projects - and stopped at Apache Slider
>>(http://slider.incubator.apache.org) given its association with some of
>>the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>myself etc.).
>>
>>Slider client & AM already handles a great deal of the functionality that
>>we need. I posit that assimilating the client, ApplicationMaster etc of
>>an existing framework like Apache Slider can serve our purpose really
>>well. My early informal discussions about this with few Hadoop and Slider
>>community members yielded generally favourable feedback.
>>
>>The Apache Slider incubator community also discussed this and expressed
>>generally positive interest in YARN taking up Slider’s key pieces, you
>>can see that discussion here: https://s.apache.org/0hoh.
>>
>>So in summary, we are looking to the following
>>
>>- Code
>>   — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>Apache Hadoop for providing a native services experiences in YARN
>>   — Leave for now some of the pieces behind in Apache Slider - (a)
>>Slider agent as we won’t need it, (b) Slider packages that need more
>>deliberation in terms of where they will live in the long term.
>>   — Create a branch in YARN, copy this code over into a new module(s),
>>and work towards completing a functioning app running on YARN.
>>
>>- Communities & releases
>>   — Good thing is that many of Apache Slider community members are
>>already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>without complicating things much, a proposed path forward is active
>>participation in the branch (as branch committers?) and eventually in
>>mainline YARN and thus go through a natural progression to committership
>>/ PMC. Given that most of the members are stalwarts in the Apache
>>communities, this should be a cinch IMO.
>>   — The work on this new code can start, and depending on its state, and
>>assuming that the experiment succeeds, can be merged into trunk and later
>>picked up in the next nearest & feasible Apache Hadoop release.
>>   — While the work on forked-over-code goes on till we have a
>>functioning app, the current Apache Slider project continues to live,
>>with supporting releases etc. At some point in the future, when things
>>become clearer, more decisions can be taken on the (parts left behind in
>>the) incubating project’s future.
>>
>>Thoughts?
>>
>>Thanks
>>+Vinod
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>



---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Siddharth Seth <ss...@apache.org>]

+1 as a step towards better service support in YARN.

In terms of compatibility and Slider fixes - is the intent to allow
existing slider packages to work without migrating over to the new
agent-less, etc implementation in Hadoop. e.g. Assuming the slider
integration work is released in Hadoop 2.xx - would an existing slider
package which works on Hadoop-2.6 + Slider 0.91.0 continue to work without
changes?

On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gs...@hortonworks.com> wrote:

> +1 forwarding from the discussion in Slider DL
>
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.
>
> -Gour
>
> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
> >Hi, Hadoop YARN community!
> >
> >(Cross-posting across Hadoop and Slider communities)
> >
> >I opened a JIRA a little while ago to pursue a native YARN framework for
> >services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> >It is part of a bigger effort that a bunch of us YARN community members
> >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> >and first-class support for services in YARN.
> >
> >The idea is that with our current attention on making services
> >first­-class, it's time to take a fresh look at how we can make Apache
> >Hadoop YARN support services well out of the box. I’ve been looking at
> >various possibilities - ranging from a custom new framework room scratch
> >to using one of the existing projects - and stopped at Apache Slider
> >(http://slider.incubator.apache.org) given its association with some of
> >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> >myself etc.).
> >
> >Slider client & AM already handles a great deal of the functionality that
> >we need. I posit that assimilating the client, ApplicationMaster etc of
> >an existing framework like Apache Slider can serve our purpose really
> >well. My early informal discussions about this with few Hadoop and Slider
> >community members yielded generally favourable feedback.
> >
> >The Apache Slider incubator community also discussed this and expressed
> >generally positive interest in YARN taking up Slider’s key pieces, you
> >can see that discussion here: https://s.apache.org/0hoh.
> >
> >So in summary, we are looking to the following
> >
> >- Code
> >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> >Apache Hadoop for providing a native services experiences in YARN
> >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> >Slider agent as we won’t need it, (b) Slider packages that need more
> >deliberation in terms of where they will live in the long term.
> >   ― Create a branch in YARN, copy this code over into a new module(s),
> >and work towards completing a functioning app running on YARN.
> >
> >- Communities & releases
> >   ― Good thing is that many of Apache Slider community members are
> >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> >without complicating things much, a proposed path forward is active
> >participation in the branch (as branch committers?) and eventually in
> >mainline YARN and thus go through a natural progression to committership
> >/ PMC. Given that most of the members are stalwarts in the Apache
> >communities, this should be a cinch IMO.
> >   ― The work on this new code can start, and depending on its state, and
> >assuming that the experiment succeeds, can be merged into trunk and later
> >picked up in the next nearest & feasible Apache Hadoop release.
> >   ― While the work on forked-over-code goes on till we have a
> >functioning app, the current Apache Slider project continues to live,
> >with supporting releases etc. At some point in the future, when things
> >become clearer, more decisions can be taken on the (parts left behind in
> >the) incubating project’s future.
> >
> >Thoughts?
> >
> >Thanks
> >+Vinod
>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Sangjin Lee <sj...@apache.org>]

+1. It would be great if it becomes the default way for services to run on
YARN. That would also make it easier to make YARN features support services
better. As you mentioned, there would be a lot of logistics to work out,
and we'll need to talk about them in more detail as they come up.

Regards,
Sangjin

On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gs...@hortonworks.com> wrote:

> +1 forwarding from the discussion in Slider DL
>
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.
>
> -Gour
>
> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:
>
> >Hi, Hadoop YARN community!
> >
> >(Cross-posting across Hadoop and Slider communities)
> >
> >I opened a JIRA a little while ago to pursue a native YARN framework for
> >services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> >It is part of a bigger effort that a bunch of us YARN community members
> >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> >and first-class support for services in YARN.
> >
> >The idea is that with our current attention on making services
> >first­-class, it's time to take a fresh look at how we can make Apache
> >Hadoop YARN support services well out of the box. I’ve been looking at
> >various possibilities - ranging from a custom new framework room scratch
> >to using one of the existing projects - and stopped at Apache Slider
> >(http://slider.incubator.apache.org) given its association with some of
> >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> >myself etc.).
> >
> >Slider client & AM already handles a great deal of the functionality that
> >we need. I posit that assimilating the client, ApplicationMaster etc of
> >an existing framework like Apache Slider can serve our purpose really
> >well. My early informal discussions about this with few Hadoop and Slider
> >community members yielded generally favourable feedback.
> >
> >The Apache Slider incubator community also discussed this and expressed
> >generally positive interest in YARN taking up Slider’s key pieces, you
> >can see that discussion here: https://s.apache.org/0hoh.
> >
> >So in summary, we are looking to the following
> >
> >- Code
> >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> >Apache Hadoop for providing a native services experiences in YARN
> >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> >Slider agent as we won’t need it, (b) Slider packages that need more
> >deliberation in terms of where they will live in the long term.
> >   ― Create a branch in YARN, copy this code over into a new module(s),
> >and work towards completing a functioning app running on YARN.
> >
> >- Communities & releases
> >   ― Good thing is that many of Apache Slider community members are
> >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> >without complicating things much, a proposed path forward is active
> >participation in the branch (as branch committers?) and eventually in
> >mainline YARN and thus go through a natural progression to committership
> >/ PMC. Given that most of the members are stalwarts in the Apache
> >communities, this should be a cinch IMO.
> >   ― The work on this new code can start, and depending on its state, and
> >assuming that the experiment succeeds, can be merged into trunk and later
> >picked up in the next nearest & feasible Apache Hadoop release.
> >   ― While the work on forked-over-code goes on till we have a
> >functioning app, the current Apache Slider project continues to live,
> >with supporting releases etc. At some point in the future, when things
> >become clearer, more decisions can be taken on the (parts left behind in
> >the) incubating project’s future.
> >
> >Thoughts?
> >
> >Thanks
> >+Vinod
>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

This sounds like a good plan for ongoing existence of the code in both the places, Gour!

Thanks
+Vinod

> On Jul 18, 2016, at 12:23 PM, Gour Saha <gsaha@hortonworks.com <ma...@hortonworks.com>> wrote:
> 
> Note: On the Slider side, we plan to create a branch corresponding to this
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent and
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase.

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Gour Saha <gs...@hortonworks.com>]

+1 forwarding from the discussion in Slider DL

Note: On the Slider side, we plan to create a branch corresponding to this
YARN branch. In this, we intend to remove all the pieces which will move
to the YARN branch (Slider core, AM, client). We will retain the Agent and
app-packages which will then depend on the new hadoop-slider module (in
addition to the existing hadoop modules that it already depends on). This
will create a single view of Slider codebase, exactly as it stands today,
fully supporting the current app-packages. Slider can even release its
future versions from this new branch, once the hadoop-sldier module
reaches a matured state. It will be an easier path for existing Slider
users/app-owners to move to the future stable state of Slider completely
off of Hadoop YARN codebase. All that would be expected is to migrate the
app-packages to the agent-less version. Of course the Slider community
will do the migration for the current app-packages in the Slider codebase.

-Gour

On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:

>Hi, Hadoop YARN community!
>
>(Cross-posting across Hadoop and Slider communities)
>
>I opened a JIRA a little while ago to pursue a native YARN framework for
>services: https://issues.apache.org/jira/browse/YARN-5079.
>
>It is part of a bigger effort that a bunch of us YARN community members
>are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>and first-class support for services in YARN.
>
>The idea is that with our current attention on making services
>first­-class, it's time to take a fresh look at how we can make Apache
>Hadoop YARN support services well out of the box. I’ve been looking at
>various possibilities - ranging from a custom new framework room scratch
>to using one of the existing projects - and stopped at Apache Slider
>(http://slider.incubator.apache.org) given its association with some of
>the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>myself etc.).
>
>Slider client & AM already handles a great deal of the functionality that
>we need. I posit that assimilating the client, ApplicationMaster etc of
>an existing framework like Apache Slider can serve our purpose really
>well. My early informal discussions about this with few Hadoop and Slider
>community members yielded generally favourable feedback.
>
>The Apache Slider incubator community also discussed this and expressed
>generally positive interest in YARN taking up Slider’s key pieces, you
>can see that discussion here: https://s.apache.org/0hoh.
>
>So in summary, we are looking to the following
>
>- Code
>   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>Apache Hadoop for providing a native services experiences in YARN
>   ― Leave for now some of the pieces behind in Apache Slider - (a)
>Slider agent as we won’t need it, (b) Slider packages that need more
>deliberation in terms of where they will live in the long term.
>   ― Create a branch in YARN, copy this code over into a new module(s),
>and work towards completing a functioning app running on YARN.
>
>- Communities & releases
>   ― Good thing is that many of Apache Slider community members are
>already seasoned folk in the Apache Hadoop ecosystem projects. For those
>committers & PMC in Slider that are not yet Hadoop committers / PMC,
>without complicating things much, a proposed path forward is active
>participation in the branch (as branch committers?) and eventually in
>mainline YARN and thus go through a natural progression to committership
>/ PMC. Given that most of the members are stalwarts in the Apache
>communities, this should be a cinch IMO.
>   ― The work on this new code can start, and depending on its state, and
>assuming that the experiment succeeds, can be merged into trunk and later
>picked up in the next nearest & feasible Apache Hadoop release.
>   ― While the work on forked-over-code goes on till we have a
>functioning app, the current Apache Slider project continues to live,
>with supporting releases etc. At some point in the future, when things
>become clearer, more decisions can be taken on the (parts left behind in
>the) incubating project’s future.
>
>Thoughts?
>
>Thanks
>+Vinod

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Thanks for the discussion and the feedback, everyone!

I sense a general consensus, so am going to go ahead with the branch proposal.

Please regard this as the conclusion of the discussion here. If you have more inputs, feel free to jump on to the JIRA YARN-5079!

Thanks again,
+Vinod

> On Jul 14, 2016, at 7:36 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi, Hadoop YARN community!
> 
> (Cross-posting across Hadoop and Slider communities)
> 
> I opened a JIRA a little while ago to pursue a native YARN framework for services: https://issues.apache.org/jira/browse/YARN-5079.
> 
> It is part of a bigger effort that a bunch of us YARN community members are interested in making progress on: YARN-4692 - [Umbrella] Simplified and first-class support for services in YARN.
> 
> The idea is that with our current attention on making services first­-class, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. I’ve been looking at various possibilities - ranging from a custom new framework room scratch to using one of the existing projects - and stopped at Apache Slider (http://slider.incubator.apache.org) given its association with some of the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy, myself etc.).
> 
> Slider client & AM already handles a great deal of the functionality that we need. I posit that assimilating the client, ApplicationMaster etc of an existing framework like Apache Slider can serve our purpose really well. My early informal discussions about this with few Hadoop and Slider community members yielded generally favourable feedback.
> 
> The Apache Slider incubator community also discussed this and expressed generally positive interest in YARN taking up Slider’s key pieces, you can see that discussion here: https://s.apache.org/0hoh.
> 
> So in summary, we are looking to the following
> 
> - Code
>   — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into Apache Hadoop for providing a native services experiences in YARN
>   — Leave for now some of the pieces behind in Apache Slider - (a) Slider agent as we won’t need it, (b) Slider packages that need more deliberation in terms of where they will live in the long term.
>   — Create a branch in YARN, copy this code over into a new module(s), and work towards completing a functioning app running on YARN.
> 
> - Communities & releases
>   — Good thing is that many of Apache Slider community members are already seasoned folk in the Apache Hadoop ecosystem projects. For those committers & PMC in Slider that are not yet Hadoop committers / PMC, without complicating things much, a proposed path forward is active participation in the branch (as branch committers?) and eventually in mainline YARN and thus go through a natural progression to committership / PMC. Given that most of the members are stalwarts in the Apache communities, this should be a cinch IMO.
>   — The work on this new code can start, and depending on its state, and assuming that the experiment succeeds, can be merged into trunk and later picked up in the next nearest & feasible Apache Hadoop release.
>   — While the work on forked-over-code goes on till we have a functioning app, the current Apache Slider project continues to live, with supporting releases etc. At some point in the future, when things become clearer, more decisions can be taken on the (parts left behind in the) incubating project’s future.
> 
> Thoughts?
> 
> Thanks
> +Vinod


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Vinod Kumar Vavilapalli <vi...@apache.org>]

Thanks for the discussion and the feedback, everyone!

I sense a general consensus, so am going to go ahead with the branch proposal.

Please regard this as the conclusion of the discussion here. If you have more inputs, feel free to jump on to the JIRA YARN-5079!

Thanks again,
+Vinod

> On Jul 14, 2016, at 7:36 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> 
> Hi, Hadoop YARN community!
> 
> (Cross-posting across Hadoop and Slider communities)
> 
> I opened a JIRA a little while ago to pursue a native YARN framework for services: https://issues.apache.org/jira/browse/YARN-5079.
> 
> It is part of a bigger effort that a bunch of us YARN community members are interested in making progress on: YARN-4692 - [Umbrella] Simplified and first-class support for services in YARN.
> 
> The idea is that with our current attention on making services first­-class, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. I’ve been looking at various possibilities - ranging from a custom new framework room scratch to using one of the existing projects - and stopped at Apache Slider (http://slider.incubator.apache.org) given its association with some of the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy, myself etc.).
> 
> Slider client & AM already handles a great deal of the functionality that we need. I posit that assimilating the client, ApplicationMaster etc of an existing framework like Apache Slider can serve our purpose really well. My early informal discussions about this with few Hadoop and Slider community members yielded generally favourable feedback.
> 
> The Apache Slider incubator community also discussed this and expressed generally positive interest in YARN taking up Slider’s key pieces, you can see that discussion here: https://s.apache.org/0hoh.
> 
> So in summary, we are looking to the following
> 
> - Code
>   — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into Apache Hadoop for providing a native services experiences in YARN
>   — Leave for now some of the pieces behind in Apache Slider - (a) Slider agent as we won’t need it, (b) Slider packages that need more deliberation in terms of where they will live in the long term.
>   — Create a branch in YARN, copy this code over into a new module(s), and work towards completing a functioning app running on YARN.
> 
> - Communities & releases
>   — Good thing is that many of Apache Slider community members are already seasoned folk in the Apache Hadoop ecosystem projects. For those committers & PMC in Slider that are not yet Hadoop committers / PMC, without complicating things much, a proposed path forward is active participation in the branch (as branch committers?) and eventually in mainline YARN and thus go through a natural progression to committership / PMC. Given that most of the members are stalwarts in the Apache communities, this should be a cinch IMO.
>   — The work on this new code can start, and depending on its state, and assuming that the experiment succeeds, can be merged into trunk and later picked up in the next nearest & feasible Apache Hadoop release.
>   — While the work on forked-over-code goes on till we have a functioning app, the current Apache Slider project continues to live, with supporting releases etc. At some point in the future, when things become clearer, more decisions can be taken on the (parts left behind in the) incubating project’s future.
> 
> Thoughts?
> 
> Thanks
> +Vinod

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Karthik Kambatla <ka...@cloudera.com>]

+1. Merging the core parts of Slider into YARN should help with faster
development on service-oriented features.


On Tue, Jul 19, 2016 at 12:38 PM, Wangda Tan <wh...@gmail.com> wrote:

> +1,  this can avoid lots efforts for end user to run services on YARN.
>
> On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi <bi...@gmail.com>
> wrote:
>
> > +1. I think making Slider the basis of a services framework in YARN will
> > allow us to significantly improve YARN's support for services.
> >
> > On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org
> > > wrote:
> >
> > > Hi, Hadoop YARN community!
> > >
> > > (Cross-posting across Hadoop and Slider communities)
> > >
> > > I opened a JIRA a little while ago to pursue a native YARN framework
> for
> > > services: https://issues.apache.org/jira/browse/YARN-5079.
> > >
> > > It is part of a bigger effort that a bunch of us YARN community members
> > > are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> > and
> > > first-class support for services in YARN.
> > >
> > > The idea is that with our current attention on making services
> > > first­-class, it's time to take a fresh look at how we can make Apache
> > > Hadoop YARN support services well out of the box. I’ve been looking at
> > > various possibilities - ranging from a custom new framework room
> scratch
> > to
> > > using one of the existing projects - and stopped at Apache Slider (
> > > http://slider.incubator.apache.org) given its association with some of
> > > the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > > myself etc.).
> > >
> > > Slider client & AM already handles a great deal of the functionality
> that
> > > we need. I posit that assimilating the client, ApplicationMaster etc of
> > an
> > > existing framework like Apache Slider can serve our purpose really
> well.
> > My
> > > early informal discussions about this with few Hadoop and Slider
> > community
> > > members yielded generally favourable feedback.
> > >
> > > The Apache Slider incubator community also discussed this and expressed
> > > generally positive interest in YARN taking up Slider’s key pieces, you
> > can
> > > see that discussion here: https://s.apache.org/0hoh.
> > >
> > > So in summary, we are looking to the following
> > >
> > > - Code
> > >    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > > Apache Hadoop for providing a native services experiences in YARN
> > >    — Leave for now some of the pieces behind in Apache Slider - (a)
> > Slider
> > > agent as we won’t need it, (b) Slider packages that need more
> > deliberation
> > > in terms of where they will live in the long term.
> > >    — Create a branch in YARN, copy this code over into a new module(s),
> > > and work towards completing a functioning app running on YARN.
> > >
> > > - Communities & releases
> > >    — Good thing is that many of Apache Slider community members are
> > > already seasoned folk in the Apache Hadoop ecosystem projects. For
> those
> > > committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > > without complicating things much, a proposed path forward is active
> > > participation in the branch (as branch committers?) and eventually in
> > > mainline YARN and thus go through a natural progression to
> committership
> > /
> > > PMC. Given that most of the members are stalwarts in the Apache
> > > communities, this should be a cinch IMO.
> > >    — The work on this new code can start, and depending on its state,
> and
> > > assuming that the experiment succeeds, can be merged into trunk and
> later
> > > picked up in the next nearest & feasible Apache Hadoop release.
> > >    — While the work on forked-over-code goes on till we have a
> > functioning
> > > app, the current Apache Slider project continues to live, with
> supporting
> > > releases etc. At some point in the future, when things become clearer,
> > more
> > > decisions can be taken on the (parts left behind in the) incubating
> > > project’s future.
> > >
> > > Thoughts?
> > >
> > > Thanks
> > > +Vinod
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Karthik Kambatla <ka...@cloudera.com>]

+1. Merging the core parts of Slider into YARN should help with faster
development on service-oriented features.


On Tue, Jul 19, 2016 at 12:38 PM, Wangda Tan <wh...@gmail.com> wrote:

> +1,  this can avoid lots efforts for end user to run services on YARN.
>
> On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi <bi...@gmail.com>
> wrote:
>
> > +1. I think making Slider the basis of a services framework in YARN will
> > allow us to significantly improve YARN's support for services.
> >
> > On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org
> > > wrote:
> >
> > > Hi, Hadoop YARN community!
> > >
> > > (Cross-posting across Hadoop and Slider communities)
> > >
> > > I opened a JIRA a little while ago to pursue a native YARN framework
> for
> > > services: https://issues.apache.org/jira/browse/YARN-5079.
> > >
> > > It is part of a bigger effort that a bunch of us YARN community members
> > > are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> > and
> > > first-class support for services in YARN.
> > >
> > > The idea is that with our current attention on making services
> > > first­-class, it's time to take a fresh look at how we can make Apache
> > > Hadoop YARN support services well out of the box. I’ve been looking at
> > > various possibilities - ranging from a custom new framework room
> scratch
> > to
> > > using one of the existing projects - and stopped at Apache Slider (
> > > http://slider.incubator.apache.org) given its association with some of
> > > the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > > myself etc.).
> > >
> > > Slider client & AM already handles a great deal of the functionality
> that
> > > we need. I posit that assimilating the client, ApplicationMaster etc of
> > an
> > > existing framework like Apache Slider can serve our purpose really
> well.
> > My
> > > early informal discussions about this with few Hadoop and Slider
> > community
> > > members yielded generally favourable feedback.
> > >
> > > The Apache Slider incubator community also discussed this and expressed
> > > generally positive interest in YARN taking up Slider’s key pieces, you
> > can
> > > see that discussion here: https://s.apache.org/0hoh.
> > >
> > > So in summary, we are looking to the following
> > >
> > > - Code
> > >    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > > Apache Hadoop for providing a native services experiences in YARN
> > >    — Leave for now some of the pieces behind in Apache Slider - (a)
> > Slider
> > > agent as we won’t need it, (b) Slider packages that need more
> > deliberation
> > > in terms of where they will live in the long term.
> > >    — Create a branch in YARN, copy this code over into a new module(s),
> > > and work towards completing a functioning app running on YARN.
> > >
> > > - Communities & releases
> > >    — Good thing is that many of Apache Slider community members are
> > > already seasoned folk in the Apache Hadoop ecosystem projects. For
> those
> > > committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > > without complicating things much, a proposed path forward is active
> > > participation in the branch (as branch committers?) and eventually in
> > > mainline YARN and thus go through a natural progression to
> committership
> > /
> > > PMC. Given that most of the members are stalwarts in the Apache
> > > communities, this should be a cinch IMO.
> > >    — The work on this new code can start, and depending on its state,
> and
> > > assuming that the experiment succeeds, can be merged into trunk and
> later
> > > picked up in the next nearest & feasible Apache Hadoop release.
> > >    — While the work on forked-over-code goes on till we have a
> > functioning
> > > app, the current Apache Slider project continues to live, with
> supporting
> > > releases etc. At some point in the future, when things become clearer,
> > more
> > > decisions can be taken on the (parts left behind in the) incubating
> > > project’s future.
> > >
> > > Thoughts?
> > >
> > > Thanks
> > > +Vinod
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Wangda Tan <wh...@gmail.com>]

+1,  this can avoid lots efforts for end user to run services on YARN.

On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi <bi...@gmail.com>
wrote:

> +1. I think making Slider the basis of a services framework in YARN will
> allow us to significantly improve YARN's support for services.
>
> On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > Hi, Hadoop YARN community!
> >
> > (Cross-posting across Hadoop and Slider communities)
> >
> > I opened a JIRA a little while ago to pursue a native YARN framework for
> > services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> > It is part of a bigger effort that a bunch of us YARN community members
> > are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> and
> > first-class support for services in YARN.
> >
> > The idea is that with our current attention on making services
> > first­-class, it's time to take a fresh look at how we can make Apache
> > Hadoop YARN support services well out of the box. I’ve been looking at
> > various possibilities - ranging from a custom new framework room scratch
> to
> > using one of the existing projects - and stopped at Apache Slider (
> > http://slider.incubator.apache.org) given its association with some of
> > the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > myself etc.).
> >
> > Slider client & AM already handles a great deal of the functionality that
> > we need. I posit that assimilating the client, ApplicationMaster etc of
> an
> > existing framework like Apache Slider can serve our purpose really well.
> My
> > early informal discussions about this with few Hadoop and Slider
> community
> > members yielded generally favourable feedback.
> >
> > The Apache Slider incubator community also discussed this and expressed
> > generally positive interest in YARN taking up Slider’s key pieces, you
> can
> > see that discussion here: https://s.apache.org/0hoh.
> >
> > So in summary, we are looking to the following
> >
> > - Code
> >    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > Apache Hadoop for providing a native services experiences in YARN
> >    — Leave for now some of the pieces behind in Apache Slider - (a)
> Slider
> > agent as we won’t need it, (b) Slider packages that need more
> deliberation
> > in terms of where they will live in the long term.
> >    — Create a branch in YARN, copy this code over into a new module(s),
> > and work towards completing a functioning app running on YARN.
> >
> > - Communities & releases
> >    — Good thing is that many of Apache Slider community members are
> > already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > without complicating things much, a proposed path forward is active
> > participation in the branch (as branch committers?) and eventually in
> > mainline YARN and thus go through a natural progression to committership
> /
> > PMC. Given that most of the members are stalwarts in the Apache
> > communities, this should be a cinch IMO.
> >    — The work on this new code can start, and depending on its state, and
> > assuming that the experiment succeeds, can be merged into trunk and later
> > picked up in the next nearest & feasible Apache Hadoop release.
> >    — While the work on forked-over-code goes on till we have a
> functioning
> > app, the current Apache Slider project continues to live, with supporting
> > releases etc. At some point in the future, when things become clearer,
> more
> > decisions can be taken on the (parts left behind in the) incubating
> > project’s future.
> >
> > Thoughts?
> >
> > Thanks
> > +Vinod
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Wangda Tan <wh...@gmail.com>]

+1,  this can avoid lots efforts for end user to run services on YARN.

On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi <bi...@gmail.com>
wrote:

> +1. I think making Slider the basis of a services framework in YARN will
> allow us to significantly improve YARN's support for services.
>
> On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > Hi, Hadoop YARN community!
> >
> > (Cross-posting across Hadoop and Slider communities)
> >
> > I opened a JIRA a little while ago to pursue a native YARN framework for
> > services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> > It is part of a bigger effort that a bunch of us YARN community members
> > are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> and
> > first-class support for services in YARN.
> >
> > The idea is that with our current attention on making services
> > first­-class, it's time to take a fresh look at how we can make Apache
> > Hadoop YARN support services well out of the box. I’ve been looking at
> > various possibilities - ranging from a custom new framework room scratch
> to
> > using one of the existing projects - and stopped at Apache Slider (
> > http://slider.incubator.apache.org) given its association with some of
> > the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > myself etc.).
> >
> > Slider client & AM already handles a great deal of the functionality that
> > we need. I posit that assimilating the client, ApplicationMaster etc of
> an
> > existing framework like Apache Slider can serve our purpose really well.
> My
> > early informal discussions about this with few Hadoop and Slider
> community
> > members yielded generally favourable feedback.
> >
> > The Apache Slider incubator community also discussed this and expressed
> > generally positive interest in YARN taking up Slider’s key pieces, you
> can
> > see that discussion here: https://s.apache.org/0hoh.
> >
> > So in summary, we are looking to the following
> >
> > - Code
> >    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > Apache Hadoop for providing a native services experiences in YARN
> >    — Leave for now some of the pieces behind in Apache Slider - (a)
> Slider
> > agent as we won’t need it, (b) Slider packages that need more
> deliberation
> > in terms of where they will live in the long term.
> >    — Create a branch in YARN, copy this code over into a new module(s),
> > and work towards completing a functioning app running on YARN.
> >
> > - Communities & releases
> >    — Good thing is that many of Apache Slider community members are
> > already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > without complicating things much, a proposed path forward is active
> > participation in the branch (as branch committers?) and eventually in
> > mainline YARN and thus go through a natural progression to committership
> /
> > PMC. Given that most of the members are stalwarts in the Apache
> > communities, this should be a cinch IMO.
> >    — The work on this new code can start, and depending on its state, and
> > assuming that the experiment succeeds, can be merged into trunk and later
> > picked up in the next nearest & feasible Apache Hadoop release.
> >    — While the work on forked-over-code goes on till we have a
> functioning
> > app, the current Apache Slider project continues to live, with supporting
> > releases etc. At some point in the future, when things become clearer,
> more
> > decisions can be taken on the (parts left behind in the) incubating
> > project’s future.
> >
> > Thoughts?
> >
> > Thanks
> > +Vinod
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Zhihai Xu <zh...@uber.com>]

+1. Looking forward to this new feature,this will open a door for Yarn to
support more use cases.

zhihai


On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi <bi...@gmail.com>
wrote:

> +1. I think making Slider the basis of a services framework in YARN will
> allow us to significantly improve YARN's support for services.
>
> On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > Hi, Hadoop YARN community!
> >
> > (Cross-posting across Hadoop and Slider communities)
> >
> > I opened a JIRA a little while ago to pursue a native YARN framework for
> > services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> > It is part of a bigger effort that a bunch of us YARN community members
> > are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> and
> > first-class support for services in YARN.
> >
> > The idea is that with our current attention on making services
> > first­-class, it's time to take a fresh look at how we can make Apache
> > Hadoop YARN support services well out of the box. I’ve been looking at
> > various possibilities - ranging from a custom new framework room scratch
> to
> > using one of the existing projects - and stopped at Apache Slider (
> > http://slider.incubator.apache.org) given its association with some of
> > the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > myself etc.).
> >
> > Slider client & AM already handles a great deal of the functionality that
> > we need. I posit that assimilating the client, ApplicationMaster etc of
> an
> > existing framework like Apache Slider can serve our purpose really well.
> My
> > early informal discussions about this with few Hadoop and Slider
> community
> > members yielded generally favourable feedback.
> >
> > The Apache Slider incubator community also discussed this and expressed
> > generally positive interest in YARN taking up Slider’s key pieces, you
> can
> > see that discussion here: https://s.apache.org/0hoh.
> >
> > So in summary, we are looking to the following
> >
> > - Code
> >    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > Apache Hadoop for providing a native services experiences in YARN
> >    — Leave for now some of the pieces behind in Apache Slider - (a)
> Slider
> > agent as we won’t need it, (b) Slider packages that need more
> deliberation
> > in terms of where they will live in the long term.
> >    — Create a branch in YARN, copy this code over into a new module(s),
> > and work towards completing a functioning app running on YARN.
> >
> > - Communities & releases
> >    — Good thing is that many of Apache Slider community members are
> > already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > without complicating things much, a proposed path forward is active
> > participation in the branch (as branch committers?) and eventually in
> > mainline YARN and thus go through a natural progression to committership
> /
> > PMC. Given that most of the members are stalwarts in the Apache
> > communities, this should be a cinch IMO.
> >    — The work on this new code can start, and depending on its state, and
> > assuming that the experiment succeeds, can be merged into trunk and later
> > picked up in the next nearest & feasible Apache Hadoop release.
> >    — While the work on forked-over-code goes on till we have a
> functioning
> > app, the current Apache Slider project continues to live, with supporting
> > releases etc. At some point in the future, when things become clearer,
> more
> > decisions can be taken on the (parts left behind in the) incubating
> > project’s future.
> >
> > Thoughts?
> >
> > Thanks
> > +Vinod
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Billie Rinaldi <bi...@gmail.com>]

+1. I think making Slider the basis of a services framework in YARN will
allow us to significantly improve YARN's support for services.

On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> Hi, Hadoop YARN community!
>
> (Cross-posting across Hadoop and Slider communities)
>
> I opened a JIRA a little while ago to pursue a native YARN framework for
> services: https://issues.apache.org/jira/browse/YARN-5079.
>
> It is part of a bigger effort that a bunch of us YARN community members
> are interested in making progress on: YARN-4692 - [Umbrella] Simplified and
> first-class support for services in YARN.
>
> The idea is that with our current attention on making services
> first­-class, it's time to take a fresh look at how we can make Apache
> Hadoop YARN support services well out of the box. I’ve been looking at
> various possibilities - ranging from a custom new framework room scratch to
> using one of the existing projects - and stopped at Apache Slider (
> http://slider.incubator.apache.org) given its association with some of
> the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> myself etc.).
>
> Slider client & AM already handles a great deal of the functionality that
> we need. I posit that assimilating the client, ApplicationMaster etc of an
> existing framework like Apache Slider can serve our purpose really well. My
> early informal discussions about this with few Hadoop and Slider community
> members yielded generally favourable feedback.
>
> The Apache Slider incubator community also discussed this and expressed
> generally positive interest in YARN taking up Slider’s key pieces, you can
> see that discussion here: https://s.apache.org/0hoh.
>
> So in summary, we are looking to the following
>
> - Code
>    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> Apache Hadoop for providing a native services experiences in YARN
>    — Leave for now some of the pieces behind in Apache Slider - (a) Slider
> agent as we won’t need it, (b) Slider packages that need more deliberation
> in terms of where they will live in the long term.
>    — Create a branch in YARN, copy this code over into a new module(s),
> and work towards completing a functioning app running on YARN.
>
> - Communities & releases
>    — Good thing is that many of Apache Slider community members are
> already seasoned folk in the Apache Hadoop ecosystem projects. For those
> committers & PMC in Slider that are not yet Hadoop committers / PMC,
> without complicating things much, a proposed path forward is active
> participation in the branch (as branch committers?) and eventually in
> mainline YARN and thus go through a natural progression to committership /
> PMC. Given that most of the members are stalwarts in the Apache
> communities, this should be a cinch IMO.
>    — The work on this new code can start, and depending on its state, and
> assuming that the experiment succeeds, can be merged into trunk and later
> picked up in the next nearest & feasible Apache Hadoop release.
>    — While the work on forked-over-code goes on till we have a functioning
> app, the current Apache Slider project continues to live, with supporting
> releases etc. At some point in the future, when things become clearer, more
> decisions can be taken on the (parts left behind in the) incubating
> project’s future.
>
> Thoughts?
>
> Thanks
> +Vinod
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Gour Saha <gs...@hortonworks.com>]

The API layer effort as in https://issues.apache.org/jira/browse/YARN-4793
is REST, defined in swagger format (the first draft version is uploaded to
YARN-4793). Proposal is to source control this as the entry point for the
APIs. The server side stubs (Java) will be generated from this definition,
and used for the server side API implementation.

The client side stubs can also be generated from this swagger format in a
wide range of languages -
http://swagger.io/swagger-codegen/

https://github.com/swagger-api/swagger-codegen/tree/master

-Gour

On 7/18/16, 8:43 PM, "Tsuyoshi Ozawa" <oz...@apache.org> wrote:

>+1,
>
>From user's point of view, the high-level resource management API
>without any additional dependency is much comfortable than using
>low-level APIs. I believe the way should increase YARN applications
>including long-running services. From developer's point of view, the
>integration make YARN more stable and feature-rich since we can get
>feedback about YARN low-level APIs including backward compatibility,
>features we need, and so on.
>
>BTW, I have a question about language-bindings: can we prepare python,
>C++, or another language APIs over Slider? Is it out of focus of this
>discussion?
>
>Thanks,
>- Tsuyoshi
>
>On Fri, Jul 15, 2016 at 11:36 AM, Vinod Kumar Vavilapalli
><vi...@apache.org> wrote:
>> Hi, Hadoop YARN community!
>>
>> (Cross-posting across Hadoop and Slider communities)
>>
>> I opened a JIRA a little while ago to pursue a native YARN framework
>>for services: https://issues.apache.org/jira/browse/YARN-5079.
>>
>> It is part of a bigger effort that a bunch of us YARN community members
>>are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>and first-class support for services in YARN.
>>
>> The idea is that with our current attention on making services
>>first­-class, it's time to take a fresh look at how we can make Apache
>>Hadoop YARN support services well out of the box. I’ve been looking at
>>various possibilities - ranging from a custom new framework room scratch
>>to using one of the existing projects - and stopped at Apache Slider
>>(http://slider.incubator.apache.org) given its association with some of
>>the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>myself etc.).
>>
>> Slider client & AM already handles a great deal of the functionality
>>that we need. I posit that assimilating the client, ApplicationMaster
>>etc of an existing framework like Apache Slider can serve our purpose
>>really well. My early informal discussions about this with few Hadoop
>>and Slider community members yielded generally favourable feedback.
>>
>> The Apache Slider incubator community also discussed this and expressed
>>generally positive interest in YARN taking up Slider’s key pieces, you
>>can see that discussion here: https://s.apache.org/0hoh.
>>
>> So in summary, we are looking to the following
>>
>> - Code
>>    ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>Apache Hadoop for providing a native services experiences in YARN
>>    ― Leave for now some of the pieces behind in Apache Slider - (a)
>>Slider agent as we won’t need it, (b) Slider packages that need more
>>deliberation in terms of where they will live in the long term.
>>    ― Create a branch in YARN, copy this code over into a new module(s),
>>and work towards completing a functioning app running on YARN.
>>
>> - Communities & releases
>>    ― Good thing is that many of Apache Slider community members are
>>already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>without complicating things much, a proposed path forward is active
>>participation in the branch (as branch committers?) and eventually in
>>mainline YARN and thus go through a natural progression to committership
>>/ PMC. Given that most of the members are stalwarts in the Apache
>>communities, this should be a cinch IMO.
>>    ― The work on this new code can start, and depending on its state,
>>and assuming that the experiment succeeds, can be merged into trunk and
>>later picked up in the next nearest & feasible Apache Hadoop release.
>>    ― While the work on forked-over-code goes on till we have a
>>functioning app, the current Apache Slider project continues to live,
>>with supporting releases etc. At some point in the future, when things
>>become clearer, more decisions can be taken on the (parts left behind in
>>the) incubating project’s future.
>>
>> Thoughts?
>>
>> Thanks
>> +Vinod
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Gour Saha <gs...@hortonworks.com>]

The API layer effort as in https://issues.apache.org/jira/browse/YARN-4793
is REST, defined in swagger format (the first draft version is uploaded to
YARN-4793). Proposal is to source control this as the entry point for the
APIs. The server side stubs (Java) will be generated from this definition,
and used for the server side API implementation.

The client side stubs can also be generated from this swagger format in a
wide range of languages -
http://swagger.io/swagger-codegen/

https://github.com/swagger-api/swagger-codegen/tree/master

-Gour

On 7/18/16, 8:43 PM, "Tsuyoshi Ozawa" <oz...@apache.org> wrote:

>+1,
>
>From user's point of view, the high-level resource management API
>without any additional dependency is much comfortable than using
>low-level APIs. I believe the way should increase YARN applications
>including long-running services. From developer's point of view, the
>integration make YARN more stable and feature-rich since we can get
>feedback about YARN low-level APIs including backward compatibility,
>features we need, and so on.
>
>BTW, I have a question about language-bindings: can we prepare python,
>C++, or another language APIs over Slider? Is it out of focus of this
>discussion?
>
>Thanks,
>- Tsuyoshi
>
>On Fri, Jul 15, 2016 at 11:36 AM, Vinod Kumar Vavilapalli
><vi...@apache.org> wrote:
>> Hi, Hadoop YARN community!
>>
>> (Cross-posting across Hadoop and Slider communities)
>>
>> I opened a JIRA a little while ago to pursue a native YARN framework
>>for services: https://issues.apache.org/jira/browse/YARN-5079.
>>
>> It is part of a bigger effort that a bunch of us YARN community members
>>are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>and first-class support for services in YARN.
>>
>> The idea is that with our current attention on making services
>>first­-class, it's time to take a fresh look at how we can make Apache
>>Hadoop YARN support services well out of the box. I’ve been looking at
>>various possibilities - ranging from a custom new framework room scratch
>>to using one of the existing projects - and stopped at Apache Slider
>>(http://slider.incubator.apache.org) given its association with some of
>>the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>myself etc.).
>>
>> Slider client & AM already handles a great deal of the functionality
>>that we need. I posit that assimilating the client, ApplicationMaster
>>etc of an existing framework like Apache Slider can serve our purpose
>>really well. My early informal discussions about this with few Hadoop
>>and Slider community members yielded generally favourable feedback.
>>
>> The Apache Slider incubator community also discussed this and expressed
>>generally positive interest in YARN taking up Slider’s key pieces, you
>>can see that discussion here: https://s.apache.org/0hoh.
>>
>> So in summary, we are looking to the following
>>
>> - Code
>>    ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>Apache Hadoop for providing a native services experiences in YARN
>>    ― Leave for now some of the pieces behind in Apache Slider - (a)
>>Slider agent as we won’t need it, (b) Slider packages that need more
>>deliberation in terms of where they will live in the long term.
>>    ― Create a branch in YARN, copy this code over into a new module(s),
>>and work towards completing a functioning app running on YARN.
>>
>> - Communities & releases
>>    ― Good thing is that many of Apache Slider community members are
>>already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>without complicating things much, a proposed path forward is active
>>participation in the branch (as branch committers?) and eventually in
>>mainline YARN and thus go through a natural progression to committership
>>/ PMC. Given that most of the members are stalwarts in the Apache
>>communities, this should be a cinch IMO.
>>    ― The work on this new code can start, and depending on its state,
>>and assuming that the experiment succeeds, can be merged into trunk and
>>later picked up in the next nearest & feasible Apache Hadoop release.
>>    ― While the work on forked-over-code goes on till we have a
>>functioning app, the current Apache Slider project continues to live,
>>with supporting releases etc. At some point in the future, when things
>>become clearer, more decisions can be taken on the (parts left behind in
>>the) incubating project’s future.
>>
>> Thoughts?
>>
>> Thanks
>> +Vinod
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Tsuyoshi Ozawa <oz...@apache.org>]

+1,

From user's point of view, the high-level resource management API
without any additional dependency is much comfortable than using
low-level APIs. I believe the way should increase YARN applications
including long-running services. From developer's point of view, the
integration make YARN more stable and feature-rich since we can get
feedback about YARN low-level APIs including backward compatibility,
features we need, and so on.

BTW, I have a question about language-bindings: can we prepare python,
C++, or another language APIs over Slider? Is it out of focus of this
discussion?

Thanks,
- Tsuyoshi

On Fri, Jul 15, 2016 at 11:36 AM, Vinod Kumar Vavilapalli
<vi...@apache.org> wrote:
> Hi, Hadoop YARN community!
>
> (Cross-posting across Hadoop and Slider communities)
>
> I opened a JIRA a little while ago to pursue a native YARN framework for services: https://issues.apache.org/jira/browse/YARN-5079.
>
> It is part of a bigger effort that a bunch of us YARN community members are interested in making progress on: YARN-4692 - [Umbrella] Simplified and first-class support for services in YARN.
>
> The idea is that with our current attention on making services first­-class, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. I’ve been looking at various possibilities - ranging from a custom new framework room scratch to using one of the existing projects - and stopped at Apache Slider (http://slider.incubator.apache.org) given its association with some of the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy, myself etc.).
>
> Slider client & AM already handles a great deal of the functionality that we need. I posit that assimilating the client, ApplicationMaster etc of an existing framework like Apache Slider can serve our purpose really well. My early informal discussions about this with few Hadoop and Slider community members yielded generally favourable feedback.
>
> The Apache Slider incubator community also discussed this and expressed generally positive interest in YARN taking up Slider’s key pieces, you can see that discussion here: https://s.apache.org/0hoh.
>
> So in summary, we are looking to the following
>
> - Code
>    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into Apache Hadoop for providing a native services experiences in YARN
>    — Leave for now some of the pieces behind in Apache Slider - (a) Slider agent as we won’t need it, (b) Slider packages that need more deliberation in terms of where they will live in the long term.
>    — Create a branch in YARN, copy this code over into a new module(s), and work towards completing a functioning app running on YARN.
>
> - Communities & releases
>    — Good thing is that many of Apache Slider community members are already seasoned folk in the Apache Hadoop ecosystem projects. For those committers & PMC in Slider that are not yet Hadoop committers / PMC, without complicating things much, a proposed path forward is active participation in the branch (as branch committers?) and eventually in mainline YARN and thus go through a natural progression to committership / PMC. Given that most of the members are stalwarts in the Apache communities, this should be a cinch IMO.
>    — The work on this new code can start, and depending on its state, and assuming that the experiment succeeds, can be merged into trunk and later picked up in the next nearest & feasible Apache Hadoop release.
>    — While the work on forked-over-code goes on till we have a functioning app, the current Apache Slider project continues to live, with supporting releases etc. At some point in the future, when things become clearer, more decisions can be taken on the (parts left behind in the) incubating project’s future.
>
> Thoughts?
>
> Thanks
> +Vinod
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Gour Saha <gs...@hortonworks.com>]

+1 forwarding from the discussion in Slider DL

Note: On the Slider side, we plan to create a branch corresponding to this
YARN branch. In this, we intend to remove all the pieces which will move
to the YARN branch (Slider core, AM, client). We will retain the Agent and
app-packages which will then depend on the new hadoop-slider module (in
addition to the existing hadoop modules that it already depends on). This
will create a single view of Slider codebase, exactly as it stands today,
fully supporting the current app-packages. Slider can even release its
future versions from this new branch, once the hadoop-sldier module
reaches a matured state. It will be an easier path for existing Slider
users/app-owners to move to the future stable state of Slider completely
off of Hadoop YARN codebase. All that would be expected is to migrate the
app-packages to the agent-less version. Of course the Slider community
will do the migration for the current app-packages in the Slider codebase.

-Gour

On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vi...@apache.org> wrote:

>Hi, Hadoop YARN community!
>
>(Cross-posting across Hadoop and Slider communities)
>
>I opened a JIRA a little while ago to pursue a native YARN framework for
>services: https://issues.apache.org/jira/browse/YARN-5079.
>
>It is part of a bigger effort that a bunch of us YARN community members
>are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>and first-class support for services in YARN.
>
>The idea is that with our current attention on making services
>first­-class, it's time to take a fresh look at how we can make Apache
>Hadoop YARN support services well out of the box. I’ve been looking at
>various possibilities - ranging from a custom new framework room scratch
>to using one of the existing projects - and stopped at Apache Slider
>(http://slider.incubator.apache.org) given its association with some of
>the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>myself etc.).
>
>Slider client & AM already handles a great deal of the functionality that
>we need. I posit that assimilating the client, ApplicationMaster etc of
>an existing framework like Apache Slider can serve our purpose really
>well. My early informal discussions about this with few Hadoop and Slider
>community members yielded generally favourable feedback.
>
>The Apache Slider incubator community also discussed this and expressed
>generally positive interest in YARN taking up Slider’s key pieces, you
>can see that discussion here: https://s.apache.org/0hoh.
>
>So in summary, we are looking to the following
>
>- Code
>   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>Apache Hadoop for providing a native services experiences in YARN
>   ― Leave for now some of the pieces behind in Apache Slider - (a)
>Slider agent as we won’t need it, (b) Slider packages that need more
>deliberation in terms of where they will live in the long term.
>   ― Create a branch in YARN, copy this code over into a new module(s),
>and work towards completing a functioning app running on YARN.
>
>- Communities & releases
>   ― Good thing is that many of Apache Slider community members are
>already seasoned folk in the Apache Hadoop ecosystem projects. For those
>committers & PMC in Slider that are not yet Hadoop committers / PMC,
>without complicating things much, a proposed path forward is active
>participation in the branch (as branch committers?) and eventually in
>mainline YARN and thus go through a natural progression to committership
>/ PMC. Given that most of the members are stalwarts in the Apache
>communities, this should be a cinch IMO.
>   ― The work on this new code can start, and depending on its state, and
>assuming that the experiment succeeds, can be merged into trunk and later
>picked up in the next nearest & feasible Apache Hadoop release.
>   ― While the work on forked-over-code goes on till we have a
>functioning app, the current Apache Slider project continues to live,
>with supporting releases etc. At some point in the future, when things
>become clearer, more decisions can be taken on the (parts left behind in
>the) incubating project’s future.
>
>Thoughts?
>
>Thanks
>+Vinod

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Billie Rinaldi <bi...@gmail.com>]

+1. I think making Slider the basis of a services framework in YARN will
allow us to significantly improve YARN's support for services.

On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> Hi, Hadoop YARN community!
>
> (Cross-posting across Hadoop and Slider communities)
>
> I opened a JIRA a little while ago to pursue a native YARN framework for
> services: https://issues.apache.org/jira/browse/YARN-5079.
>
> It is part of a bigger effort that a bunch of us YARN community members
> are interested in making progress on: YARN-4692 - [Umbrella] Simplified and
> first-class support for services in YARN.
>
> The idea is that with our current attention on making services
> first­-class, it's time to take a fresh look at how we can make Apache
> Hadoop YARN support services well out of the box. I’ve been looking at
> various possibilities - ranging from a custom new framework room scratch to
> using one of the existing projects - and stopped at Apache Slider (
> http://slider.incubator.apache.org) given its association with some of
> the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> myself etc.).
>
> Slider client & AM already handles a great deal of the functionality that
> we need. I posit that assimilating the client, ApplicationMaster etc of an
> existing framework like Apache Slider can serve our purpose really well. My
> early informal discussions about this with few Hadoop and Slider community
> members yielded generally favourable feedback.
>
> The Apache Slider incubator community also discussed this and expressed
> generally positive interest in YARN taking up Slider’s key pieces, you can
> see that discussion here: https://s.apache.org/0hoh.
>
> So in summary, we are looking to the following
>
> - Code
>    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> Apache Hadoop for providing a native services experiences in YARN
>    — Leave for now some of the pieces behind in Apache Slider - (a) Slider
> agent as we won’t need it, (b) Slider packages that need more deliberation
> in terms of where they will live in the long term.
>    — Create a branch in YARN, copy this code over into a new module(s),
> and work towards completing a functioning app running on YARN.
>
> - Communities & releases
>    — Good thing is that many of Apache Slider community members are
> already seasoned folk in the Apache Hadoop ecosystem projects. For those
> committers & PMC in Slider that are not yet Hadoop committers / PMC,
> without complicating things much, a proposed path forward is active
> participation in the branch (as branch committers?) and eventually in
> mainline YARN and thus go through a natural progression to committership /
> PMC. Given that most of the members are stalwarts in the Apache
> communities, this should be a cinch IMO.
>    — The work on this new code can start, and depending on its state, and
> assuming that the experiment succeeds, can be merged into trunk and later
> picked up in the next nearest & feasible Apache Hadoop release.
>    — While the work on forked-over-code goes on till we have a functioning
> app, the current Apache Slider project continues to live, with supporting
> releases etc. At some point in the future, when things become clearer, more
> decisions can be taken on the (parts left behind in the) incubating
> project’s future.
>
> Thoughts?
>
> Thanks
> +Vinod
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

[Tsuyoshi Ozawa <oz...@apache.org>]

+1,

From user's point of view, the high-level resource management API
without any additional dependency is much comfortable than using
low-level APIs. I believe the way should increase YARN applications
including long-running services. From developer's point of view, the
integration make YARN more stable and feature-rich since we can get
feedback about YARN low-level APIs including backward compatibility,
features we need, and so on.

BTW, I have a question about language-bindings: can we prepare python,
C++, or another language APIs over Slider? Is it out of focus of this
discussion?

Thanks,
- Tsuyoshi

On Fri, Jul 15, 2016 at 11:36 AM, Vinod Kumar Vavilapalli
<vi...@apache.org> wrote:
> Hi, Hadoop YARN community!
>
> (Cross-posting across Hadoop and Slider communities)
>
> I opened a JIRA a little while ago to pursue a native YARN framework for services: https://issues.apache.org/jira/browse/YARN-5079.
>
> It is part of a bigger effort that a bunch of us YARN community members are interested in making progress on: YARN-4692 - [Umbrella] Simplified and first-class support for services in YARN.
>
> The idea is that with our current attention on making services first­-class, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. I’ve been looking at various possibilities - ranging from a custom new framework room scratch to using one of the existing projects - and stopped at Apache Slider (http://slider.incubator.apache.org) given its association with some of the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy, myself etc.).
>
> Slider client & AM already handles a great deal of the functionality that we need. I posit that assimilating the client, ApplicationMaster etc of an existing framework like Apache Slider can serve our purpose really well. My early informal discussions about this with few Hadoop and Slider community members yielded generally favourable feedback.
>
> The Apache Slider incubator community also discussed this and expressed generally positive interest in YARN taking up Slider’s key pieces, you can see that discussion here: https://s.apache.org/0hoh.
>
> So in summary, we are looking to the following
>
> - Code
>    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into Apache Hadoop for providing a native services experiences in YARN
>    — Leave for now some of the pieces behind in Apache Slider - (a) Slider agent as we won’t need it, (b) Slider packages that need more deliberation in terms of where they will live in the long term.
>    — Create a branch in YARN, copy this code over into a new module(s), and work towards completing a functioning app running on YARN.
>
> - Communities & releases
>    — Good thing is that many of Apache Slider community members are already seasoned folk in the Apache Hadoop ecosystem projects. For those committers & PMC in Slider that are not yet Hadoop committers / PMC, without complicating things much, a proposed path forward is active participation in the branch (as branch committers?) and eventually in mainline YARN and thus go through a natural progression to committership / PMC. Given that most of the members are stalwarts in the Apache communities, this should be a cinch IMO.
>    — The work on this new code can start, and depending on its state, and assuming that the experiment succeeds, can be merged into trunk and later picked up in the next nearest & feasible Apache Hadoop release.
>    — While the work on forked-over-code goes on till we have a functioning app, the current Apache Slider project continues to live, with supporting releases etc. At some point in the future, when things become clearer, more decisions can be taken on the (parts left behind in the) incubating project’s future.
>
> Thoughts?
>
> Thanks
> +Vinod
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org