You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Manoj Samel <ma...@gmail.com> on 2016/01/13 18:19:30 UTC
Slider AM restart during upgrade - can it be optional / user
controlled ?
Hi,
In recent thread about existing components taking long time to heartbeat
after upgrade; it was mentioned that "slider AM restarts itself to pick up
Yarn parameters"
If the upgrade simply consist of starting a new component, is the AM
restart still needed ? I.e. if the hadoop cluster configuration has not
changed and the Yarn parameters in the slider applications have not changed
for any components and upgrade is being used just to start a new component;
is AM restart still needed ?
The use case is for service running large number of components. When a new
user is enabled for service, a new component is started for that user.
However, no changes are done for hadoop cluster or in the slider
application configuration for any existing components. When large number of
users (i.e. components) are already running, AM restart means these
existing components have to be informed of new AM and vice versa.
In such cases, if AM restart is not needed, can it be avoided e.g. by
having a new option in Upgrade command ?
Thanks,
Manoj
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Manoj Samel <ma...@gmail.com>.
Improvement jira filed - https://issues.apache.org/jira/browse/SLIDER-1066
Thanks all for your time and insights.
Manoj
On Mon, Jan 18, 2016 at 6:56 PM, Gour Saha <gs...@hortonworks.com> wrote:
> Yes please file a jira. Let’s discuss it there further.
>
> On 1/18/16, 3:57 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>
> >Point take.
> >
> >For now, we have chosen to go with single app. And exactly to your point -
> >we don't want the AM to restart when we know that there is no change to be
> >picked up so restarting in this case has more downside (AM issue etc.) -
> >hence the feature ask.
> >
> >Any pros/conn on making this possible ? Should I file a feature jira ?
> >
> >Thanks
> >
> >On Mon, Jan 18, 2016 at 3:05 PM, Gour Saha <gs...@hortonworks.com> wrote:
> >
> >> Inline.
> >>
> >> On 1/14/16, 4:52 PM, "Manoj Samel" <ma...@gmail.com> wrote:
> >>
> >> >Hi Gour,
> >> >
> >> >The use case is described in
> >> >https://issues.apache.org/jira/browse/SLIDER-1046
> >> >
> >> >Regarding your latest proposals -
> >> >
> >> >1. Can a "slider update" be used - As I recall, slider update is only
> >>when
> >> >application is stopped. We don't stop the application when adding new
> >>user
> >>
> >> [Gour] good point. I forgot that.
> >> >
> >> >2. Can a new slider app be used for each user - Hummm .. this is
> >> >interesting one ;^). After some thoughts, I see couple of issues
> >> >
> >> >Issue 1 - With large number of users, we have equally large number of
> >>AMs.
> >> >Even if AM footprint is small, it could grow in future and # of AM
> >>will be
> >> >a issue in any case.
> >> >
> >> >Issue 2 - Large number of applications will become issue. E.g. when
> >>doing
> >> >application upgrade, instead of doing one AM upgrade followed by
> >>rolling
> >> >upgrade of users, it will be larger number of AM upgrades. I think
> >>this is
> >> >less manageable. Also other operations like shutting down the app (in
> >> >event
> >> >of cluster downtime etc.) will be getting cumbersome
> >> >
> >> >Single app with different components per user seems still the best
> >>choice
> >> >...
> >>
> >> [Gour] Makes sense, but keep in mind that a single app is also subject
> >>to
> >> a single point of failure for all users (say in the unforeseen case of
> >>an
> >> issue during Slider AM or application upgrade). Note, multiple
> >> applications upgrade will be performed by a script or orchestrator, just
> >> like you would have to do to the equal number of components for a single
> >> app as well.
> >> >
> >> >Thanks,
> >> >
> >> >Manoj
> >> >
> >> >On Wed, Jan 13, 2016 at 2:08 PM, Gour Saha <gs...@hortonworks.com>
> >>wrote:
> >> >
> >> >> Can you look into ³slider update² (not upgrade) and then subsequently
> >> >> followed by appropriate ³slider flex² to bring up components for new
> >> >>users?
> >> >>
> >> >> It might be helpful if you can provide a brief explanation of your
> >> >> usecase. Just trying to see if mapping a new user to a new slider app
> >> >> (with a single component definition) makes more sense, compared to
> >>your
> >> >> current approach. The overhead of a Slider AM is very low.
> >> >>
> >> >> -Gour
> >> >>
> >> >> On 1/13/16, 10:35 AM, "Manoj Samel" <ma...@gmail.com>
> wrote:
> >> >>
> >> >> >Flex isn't option for couple of reasons in above use case
> >> >> >
> >> >> >1. For each new user, different parameters need to be passed when
> >> >> >starting.
> >> >> >New component allows control over component name and parameters
> >> >> >2. More importantly, when user is 'decomissioned', the component for
> >> >>that
> >> >> >user should be stopped for good. This is achieved at present by
> >>naming
> >> >> >each
> >> >> >component after respective user and stopping that when user is gone.
> >> >> >
> >> >> >Thanks,
> >> >> >
> >> >> >Manoj
> >> >> >
> >> >> >On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran
> >> >><st...@hortonworks.com>
> >> >> >wrote:
> >> >> >
> >> >> >> you should just be flexing here
> >> >> >>
> >> >> >> > On 13 Jan 2016, at 09:19, Manoj Samel <manojsameltech@gmail.com
> >
> >> >> >>wrote:
> >> >> >> >
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > In recent thread about existing components taking long time to
> >> >> >>heartbeat
> >> >> >> > after upgrade; it was mentioned that "slider AM restarts itself
> >>to
> >> >> >>pick
> >> >> >> up
> >> >> >> > Yarn parameters"
> >> >> >> >
> >> >> >> > If the upgrade simply consist of starting a new component, is
> >>the
> >> >>AM
> >> >> >> > restart still needed ? I.e. if the hadoop cluster configuration
> >>has
> >> >> >>not
> >> >> >> > changed and the Yarn parameters in the slider applications have
> >>not
> >> >> >> changed
> >> >> >> > for any components and upgrade is being used just to start a new
> >> >> >> component;
> >> >> >> > is AM restart still needed ?
> >> >> >> >
> >> >> >> > The use case is for service running large number of components.
> >> >>When a
> >> >> >> new
> >> >> >> > user is enabled for service, a new component is started for that
> >> >>user.
> >> >> >> > However, no changes are done for hadoop cluster or in the slider
> >> >> >> > application configuration for any existing components. When
> >>large
> >> >> >>number
> >> >> >> of
> >> >> >> > users (i.e. components) are already running, AM restart means
> >>these
> >> >> >> > existing components have to be informed of new AM and vice
> >>versa.
> >> >> >> >
> >> >> >> > In such cases, if AM restart is not needed, can it be avoided
> >>e.g.
> >> >>by
> >> >> >> > having a new option in Upgrade command ?
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> >
> >> >> >> > Manoj
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
>
>
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Gour Saha <gs...@hortonworks.com>.
Yes please file a jira. Let’s discuss it there further.
On 1/18/16, 3:57 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>Point take.
>
>For now, we have chosen to go with single app. And exactly to your point -
>we don't want the AM to restart when we know that there is no change to be
>picked up so restarting in this case has more downside (AM issue etc.) -
>hence the feature ask.
>
>Any pros/conn on making this possible ? Should I file a feature jira ?
>
>Thanks
>
>On Mon, Jan 18, 2016 at 3:05 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
>> Inline.
>>
>> On 1/14/16, 4:52 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>>
>> >Hi Gour,
>> >
>> >The use case is described in
>> >https://issues.apache.org/jira/browse/SLIDER-1046
>> >
>> >Regarding your latest proposals -
>> >
>> >1. Can a "slider update" be used - As I recall, slider update is only
>>when
>> >application is stopped. We don't stop the application when adding new
>>user
>>
>> [Gour] good point. I forgot that.
>> >
>> >2. Can a new slider app be used for each user - Hummm .. this is
>> >interesting one ;^). After some thoughts, I see couple of issues
>> >
>> >Issue 1 - With large number of users, we have equally large number of
>>AMs.
>> >Even if AM footprint is small, it could grow in future and # of AM
>>will be
>> >a issue in any case.
>> >
>> >Issue 2 - Large number of applications will become issue. E.g. when
>>doing
>> >application upgrade, instead of doing one AM upgrade followed by
>>rolling
>> >upgrade of users, it will be larger number of AM upgrades. I think
>>this is
>> >less manageable. Also other operations like shutting down the app (in
>> >event
>> >of cluster downtime etc.) will be getting cumbersome
>> >
>> >Single app with different components per user seems still the best
>>choice
>> >...
>>
>> [Gour] Makes sense, but keep in mind that a single app is also subject
>>to
>> a single point of failure for all users (say in the unforeseen case of
>>an
>> issue during Slider AM or application upgrade). Note, multiple
>> applications upgrade will be performed by a script or orchestrator, just
>> like you would have to do to the equal number of components for a single
>> app as well.
>> >
>> >Thanks,
>> >
>> >Manoj
>> >
>> >On Wed, Jan 13, 2016 at 2:08 PM, Gour Saha <gs...@hortonworks.com>
>>wrote:
>> >
>> >> Can you look into ³slider update² (not upgrade) and then subsequently
>> >> followed by appropriate ³slider flex² to bring up components for new
>> >>users?
>> >>
>> >> It might be helpful if you can provide a brief explanation of your
>> >> usecase. Just trying to see if mapping a new user to a new slider app
>> >> (with a single component definition) makes more sense, compared to
>>your
>> >> current approach. The overhead of a Slider AM is very low.
>> >>
>> >> -Gour
>> >>
>> >> On 1/13/16, 10:35 AM, "Manoj Samel" <ma...@gmail.com> wrote:
>> >>
>> >> >Flex isn't option for couple of reasons in above use case
>> >> >
>> >> >1. For each new user, different parameters need to be passed when
>> >> >starting.
>> >> >New component allows control over component name and parameters
>> >> >2. More importantly, when user is 'decomissioned', the component for
>> >>that
>> >> >user should be stopped for good. This is achieved at present by
>>naming
>> >> >each
>> >> >component after respective user and stopping that when user is gone.
>> >> >
>> >> >Thanks,
>> >> >
>> >> >Manoj
>> >> >
>> >> >On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran
>> >><st...@hortonworks.com>
>> >> >wrote:
>> >> >
>> >> >> you should just be flexing here
>> >> >>
>> >> >> > On 13 Jan 2016, at 09:19, Manoj Samel <ma...@gmail.com>
>> >> >>wrote:
>> >> >> >
>> >> >> > Hi,
>> >> >> >
>> >> >> > In recent thread about existing components taking long time to
>> >> >>heartbeat
>> >> >> > after upgrade; it was mentioned that "slider AM restarts itself
>>to
>> >> >>pick
>> >> >> up
>> >> >> > Yarn parameters"
>> >> >> >
>> >> >> > If the upgrade simply consist of starting a new component, is
>>the
>> >>AM
>> >> >> > restart still needed ? I.e. if the hadoop cluster configuration
>>has
>> >> >>not
>> >> >> > changed and the Yarn parameters in the slider applications have
>>not
>> >> >> changed
>> >> >> > for any components and upgrade is being used just to start a new
>> >> >> component;
>> >> >> > is AM restart still needed ?
>> >> >> >
>> >> >> > The use case is for service running large number of components.
>> >>When a
>> >> >> new
>> >> >> > user is enabled for service, a new component is started for that
>> >>user.
>> >> >> > However, no changes are done for hadoop cluster or in the slider
>> >> >> > application configuration for any existing components. When
>>large
>> >> >>number
>> >> >> of
>> >> >> > users (i.e. components) are already running, AM restart means
>>these
>> >> >> > existing components have to be informed of new AM and vice
>>versa.
>> >> >> >
>> >> >> > In such cases, if AM restart is not needed, can it be avoided
>>e.g.
>> >>by
>> >> >> > having a new option in Upgrade command ?
>> >> >> >
>> >> >> > Thanks,
>> >> >> >
>> >> >> > Manoj
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Manoj Samel <ma...@gmail.com>.
Point take.
For now, we have chosen to go with single app. And exactly to your point -
we don't want the AM to restart when we know that there is no change to be
picked up so restarting in this case has more downside (AM issue etc.) -
hence the feature ask.
Any pros/conn on making this possible ? Should I file a feature jira ?
Thanks
On Mon, Jan 18, 2016 at 3:05 PM, Gour Saha <gs...@hortonworks.com> wrote:
> Inline.
>
> On 1/14/16, 4:52 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>
> >Hi Gour,
> >
> >The use case is described in
> >https://issues.apache.org/jira/browse/SLIDER-1046
> >
> >Regarding your latest proposals -
> >
> >1. Can a "slider update" be used - As I recall, slider update is only when
> >application is stopped. We don't stop the application when adding new user
>
> [Gour] good point. I forgot that.
> >
> >2. Can a new slider app be used for each user - Hummm .. this is
> >interesting one ;^). After some thoughts, I see couple of issues
> >
> >Issue 1 - With large number of users, we have equally large number of AMs.
> >Even if AM footprint is small, it could grow in future and # of AM will be
> >a issue in any case.
> >
> >Issue 2 - Large number of applications will become issue. E.g. when doing
> >application upgrade, instead of doing one AM upgrade followed by rolling
> >upgrade of users, it will be larger number of AM upgrades. I think this is
> >less manageable. Also other operations like shutting down the app (in
> >event
> >of cluster downtime etc.) will be getting cumbersome
> >
> >Single app with different components per user seems still the best choice
> >...
>
> [Gour] Makes sense, but keep in mind that a single app is also subject to
> a single point of failure for all users (say in the unforeseen case of an
> issue during Slider AM or application upgrade). Note, multiple
> applications upgrade will be performed by a script or orchestrator, just
> like you would have to do to the equal number of components for a single
> app as well.
> >
> >Thanks,
> >
> >Manoj
> >
> >On Wed, Jan 13, 2016 at 2:08 PM, Gour Saha <gs...@hortonworks.com> wrote:
> >
> >> Can you look into ³slider update² (not upgrade) and then subsequently
> >> followed by appropriate ³slider flex² to bring up components for new
> >>users?
> >>
> >> It might be helpful if you can provide a brief explanation of your
> >> usecase. Just trying to see if mapping a new user to a new slider app
> >> (with a single component definition) makes more sense, compared to your
> >> current approach. The overhead of a Slider AM is very low.
> >>
> >> -Gour
> >>
> >> On 1/13/16, 10:35 AM, "Manoj Samel" <ma...@gmail.com> wrote:
> >>
> >> >Flex isn't option for couple of reasons in above use case
> >> >
> >> >1. For each new user, different parameters need to be passed when
> >> >starting.
> >> >New component allows control over component name and parameters
> >> >2. More importantly, when user is 'decomissioned', the component for
> >>that
> >> >user should be stopped for good. This is achieved at present by naming
> >> >each
> >> >component after respective user and stopping that when user is gone.
> >> >
> >> >Thanks,
> >> >
> >> >Manoj
> >> >
> >> >On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran
> >><st...@hortonworks.com>
> >> >wrote:
> >> >
> >> >> you should just be flexing here
> >> >>
> >> >> > On 13 Jan 2016, at 09:19, Manoj Samel <ma...@gmail.com>
> >> >>wrote:
> >> >> >
> >> >> > Hi,
> >> >> >
> >> >> > In recent thread about existing components taking long time to
> >> >>heartbeat
> >> >> > after upgrade; it was mentioned that "slider AM restarts itself to
> >> >>pick
> >> >> up
> >> >> > Yarn parameters"
> >> >> >
> >> >> > If the upgrade simply consist of starting a new component, is the
> >>AM
> >> >> > restart still needed ? I.e. if the hadoop cluster configuration has
> >> >>not
> >> >> > changed and the Yarn parameters in the slider applications have not
> >> >> changed
> >> >> > for any components and upgrade is being used just to start a new
> >> >> component;
> >> >> > is AM restart still needed ?
> >> >> >
> >> >> > The use case is for service running large number of components.
> >>When a
> >> >> new
> >> >> > user is enabled for service, a new component is started for that
> >>user.
> >> >> > However, no changes are done for hadoop cluster or in the slider
> >> >> > application configuration for any existing components. When large
> >> >>number
> >> >> of
> >> >> > users (i.e. components) are already running, AM restart means these
> >> >> > existing components have to be informed of new AM and vice versa.
> >> >> >
> >> >> > In such cases, if AM restart is not needed, can it be avoided e.g.
> >>by
> >> >> > having a new option in Upgrade command ?
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > Manoj
> >> >>
> >> >>
> >>
> >>
>
>
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Gour Saha <gs...@hortonworks.com>.
Inline.
On 1/14/16, 4:52 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>Hi Gour,
>
>The use case is described in
>https://issues.apache.org/jira/browse/SLIDER-1046
>
>Regarding your latest proposals -
>
>1. Can a "slider update" be used - As I recall, slider update is only when
>application is stopped. We don't stop the application when adding new user
[Gour] good point. I forgot that.
>
>2. Can a new slider app be used for each user - Hummm .. this is
>interesting one ;^). After some thoughts, I see couple of issues
>
>Issue 1 - With large number of users, we have equally large number of AMs.
>Even if AM footprint is small, it could grow in future and # of AM will be
>a issue in any case.
>
>Issue 2 - Large number of applications will become issue. E.g. when doing
>application upgrade, instead of doing one AM upgrade followed by rolling
>upgrade of users, it will be larger number of AM upgrades. I think this is
>less manageable. Also other operations like shutting down the app (in
>event
>of cluster downtime etc.) will be getting cumbersome
>
>Single app with different components per user seems still the best choice
>...
[Gour] Makes sense, but keep in mind that a single app is also subject to
a single point of failure for all users (say in the unforeseen case of an
issue during Slider AM or application upgrade). Note, multiple
applications upgrade will be performed by a script or orchestrator, just
like you would have to do to the equal number of components for a single
app as well.
>
>Thanks,
>
>Manoj
>
>On Wed, Jan 13, 2016 at 2:08 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
>> Can you look into ³slider update² (not upgrade) and then subsequently
>> followed by appropriate ³slider flex² to bring up components for new
>>users?
>>
>> It might be helpful if you can provide a brief explanation of your
>> usecase. Just trying to see if mapping a new user to a new slider app
>> (with a single component definition) makes more sense, compared to your
>> current approach. The overhead of a Slider AM is very low.
>>
>> -Gour
>>
>> On 1/13/16, 10:35 AM, "Manoj Samel" <ma...@gmail.com> wrote:
>>
>> >Flex isn't option for couple of reasons in above use case
>> >
>> >1. For each new user, different parameters need to be passed when
>> >starting.
>> >New component allows control over component name and parameters
>> >2. More importantly, when user is 'decomissioned', the component for
>>that
>> >user should be stopped for good. This is achieved at present by naming
>> >each
>> >component after respective user and stopping that when user is gone.
>> >
>> >Thanks,
>> >
>> >Manoj
>> >
>> >On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran
>><st...@hortonworks.com>
>> >wrote:
>> >
>> >> you should just be flexing here
>> >>
>> >> > On 13 Jan 2016, at 09:19, Manoj Samel <ma...@gmail.com>
>> >>wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > In recent thread about existing components taking long time to
>> >>heartbeat
>> >> > after upgrade; it was mentioned that "slider AM restarts itself to
>> >>pick
>> >> up
>> >> > Yarn parameters"
>> >> >
>> >> > If the upgrade simply consist of starting a new component, is the
>>AM
>> >> > restart still needed ? I.e. if the hadoop cluster configuration has
>> >>not
>> >> > changed and the Yarn parameters in the slider applications have not
>> >> changed
>> >> > for any components and upgrade is being used just to start a new
>> >> component;
>> >> > is AM restart still needed ?
>> >> >
>> >> > The use case is for service running large number of components.
>>When a
>> >> new
>> >> > user is enabled for service, a new component is started for that
>>user.
>> >> > However, no changes are done for hadoop cluster or in the slider
>> >> > application configuration for any existing components. When large
>> >>number
>> >> of
>> >> > users (i.e. components) are already running, AM restart means these
>> >> > existing components have to be informed of new AM and vice versa.
>> >> >
>> >> > In such cases, if AM restart is not needed, can it be avoided e.g.
>>by
>> >> > having a new option in Upgrade command ?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Manoj
>> >>
>> >>
>>
>>
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Manoj Samel <ma...@gmail.com>.
Hi Gour,
The use case is described in
https://issues.apache.org/jira/browse/SLIDER-1046
Regarding your latest proposals -
1. Can a "slider update" be used - As I recall, slider update is only when
application is stopped. We don't stop the application when adding new user
2. Can a new slider app be used for each user - Hummm .. this is
interesting one ;^). After some thoughts, I see couple of issues
Issue 1 - With large number of users, we have equally large number of AMs.
Even if AM footprint is small, it could grow in future and # of AM will be
a issue in any case.
Issue 2 - Large number of applications will become issue. E.g. when doing
application upgrade, instead of doing one AM upgrade followed by rolling
upgrade of users, it will be larger number of AM upgrades. I think this is
less manageable. Also other operations like shutting down the app (in event
of cluster downtime etc.) will be getting cumbersome
Single app with different components per user seems still the best choice
...
Thanks,
Manoj
On Wed, Jan 13, 2016 at 2:08 PM, Gour Saha <gs...@hortonworks.com> wrote:
> Can you look into ³slider update² (not upgrade) and then subsequently
> followed by appropriate ³slider flex² to bring up components for new users?
>
> It might be helpful if you can provide a brief explanation of your
> usecase. Just trying to see if mapping a new user to a new slider app
> (with a single component definition) makes more sense, compared to your
> current approach. The overhead of a Slider AM is very low.
>
> -Gour
>
> On 1/13/16, 10:35 AM, "Manoj Samel" <ma...@gmail.com> wrote:
>
> >Flex isn't option for couple of reasons in above use case
> >
> >1. For each new user, different parameters need to be passed when
> >starting.
> >New component allows control over component name and parameters
> >2. More importantly, when user is 'decomissioned', the component for that
> >user should be stopped for good. This is achieved at present by naming
> >each
> >component after respective user and stopping that when user is gone.
> >
> >Thanks,
> >
> >Manoj
> >
> >On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran <st...@hortonworks.com>
> >wrote:
> >
> >> you should just be flexing here
> >>
> >> > On 13 Jan 2016, at 09:19, Manoj Samel <ma...@gmail.com>
> >>wrote:
> >> >
> >> > Hi,
> >> >
> >> > In recent thread about existing components taking long time to
> >>heartbeat
> >> > after upgrade; it was mentioned that "slider AM restarts itself to
> >>pick
> >> up
> >> > Yarn parameters"
> >> >
> >> > If the upgrade simply consist of starting a new component, is the AM
> >> > restart still needed ? I.e. if the hadoop cluster configuration has
> >>not
> >> > changed and the Yarn parameters in the slider applications have not
> >> changed
> >> > for any components and upgrade is being used just to start a new
> >> component;
> >> > is AM restart still needed ?
> >> >
> >> > The use case is for service running large number of components. When a
> >> new
> >> > user is enabled for service, a new component is started for that user.
> >> > However, no changes are done for hadoop cluster or in the slider
> >> > application configuration for any existing components. When large
> >>number
> >> of
> >> > users (i.e. components) are already running, AM restart means these
> >> > existing components have to be informed of new AM and vice versa.
> >> >
> >> > In such cases, if AM restart is not needed, can it be avoided e.g. by
> >> > having a new option in Upgrade command ?
> >> >
> >> > Thanks,
> >> >
> >> > Manoj
> >>
> >>
>
>
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Gour Saha <gs...@hortonworks.com>.
Can you look into ³slider update² (not upgrade) and then subsequently
followed by appropriate ³slider flex² to bring up components for new users?
It might be helpful if you can provide a brief explanation of your
usecase. Just trying to see if mapping a new user to a new slider app
(with a single component definition) makes more sense, compared to your
current approach. The overhead of a Slider AM is very low.
-Gour
On 1/13/16, 10:35 AM, "Manoj Samel" <ma...@gmail.com> wrote:
>Flex isn't option for couple of reasons in above use case
>
>1. For each new user, different parameters need to be passed when
>starting.
>New component allows control over component name and parameters
>2. More importantly, when user is 'decomissioned', the component for that
>user should be stopped for good. This is achieved at present by naming
>each
>component after respective user and stopping that when user is gone.
>
>Thanks,
>
>Manoj
>
>On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran <st...@hortonworks.com>
>wrote:
>
>> you should just be flexing here
>>
>> > On 13 Jan 2016, at 09:19, Manoj Samel <ma...@gmail.com>
>>wrote:
>> >
>> > Hi,
>> >
>> > In recent thread about existing components taking long time to
>>heartbeat
>> > after upgrade; it was mentioned that "slider AM restarts itself to
>>pick
>> up
>> > Yarn parameters"
>> >
>> > If the upgrade simply consist of starting a new component, is the AM
>> > restart still needed ? I.e. if the hadoop cluster configuration has
>>not
>> > changed and the Yarn parameters in the slider applications have not
>> changed
>> > for any components and upgrade is being used just to start a new
>> component;
>> > is AM restart still needed ?
>> >
>> > The use case is for service running large number of components. When a
>> new
>> > user is enabled for service, a new component is started for that user.
>> > However, no changes are done for hadoop cluster or in the slider
>> > application configuration for any existing components. When large
>>number
>> of
>> > users (i.e. components) are already running, AM restart means these
>> > existing components have to be informed of new AM and vice versa.
>> >
>> > In such cases, if AM restart is not needed, can it be avoided e.g. by
>> > having a new option in Upgrade command ?
>> >
>> > Thanks,
>> >
>> > Manoj
>>
>>
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Manoj Samel <ma...@gmail.com>.
Flex isn't option for couple of reasons in above use case
1. For each new user, different parameters need to be passed when starting.
New component allows control over component name and parameters
2. More importantly, when user is 'decomissioned', the component for that
user should be stopped for good. This is achieved at present by naming each
component after respective user and stopping that when user is gone.
Thanks,
Manoj
On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> you should just be flexing here
>
> > On 13 Jan 2016, at 09:19, Manoj Samel <ma...@gmail.com> wrote:
> >
> > Hi,
> >
> > In recent thread about existing components taking long time to heartbeat
> > after upgrade; it was mentioned that "slider AM restarts itself to pick
> up
> > Yarn parameters"
> >
> > If the upgrade simply consist of starting a new component, is the AM
> > restart still needed ? I.e. if the hadoop cluster configuration has not
> > changed and the Yarn parameters in the slider applications have not
> changed
> > for any components and upgrade is being used just to start a new
> component;
> > is AM restart still needed ?
> >
> > The use case is for service running large number of components. When a
> new
> > user is enabled for service, a new component is started for that user.
> > However, no changes are done for hadoop cluster or in the slider
> > application configuration for any existing components. When large number
> of
> > users (i.e. components) are already running, AM restart means these
> > existing components have to be informed of new AM and vice versa.
> >
> > In such cases, if AM restart is not needed, can it be avoided e.g. by
> > having a new option in Upgrade command ?
> >
> > Thanks,
> >
> > Manoj
>
>
Re: Slider AM restart during upgrade - can it be optional / user
controlled ?
Posted by Steve Loughran <st...@hortonworks.com>.
you should just be flexing here
> On 13 Jan 2016, at 09:19, Manoj Samel <ma...@gmail.com> wrote:
>
> Hi,
>
> In recent thread about existing components taking long time to heartbeat
> after upgrade; it was mentioned that "slider AM restarts itself to pick up
> Yarn parameters"
>
> If the upgrade simply consist of starting a new component, is the AM
> restart still needed ? I.e. if the hadoop cluster configuration has not
> changed and the Yarn parameters in the slider applications have not changed
> for any components and upgrade is being used just to start a new component;
> is AM restart still needed ?
>
> The use case is for service running large number of components. When a new
> user is enabled for service, a new component is started for that user.
> However, no changes are done for hadoop cluster or in the slider
> application configuration for any existing components. When large number of
> users (i.e. components) are already running, AM restart means these
> existing components have to be informed of new AM and vice versa.
>
> In such cases, if AM restart is not needed, can it be avoided e.g. by
> having a new option in Upgrade command ?
>
> Thanks,
>
> Manoj