You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Jiwon Seo <ji...@stanford.edu> on 2015/07/24 20:24:07 UTC
A quick question on dynamic resizing in Slider
Hi,
I'm working on a research system for in-memory graph processing (called
SociaLite).
I'm curious if Slider supports dynamic resizing, requested by applications.
That is, if the application detects memory pressure, the it asks Slider for
more resources (e.g. nodes with XXX mb memory); then Slider will tell the
application how many nodes are allocated/assigned to the app.
Does Slider have this kind of application interface? (if so, any link to
the doc?) Or for that functionality, should I directly use Yarn API?
Thanks,
-Jiwon
Re: A quick question on dynamic resizing in Slider
Posted by Steve Loughran <st...@hortonworks.com>.
The REST API doesn't work; we need YARN to proxy the requests (YARN-2031).
The IPC API does support flexing -the caller submits a new resources.json file which is received by the AM, saved to HDFS (so that on a restart or stop/start the values preserved), then acted on. The AM reviews the #of containers and adds/removes them.
We don't explicitly have a way for the running application to change the values. this is actually something the storm team would like. Again, the IPC channel is there, it's just currently set up on the expectation that the user is tuning the cluster size.
One thing we have avoided doing is any attempt to dynamically flex cluster size based on load, the way Apache Helix does. It's hard to decide when to scale up or down, and in a multi-component system, the scaling is often across the entire system, rather than just one layer. We've concluded that (currently) it's best to publish load information and let other tools make those decisions.
-steve
> On 25 Jul 2015, at 07:12, Yong Feng <fe...@gmail.com> wrote:
>
> Regarding to Slider API, please refer to
> https://github.com/apache/incubator-slider/tree/develop/slider-core/src/main/java/org/apache/slider/client
> .
>
> Ambari has Slider View to deploy and manage Slider-based application. You
> could refer to source codes of Ambari Slider View as an example how to call
> Slider API as follows
>
> https://github.com/apache/ambari/blob/trunk/contrib/views/slider/src/main/java/org/apache/ambari/view/slider/SliderAppsViewControllerImpl.java
>
> It includes scale-out/in an existing Slider Application.
>
> BTW, the following doc describes REST API of Slider. However I never try
> REST API directly but SliderClient class as I mentioned above.
>
> http://slider.incubator.apache.org/docs/api/slider_REST_api_v2.html
>
> Thanks,
>
> Yong
>
>
>
> On Fri, Jul 24, 2015 at 5:35 PM, Jiwon Seo <ji...@stanford.edu> wrote:
>
>> Thank you for your reply.
>>
>> As far as I understand, "slider flex ... " in the link
>> http://slider.incubator.apache.org/docs/manpage.html performs scaling out
>> upon the execution of the command. My questions is, does Slider have a
>> programmable interface for scaling out? (and I'm not talking about
>> automatic scaling, but about the functionality for an application to
>> request more machines -- or memory/cpus from Slider).
>>
>> Does Slider have such interface (API)?
>>
>> Thanks!
>>
>> -Jiwon
>>
>>
>> On Fri, Jul 24, 2015 at 1:51 PM, Yong Feng <fe...@gmail.com> wrote:
>>
>>> Hi Jiwon,
>>>
>>> Slider supports horizontally scale out/in your application. Please refer
>> to
>>> following link for the user doc of slider commands which describes
>> "slider
>>> flex ..."
>>>
>>> http://slider.incubator.apache.org/docs/manpage.html
>>>
>>> Besides, YARN 1197 plans to support resizing YARN container. By
>> leveraging
>>> it, Slider is able to support vertically scale up/down your application
>>> instance, for example increasing memory from 2GB to 4GB.
>>>
>>> https://issues.apache.org/jira/browse/YARN-1197
>>>
>>> Based on slider's capabilities of scale out/in and scale up/down, you
>> could
>>> develop a driver to automatically flex your application. As I know there
>> is
>>> a team who is working on an open source general auto-scaler for
>> YARN/Slider
>>> based application which is named as Jaguar. You may contact them for
>>> details.
>>>
>>>
>>>
>> https://hadoopsummit.uservoice.com/forums/283262-hadoop-governance-security-operations/suggestions/7074316-project-jaguar-an-application-autoscaler-for-hado
>>>
>>> Please feel free to let me know if you have any question.
>>>
>>> Thanks,
>>>
>>> Yong
>>>
>>>
>>> On Fri, Jul 24, 2015 at 2:24 PM, Jiwon Seo <ji...@stanford.edu> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm working on a research system for in-memory graph processing (called
>>>> SociaLite).
>>>>
>>>> I'm curious if Slider supports dynamic resizing, requested by
>>> applications.
>>>> That is, if the application detects memory pressure, the it asks Slider
>>> for
>>>> more resources (e.g. nodes with XXX mb memory); then Slider will tell
>> the
>>>> application how many nodes are allocated/assigned to the app.
>>>>
>>>> Does Slider have this kind of application interface? (if so, any link
>> to
>>>> the doc?) Or for that functionality, should I directly use Yarn API?
>>>>
>>>> Thanks,
>>>>
>>>> -Jiwon
>>>>
>>>
>>
Re: A quick question on dynamic resizing in Slider
Posted by Yong Feng <fe...@gmail.com>.
Regarding to Slider API, please refer to
https://github.com/apache/incubator-slider/tree/develop/slider-core/src/main/java/org/apache/slider/client
.
Ambari has Slider View to deploy and manage Slider-based application. You
could refer to source codes of Ambari Slider View as an example how to call
Slider API as follows
https://github.com/apache/ambari/blob/trunk/contrib/views/slider/src/main/java/org/apache/ambari/view/slider/SliderAppsViewControllerImpl.java
It includes scale-out/in an existing Slider Application.
BTW, the following doc describes REST API of Slider. However I never try
REST API directly but SliderClient class as I mentioned above.
http://slider.incubator.apache.org/docs/api/slider_REST_api_v2.html
Thanks,
Yong
On Fri, Jul 24, 2015 at 5:35 PM, Jiwon Seo <ji...@stanford.edu> wrote:
> Thank you for your reply.
>
> As far as I understand, "slider flex ... " in the link
> http://slider.incubator.apache.org/docs/manpage.html performs scaling out
> upon the execution of the command. My questions is, does Slider have a
> programmable interface for scaling out? (and I'm not talking about
> automatic scaling, but about the functionality for an application to
> request more machines -- or memory/cpus from Slider).
>
> Does Slider have such interface (API)?
>
> Thanks!
>
> -Jiwon
>
>
> On Fri, Jul 24, 2015 at 1:51 PM, Yong Feng <fe...@gmail.com> wrote:
>
> > Hi Jiwon,
> >
> > Slider supports horizontally scale out/in your application. Please refer
> to
> > following link for the user doc of slider commands which describes
> "slider
> > flex ..."
> >
> > http://slider.incubator.apache.org/docs/manpage.html
> >
> > Besides, YARN 1197 plans to support resizing YARN container. By
> leveraging
> > it, Slider is able to support vertically scale up/down your application
> > instance, for example increasing memory from 2GB to 4GB.
> >
> > https://issues.apache.org/jira/browse/YARN-1197
> >
> > Based on slider's capabilities of scale out/in and scale up/down, you
> could
> > develop a driver to automatically flex your application. As I know there
> is
> > a team who is working on an open source general auto-scaler for
> YARN/Slider
> > based application which is named as Jaguar. You may contact them for
> > details.
> >
> >
> >
> https://hadoopsummit.uservoice.com/forums/283262-hadoop-governance-security-operations/suggestions/7074316-project-jaguar-an-application-autoscaler-for-hado
> >
> > Please feel free to let me know if you have any question.
> >
> > Thanks,
> >
> > Yong
> >
> >
> > On Fri, Jul 24, 2015 at 2:24 PM, Jiwon Seo <ji...@stanford.edu> wrote:
> >
> > > Hi,
> > >
> > > I'm working on a research system for in-memory graph processing (called
> > > SociaLite).
> > >
> > > I'm curious if Slider supports dynamic resizing, requested by
> > applications.
> > > That is, if the application detects memory pressure, the it asks Slider
> > for
> > > more resources (e.g. nodes with XXX mb memory); then Slider will tell
> the
> > > application how many nodes are allocated/assigned to the app.
> > >
> > > Does Slider have this kind of application interface? (if so, any link
> to
> > > the doc?) Or for that functionality, should I directly use Yarn API?
> > >
> > > Thanks,
> > >
> > > -Jiwon
> > >
> >
>
Re: A quick question on dynamic resizing in Slider
Posted by Jiwon Seo <ji...@stanford.edu>.
Thank you for your reply.
As far as I understand, "slider flex ... " in the link
http://slider.incubator.apache.org/docs/manpage.html performs scaling out
upon the execution of the command. My questions is, does Slider have a
programmable interface for scaling out? (and I'm not talking about
automatic scaling, but about the functionality for an application to
request more machines -- or memory/cpus from Slider).
Does Slider have such interface (API)?
Thanks!
-Jiwon
On Fri, Jul 24, 2015 at 1:51 PM, Yong Feng <fe...@gmail.com> wrote:
> Hi Jiwon,
>
> Slider supports horizontally scale out/in your application. Please refer to
> following link for the user doc of slider commands which describes "slider
> flex ..."
>
> http://slider.incubator.apache.org/docs/manpage.html
>
> Besides, YARN 1197 plans to support resizing YARN container. By leveraging
> it, Slider is able to support vertically scale up/down your application
> instance, for example increasing memory from 2GB to 4GB.
>
> https://issues.apache.org/jira/browse/YARN-1197
>
> Based on slider's capabilities of scale out/in and scale up/down, you could
> develop a driver to automatically flex your application. As I know there is
> a team who is working on an open source general auto-scaler for YARN/Slider
> based application which is named as Jaguar. You may contact them for
> details.
>
>
> https://hadoopsummit.uservoice.com/forums/283262-hadoop-governance-security-operations/suggestions/7074316-project-jaguar-an-application-autoscaler-for-hado
>
> Please feel free to let me know if you have any question.
>
> Thanks,
>
> Yong
>
>
> On Fri, Jul 24, 2015 at 2:24 PM, Jiwon Seo <ji...@stanford.edu> wrote:
>
> > Hi,
> >
> > I'm working on a research system for in-memory graph processing (called
> > SociaLite).
> >
> > I'm curious if Slider supports dynamic resizing, requested by
> applications.
> > That is, if the application detects memory pressure, the it asks Slider
> for
> > more resources (e.g. nodes with XXX mb memory); then Slider will tell the
> > application how many nodes are allocated/assigned to the app.
> >
> > Does Slider have this kind of application interface? (if so, any link to
> > the doc?) Or for that functionality, should I directly use Yarn API?
> >
> > Thanks,
> >
> > -Jiwon
> >
>
Re: A quick question on dynamic resizing in Slider
Posted by Yong Feng <fe...@gmail.com>.
Hi Jiwon,
Slider supports horizontally scale out/in your application. Please refer to
following link for the user doc of slider commands which describes "slider
flex ..."
http://slider.incubator.apache.org/docs/manpage.html
Besides, YARN 1197 plans to support resizing YARN container. By leveraging
it, Slider is able to support vertically scale up/down your application
instance, for example increasing memory from 2GB to 4GB.
https://issues.apache.org/jira/browse/YARN-1197
Based on slider's capabilities of scale out/in and scale up/down, you could
develop a driver to automatically flex your application. As I know there is
a team who is working on an open source general auto-scaler for YARN/Slider
based application which is named as Jaguar. You may contact them for
details.
https://hadoopsummit.uservoice.com/forums/283262-hadoop-governance-security-operations/suggestions/7074316-project-jaguar-an-application-autoscaler-for-hado
Please feel free to let me know if you have any question.
Thanks,
Yong
On Fri, Jul 24, 2015 at 2:24 PM, Jiwon Seo <ji...@stanford.edu> wrote:
> Hi,
>
> I'm working on a research system for in-memory graph processing (called
> SociaLite).
>
> I'm curious if Slider supports dynamic resizing, requested by applications.
> That is, if the application detects memory pressure, the it asks Slider for
> more resources (e.g. nodes with XXX mb memory); then Slider will tell the
> application how many nodes are allocated/assigned to the app.
>
> Does Slider have this kind of application interface? (if so, any link to
> the doc?) Or for that functionality, should I directly use Yarn API?
>
> Thanks,
>
> -Jiwon
>