[jira] [Comment Edited] (STORM-594) Auto-Scaling Resources in a Topology

["Sanchit Gupta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596885#comment-14596885 ] 

Sanchit Gupta edited comment on STORM-594 at 6/23/15 3:59 AM:
--------------------------------------------------------------

Hi, 
We worked on the project and have created an initial paper. 
Link: bit.ly/Dynamic_Storm

Please let us know about any critique, questions, or comments.

Thanks
Sanchit Gupta



was (Author: sanchitgupta05):
Hi, 
We worked on the project and have created an initial paper. 
Link: bit.ly/Storm_Dynamic

Please let us know about any critique, questions, or comments.

Thanks
Sanchit Gupta


> Auto-Scaling Resources in a Topology
> ------------------------------------
>
>                 Key: STORM-594
>                 URL: https://issues.apache.org/jira/browse/STORM-594
>             Project: Apache Storm
>          Issue Type: New Feature
>            Reporter: HARSHA BALASUBRAMANIAN
>            Assignee: Rohit Sanbhadti
>            Priority: Minor
>         Attachments: Algorithm for Auto-Scaling.pdf, Project Plan and Scope.pdf
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> A useful feature missing in Storm topologies is the ability to auto-scale resources, based on a pre-configured metric. The feature proposed here aims to build such a auto-scaling mechanism using a feedback system. A brief overview of the feature is provided here. The finer details of the required components and the scaling algorithm (uses a Feedback System) are provided in the PDFs attached.
> Brief Overview:
> Topologies may get created with or (ideally) without parallelism hints and tasks in their bolts and spouts, before submitting them, If auto-scaling is set in the topology (using a Boolean flag), the topology will also get submitted to the auto-scale module.
> The auto-scale module will read a pre-configured metric (threshold/min) from a configuration file. Using this value, the topology's resources will be modified till the threshold is reached. At each stage in the auto-scale module's execution, feedback from the previous execution will be used to tune the resources.
> The systems that need to be in place to achieve this are:
> 1. Metrics which provide the current threshold (no: of acks per minute) for a topology's spouts and bolts.
> 2. Access to Storm's CLI tool which can change a topology's resources are runtime.
> 3. A new java or clojure module which runs within the Nimbus daemon or in parallel to it. This will be the auto-scale module.
> Limitations: (This is not an exhaustive list. More will be added as the design matures. Also, some of the points here may get resolved)
> To test the feature there will be a number of limitations in the first release. As the feature matures, it will be allowed to scale more
> 1. The auto-scale module will be limited to a few topologies (maybe 4 or 5 at maximum)
> 2. New bolts will not be added to scale a topology. This feature will be limited to increasing the resources within the existing topology.
> 3. Topology resources will not be decreased when it is running at more than the required number (except for a few cases)
> 4. This feature will work only for long-running topologies where the input threshold can become equal to or greater than the required threshold



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)