You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Harsha Balasubramanian <ha...@gmail.com> on 2014/11/12 20:16:12 UTC
Fwd: Auto-Scaling
Please let me know if my project (outlined below) will be useful to Storm.
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Professional Masters in Computer Science
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
---------- Forwarded message ----------
From: Harsha Balasubramanian <ha...@gmail.com>
Date: Tue, Nov 11, 2014 at 8:15 PM
Subject: Auto-Scaling
To: dev@storm.apache.org
Hi,
I am a Graduate student at the University of Toronto. As part of my
Advanced Database Systems course, I have proposed to implement an Auto
Scaling mechanism for Storm topologies (using a Feedback System).
I've gone through the pages on how to contribute to the Storm project and
have some questions. Please let me know if auto-scaling is being worked on
currently. Also, should this be a project in StormCore or StormProcessor ?
If this project will be a good addition to Storm, I will create a new JIRA
task for it and add the details of my proposed implementation. Please do
let me know.
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Professional Masters in Computer Science
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Hi Bobby,
I seem to get everything I need working except this part. It seems like I'm
making a simple mistake. I checked the JIRA post you gave and tried the
following.
1. When I tried, c:\python>python c:\storm\bin\storm list, I got this error
********************************************
*The storm client can only be run from within a release. You appear to be
trying to run the client from a checkout of Storm's source code.*
*You can download a Storm release at
http://storm-project.net/downloads.html
<http://storm-project.net/downloads.html>*
********************************************
*c:\python>*
2. When I tried running from inside storm or storm-starter directory, I get
an error too
c:\python>python c:\storm\storm list
*python: can't open file 'c:\storm\storm': [Errno 2] No such file or
directory*
c:\python>python c:\storm\examples\storm-starter\storm list
*python: can't open file 'c:\storm\examples\storm-starter\storm': [Errno 2]
No such file or directory*
The comments in the post seems to suggest the same methods that I tried
1.
"In Windows, users should run command <PYTHON_PATH>\python bin\storm
blabla...."
2.
"I use storm-starter example.
```
C:\apache-storm-0.9.2-incubating\examples\storm-starter>python
..\..\bin\storm j
ar storm-starter-topologies-0.9.2-incubating.jar
storm.starter.WordCountTopology
production-topology remote"
Where and how should I run the storm command?
FYI: I also ran the command below whoch does not give errors but also has
no effect (atleast in the UI console). The number of workers and executors
remain the same.
c:\python>python
c:\storm\examples\storm-starter\multilang\resources\storm.py rebalance
word-count-java -w 5 -n 6 -e TestSpout1=4 -e Count=2
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Wed, Dec 24, 2014 at 4:31 PM, Harsha Balasubramanian <harshab85@gmail.com
> wrote:
> Thanks for the prompt reply. Will check that.
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
>
>> looks like there is a bug storm.cmd in make_command_arguments method.
>> Its converting -e count=10 to -e count 10 (removing =) which is causing
>> this issue.
>> If you using trunk you can look STORM-487 which makes python storm
>> command work in windows.
>>
>>
>> On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
>> > Thanks Bobby.
>> >
>> > I setup a local server and CLI, UI and my topology run fine. However
>> > there
>> > is a feature that doesn't work and I need some help to figure out why.
>> >
>> > 1. I followed the instructions here to setup the storm system,
>> >
>> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
>> >
>> > 2. My feature requires the usage of the CLI tool to re-balance the
>> > topology
>> > when it needs more resources. For that I thought I could use the
>> > following
>> > CLI command,
>> >
>> > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
>> >
>> > I found this in the storm documentation at the bottom of this page:
>> >
>> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>> >
>> > 3. When I run this command in my setup, I get the error pasted below. Is
>> > re-balancing supported at the spouts and bolts level? Am I using the
>> > wrong
>> > version of the storm jar.
>> >
>> > This works:
>> > *C:\storm>storm rebalance WordCount -n 15*
>> > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
>> > localhost:6627*
>> > *3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount
>> > is
>> > rebalancing*
>> >
>> > This does not work:
>> > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
>> > *Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>> > String index out of range: -1*
>> > * at java.lang.String.substring(Unknown Source)*
>> > * at
>> >
>> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
>> > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
>> > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
>> > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
>> > * at
>> > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
>> > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
>> > * at backtype.storm.command.rebalance.main(Unknown Source)*
>> >
>> >
>> >
>> > Thanks,
>> > Harsha
>> >
>> > *Contact Details: *
>> > Harsha Balasubramanian
>> > Graduate Student at University of Toronto
>> > Masters in Computer Science
>> >
>> > Mobile : 647-771-3596
>> > Email 1: harshab85@cs.toronto.edu
>> > Email 2: harshab85@gmail.com
>> > LinkedIn : ca.linkedin.com/in/harshabala/
>> >
>> > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
>> > <ev...@yahoo-inc.com.invalid>
>> > wrote:
>> >
>> > > Harsha,
>> > > Sorry about that, I have been out at a conference.
>> > >
>> > > for 1 you can get the number of acks so far, but you are going to
>> have to
>> > > calculate the rate yourself, it keeps track of the number of acks over
>> > > several time windows, but reports them as whole numbers.
>> > > What do you mean by a local cluster? Do you mean a single node
>> cluster
>> > > you launched yourself? then yes. If you mean a local-mode cluster,
>> then no.
>> > > - Bobby
>> > >
>> > >
>> > > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian <
>> > > harshab85@gmail.com> wrote:
>> > >
>> > >
>> > > Hi,
>> > >
>> > > Can someone pls answer my questions from the above thread.
>> > >
>> > >
>> > > *I have 2 questions for you;*
>> > > *1. In one of your earlier emails, you mentioned that there is
>> > > a limited Metrics implementation which is available in the current
>> Storm
>> > > version. *
>> > >
>> > > *Will this provide 'acks threshold', i.e, the number of acks per
>> second (or
>> > > minute)? If not, can you please suggest a way in which I can
>> implement this
>> > > myself. *
>> > >
>> > > *2. Is it possible to use the CLI and UI tools from a local cluster?*
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > > *Contact Details: *
>> > > Harsha Balasubramanian
>> > > Graduate Student at University of Toronto
>> > > Masters in Computer Science
>> > >
>> > > Mobile : 647-771-3596
>> > > Email 1: harshab85@cs.toronto.edu
>> > > Email 2: harshab85@gmail.com
>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >
>> > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
>> > > harshab85@gmail.com> wrote:
>> > > >
>> > > > Hi Bobby,
>> > > >
>> > > > Hope this email finds you well. I.m about to start designing the
>> > > > auto-scaling system. Created a JIRA for it,
>> > > > https://issues.apache.org/jira/browse/STORM-594
>> > > >
>> > > > Please take a look and let me know if there are any concerns.
>> > > >
>> > > > I have 2 questions for you;
>> > > > 1. In one of your earlier emails, you mentioned that there is a
>> *limited
>> > > *Metrics
>> > > > implementation which is available in the current Storm version.
>> > > > Will this provide 'acks threshold'?
>> > > >
>> > > > 2. Is it possible to use the CLI tool from a local cluster?
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > > *Contact Details: *
>> > > > Harsha Balasubramanian
>> > > > Graduate Student at University of Toronto
>> > > > Masters in Computer Science
>> > > >
>> > > > Mobile : 647-771-3596
>> > > > Email 1: harshab85@cs.toronto.edu
>> > > > Email 2: harshab85@gmail.com
>> > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > >
>> > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
>> > > > harshab85@gmail.com> wrote:
>> > > >>
>> > > >> Thanks for the detailed explanation Bobby. I will keep this in
>> mind when
>> > > >> I create my design document.
>> > > >>
>> > > >> I will mostly not deal with adding/removing bolts to the topology
>> and
>> > > >> restrict myself to threads and tasks. This is because of the time I
>> > > have to
>> > > >> complete the project. Once I submit my report (early January), I
>> can
>> > > extend
>> > > >> what I have implemented to more use cases.
>> > > >>
>> > > >> Thanks,
>> > > >> Harsha
>> > > >>
>> > > >> Thanks,
>> > > >> Harsha
>> > > >>
>> > > >> *Contact Details: *
>> > > >> Harsha Balasubramanian
>> > > >> Graduate Student at University of Toronto
>> > > >> Professional Masters in Computer Science
>> > > >>
>> > > >> Email 1: harshab85@cs.toronto.edu
>> > > >> Email 2: harshab85@gmail.com
>> > > >> LinkedIn : ca.linkedin.com/in/harshabala/
>> > > >>
>> > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
>> > > <evans@yahoo-inc.com.invalid
>> > > >> > wrote:
>> > > >>
>> > > >>> Sure,
>> > > >>> The main thing that storm is missing right now is an metrics
>> feedback
>> > > >>> system to the scheduler (or possibly a separate piece of code that
>> > > decides
>> > > >>> when to grow/shrink a topology). We have some basic metrics, but
>> they
>> > > are
>> > > >>> not exposed to the scheduler. The other question is how do we
>> deal
>> > > with
>> > > >>> creating/destroying new bolts, especially around dealing with
>> different
>> > > >>> groupings. Some groupings make it difficult. There a number of
>> ways
>> > > to
>> > > >>> get around this, but I think the simplest way is to not
>> create/destroy
>> > > >>> instances, but to treat it as a resources problem, and give them
>> more
>> > > or
>> > > >>> less resources as needed. But that still needs to be discussed. -
>> > > Bobby
>> > > >>>
>> > > >>>
>> > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
>> Balasubramanian <
>> > > >>> harshab85@gmail.com> wrote:
>> > > >>>
>> > > >>>
>> > > >>> Hi Bobby,
>> > > >>>
>> > > >>> Thanks for getting back to me. It is encouraging to hear that
>> this
>> > > will
>> > > >>> be a good addition to Storm if done well.
>> > > >>>
>> > > >>> I have not implemented the changes yet. My project just started.
>> It
>> > > >>> spans
>> > > >>> over the next 2 months. In a few days, I will create a JIRA task
>> and
>> > > >>> submit
>> > > >>> my proposal. It would be great to brainstorm with the Storm
>> community
>> > > and
>> > > >>> iron out my design.
>> > > >>>
>> > > >>>
>> > > >>> Thanks,
>> > > >>> Harsha
>> > > >>>
>> > > >>> Thanks,
>> > > >>> Harsha
>> > > >>>
>> > > >>> *Contact Details: *
>> > > >>> Harsha Balasubramanian
>> > > >>> Graduate Student at University of Toronto
>> > > >>> Professional Masters in Computer Science
>> > > >>>
>> > > >>> Email 1: harshab85@cs.toronto.edu
>> > > >>> Email 2: harshab85@gmail.com
>> > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
>> > > >>>
>> > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
>> > > <evans@yahoo-inc.com.invalid
>> > > >>> >
>> > > >>> wrote:
>> > > >>>
>> > > >>> > Yes, this type of a feature would be great to have. I am rally
>> > > curious
>> > > >>> > how you have done the changes, as there are a lot of potential
>> > > pitfalls
>> > > >>> > here. At a minimum it would just be great to have the feedback
>> > > >>> framework
>> > > >>> > in place so we can iterate on these changes.
>> > > >>> > - Bobby
>> > > >>> >
>> > > >>> >
>> > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
>> > > storm@harsha.io>
>> > > >>> > wrote:
>> > > >>> >
>> > > >>> >
>> > > >>> > Hi,
>> > > >>> > It will definitely interesting to the storm community. It
>> will be
>> > > >>> > great if you can create a JIRA and post your code as PR for
>> others
>> > > >>> > to try out and review the code.
>> > > >>> > Thanks,
>> > > >>> > Harsha
>> > > >>> >
>> > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
>> > > >>> > > Please let me know if my project (outlined below) will be
>> useful to
>> > > >>> > > Storm.
>> > > >>> > >
>> > > >>> > > Thanks,
>> > > >>> > > Harsha
>> > > >>> > >
>> > > >>> > > *Contact Details: *
>> > > >>> > > Harsha Balasubramanian
>> > > >>> > > Graduate Student at University of Toronto
>> > > >>> > > Professional Masters in Computer Science
>> > > >>> > >
>> > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > > >>> > > Email 2: harshab85@gmail.com
>> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > >>> > >
>> > > >>> > > ---------- Forwarded message ----------
>> > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
>> > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
>> > > >>> > > Subject: Auto-Scaling
>> > > >>> > > To: dev@storm.apache.org
>> > > >>> > >
>> > > >>> > >
>> > > >>> > > Hi,
>> > > >>> > >
>> > > >>> > > I am a Graduate student at the University of Toronto. As
>> part of
>> > > my
>> > > >>> > > Advanced Database Systems course, I have proposed to
>> implement an
>> > > >>> Auto
>> > > >>> > > Scaling mechanism for Storm topologies (using a Feedback
>> System).
>> > > >>> > >
>> > > >>> > > I've gone through the pages on how to contribute to the Storm
>> > > >>> project and
>> > > >>> > > have some questions. Please let me know if auto-scaling is
>> being
>> > > >>> worked
>> > > >>> > > on
>> > > >>> > > currently. Also, should this be a project in StormCore or
>> > > >>> StormProcessor
>> > > >>> > > ?
>> > > >>> > >
>> > > >>> > > If this project will be a good addition to Storm, I will
>> create a
>> > > new
>> > > >>> > > JIRA
>> > > >>> > > task for it and add the details of my proposed implementation.
>> > > >>> Please do
>> > > >>> > > let me know.
>> > > >>> > >
>> > > >>> > > Thanks,
>> > > >>> > > Harsha
>> > > >>> > >
>> > > >>> > > *Contact Details: *
>> > > >>> > > Harsha Balasubramanian
>> > > >>> > > Graduate Student at University of Toronto
>> > > >>> > > Professional Masters in Computer Science
>> > > >>> > >
>> > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > > >>> > > Email 2: harshab85@gmail.com
>> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > >>> >
>> > > >>> >
>> > > >>> >
>> > > >>> >
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>
>> > > >>
>> > >
>> > >
>> > >
>> > >
>>
>
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Thanks for the prompt reply. Will check that.
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
> looks like there is a bug storm.cmd in make_command_arguments method.
> Its converting -e count=10 to -e count 10 (removing =) which is causing
> this issue.
> If you using trunk you can look STORM-487 which makes python storm
> command work in windows.
>
>
> On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> > Thanks Bobby.
> >
> > I setup a local server and CLI, UI and my topology run fine. However
> > there
> > is a feature that doesn't work and I need some help to figure out why.
> >
> > 1. I followed the instructions here to setup the storm system,
> >
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> >
> > 2. My feature requires the usage of the CLI tool to re-balance the
> > topology
> > when it needs more resources. For that I thought I could use the
> > following
> > CLI command,
> >
> > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
> >
> > I found this in the storm documentation at the bottom of this page:
> >
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> >
> > 3. When I run this command in my setup, I get the error pasted below. Is
> > re-balancing supported at the spouts and bolts level? Am I using the
> > wrong
> > version of the storm jar.
> >
> > This works:
> > *C:\storm>storm rebalance WordCount -n 15*
> > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
> > localhost:6627*
> > *3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount
> > is
> > rebalancing*
> >
> > This does not work:
> > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> > *Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
> > String index out of range: -1*
> > * at java.lang.String.substring(Unknown Source)*
> > * at
> > backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> > * at
> > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> > * at backtype.storm.command.rebalance.main(Unknown Source)*
> >
> >
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> > <ev...@yahoo-inc.com.invalid>
> > wrote:
> >
> > > Harsha,
> > > Sorry about that, I have been out at a conference.
> > >
> > > for 1 you can get the number of acks so far, but you are going to have
> to
> > > calculate the rate yourself, it keeps track of the number of acks over
> > > several time windows, but reports them as whole numbers.
> > > What do you mean by a local cluster? Do you mean a single node cluster
> > > you launched yourself? then yes. If you mean a local-mode cluster,
> then no.
> > > - Bobby
> > >
> > >
> > > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian <
> > > harshab85@gmail.com> wrote:
> > >
> > >
> > > Hi,
> > >
> > > Can someone pls answer my questions from the above thread.
> > >
> > >
> > > *I have 2 questions for you;*
> > > *1. In one of your earlier emails, you mentioned that there is
> > > a limited Metrics implementation which is available in the current
> Storm
> > > version. *
> > >
> > > *Will this provide 'acks threshold', i.e, the number of acks per
> second (or
> > > minute)? If not, can you please suggest a way in which I can implement
> this
> > > myself. *
> > >
> > > *2. Is it possible to use the CLI and UI tools from a local cluster?*
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Masters in Computer Science
> > >
> > > Mobile : 647-771-3596
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> > > harshab85@gmail.com> wrote:
> > > >
> > > > Hi Bobby,
> > > >
> > > > Hope this email finds you well. I.m about to start designing the
> > > > auto-scaling system. Created a JIRA for it,
> > > > https://issues.apache.org/jira/browse/STORM-594
> > > >
> > > > Please take a look and let me know if there are any concerns.
> > > >
> > > > I have 2 questions for you;
> > > > 1. In one of your earlier emails, you mentioned that there is a
> *limited
> > > *Metrics
> > > > implementation which is available in the current Storm version.
> > > > Will this provide 'acks threshold'?
> > > >
> > > > 2. Is it possible to use the CLI tool from a local cluster?
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > *Contact Details: *
> > > > Harsha Balasubramanian
> > > > Graduate Student at University of Toronto
> > > > Masters in Computer Science
> > > >
> > > > Mobile : 647-771-3596
> > > > Email 1: harshab85@cs.toronto.edu
> > > > Email 2: harshab85@gmail.com
> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >
> > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> > > > harshab85@gmail.com> wrote:
> > > >>
> > > >> Thanks for the detailed explanation Bobby. I will keep this in mind
> when
> > > >> I create my design document.
> > > >>
> > > >> I will mostly not deal with adding/removing bolts to the topology
> and
> > > >> restrict myself to threads and tasks. This is because of the time I
> > > have to
> > > >> complete the project. Once I submit my report (early January), I can
> > > extend
> > > >> what I have implemented to more use cases.
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> *Contact Details: *
> > > >> Harsha Balasubramanian
> > > >> Graduate Student at University of Toronto
> > > >> Professional Masters in Computer Science
> > > >>
> > > >> Email 1: harshab85@cs.toronto.edu
> > > >> Email 2: harshab85@gmail.com
> > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>
> > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > > <evans@yahoo-inc.com.invalid
> > > >> > wrote:
> > > >>
> > > >>> Sure,
> > > >>> The main thing that storm is missing right now is an metrics
> feedback
> > > >>> system to the scheduler (or possibly a separate piece of code that
> > > decides
> > > >>> when to grow/shrink a topology). We have some basic metrics, but
> they
> > > are
> > > >>> not exposed to the scheduler. The other question is how do we deal
> > > with
> > > >>> creating/destroying new bolts, especially around dealing with
> different
> > > >>> groupings. Some groupings make it difficult. There a number of
> ways
> > > to
> > > >>> get around this, but I think the simplest way is to not
> create/destroy
> > > >>> instances, but to treat it as a resources problem, and give them
> more
> > > or
> > > >>> less resources as needed. But that still needs to be discussed. -
> > > Bobby
> > > >>>
> > > >>>
> > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> Balasubramanian <
> > > >>> harshab85@gmail.com> wrote:
> > > >>>
> > > >>>
> > > >>> Hi Bobby,
> > > >>>
> > > >>> Thanks for getting back to me. It is encouraging to hear that this
> > > will
> > > >>> be a good addition to Storm if done well.
> > > >>>
> > > >>> I have not implemented the changes yet. My project just started.
> It
> > > >>> spans
> > > >>> over the next 2 months. In a few days, I will create a JIRA task
> and
> > > >>> submit
> > > >>> my proposal. It would be great to brainstorm with the Storm
> community
> > > and
> > > >>> iron out my design.
> > > >>>
> > > >>>
> > > >>> Thanks,
> > > >>> Harsha
> > > >>>
> > > >>> Thanks,
> > > >>> Harsha
> > > >>>
> > > >>> *Contact Details: *
> > > >>> Harsha Balasubramanian
> > > >>> Graduate Student at University of Toronto
> > > >>> Professional Masters in Computer Science
> > > >>>
> > > >>> Email 1: harshab85@cs.toronto.edu
> > > >>> Email 2: harshab85@gmail.com
> > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>>
> > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > > <evans@yahoo-inc.com.invalid
> > > >>> >
> > > >>> wrote:
> > > >>>
> > > >>> > Yes, this type of a feature would be great to have. I am rally
> > > curious
> > > >>> > how you have done the changes, as there are a lot of potential
> > > pitfalls
> > > >>> > here. At a minimum it would just be great to have the feedback
> > > >>> framework
> > > >>> > in place so we can iterate on these changes.
> > > >>> > - Bobby
> > > >>> >
> > > >>> >
> > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> > > storm@harsha.io>
> > > >>> > wrote:
> > > >>> >
> > > >>> >
> > > >>> > Hi,
> > > >>> > It will definitely interesting to the storm community. It
> will be
> > > >>> > great if you can create a JIRA and post your code as PR for
> others
> > > >>> > to try out and review the code.
> > > >>> > Thanks,
> > > >>> > Harsha
> > > >>> >
> > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> > > >>> > > Please let me know if my project (outlined below) will be
> useful to
> > > >>> > > Storm.
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Harsha
> > > >>> > >
> > > >>> > > *Contact Details: *
> > > >>> > > Harsha Balasubramanian
> > > >>> > > Graduate Student at University of Toronto
> > > >>> > > Professional Masters in Computer Science
> > > >>> > >
> > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > >>> > > Email 2: harshab85@gmail.com
> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>> > >
> > > >>> > > ---------- Forwarded message ----------
> > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > > >>> > > Subject: Auto-Scaling
> > > >>> > > To: dev@storm.apache.org
> > > >>> > >
> > > >>> > >
> > > >>> > > Hi,
> > > >>> > >
> > > >>> > > I am a Graduate student at the University of Toronto. As part
> of
> > > my
> > > >>> > > Advanced Database Systems course, I have proposed to implement
> an
> > > >>> Auto
> > > >>> > > Scaling mechanism for Storm topologies (using a Feedback
> System).
> > > >>> > >
> > > >>> > > I've gone through the pages on how to contribute to the Storm
> > > >>> project and
> > > >>> > > have some questions. Please let me know if auto-scaling is
> being
> > > >>> worked
> > > >>> > > on
> > > >>> > > currently. Also, should this be a project in StormCore or
> > > >>> StormProcessor
> > > >>> > > ?
> > > >>> > >
> > > >>> > > If this project will be a good addition to Storm, I will
> create a
> > > new
> > > >>> > > JIRA
> > > >>> > > task for it and add the details of my proposed implementation.
> > > >>> Please do
> > > >>> > > let me know.
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Harsha
> > > >>> > >
> > > >>> > > *Contact Details: *
> > > >>> > > Harsha Balasubramanian
> > > >>> > > Graduate Student at University of Toronto
> > > >>> > > Professional Masters in Computer Science
> > > >>> > >
> > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > >>> > > Email 2: harshab85@gmail.com
> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > >
> > >
> > >
> > >
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Noted.
In that case, what will happen if there are 10 executors and 5 tasks (set
during topology creation) for a component?
Harsha
Sent from mobile
On 26 Dec 2014 19:42, "Nathan Leung" <nc...@gmail.com> wrote:
> You can't change the number of tasks. Changing the number of tasks will,
> for example, break any fields grouping you have.
> On Dec 26, 2014 7:03 PM, "Harsha Balasubramanian" <ha...@gmail.com>
> wrote:
>
> > Works Fine Now!!
> > Thanks a ton to everyone who helped me get this going. You folks are
> > incredibly patient :)
> >
> > Final questions w.r.t re-balancing topologies with CLI client:
> >
> > 1. *storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10*
> > This changes workers and executors. Is it possible to changes tasks as
> well
> > using the client. If not, what will happen when
> > no:of executors > no:tasks for a component?
> >
> > 2. I'm assuming the CLI client must be invoking some code in
> > storm-core to do the re-balancing. Is it possible to call that code
> > directly (from another class) with the required parameters?
> >
> >
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Fri, Dec 26, 2014 at 6:18 PM, Harsha Balasubramanian <
> > harshab85@gmail.com
> > > wrote:
> >
> > > ok. I will try again with my new fork now. I compared files from my
> fork
> > > and master branch of storm and they look the same.
> > >
> > > Quick question: to run the pythin script, do i need to add this env
> > > variable in windows
> > > PATHEXT = .PY
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Masters in Computer Science
> > >
> > > Mobile : 647-771-3596
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > On Fri, Dec 26, 2014 at 6:08 PM, Harsha <st...@harsha.io> wrote:
> > >
> > >> your repo looks fine to me. Check your storm dist here
> > >>
> > >>
> >
> https://drive.google.com/a/hortonworks.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing#
> > >> python file bin/storm not the same as the one in your repo.
> > >> do a mvn clean package under storm-dist/binary
> > >>
> > >> On Fri, Dec 26, 2014, at 03:05 PM, Harsha Balasubramanian wrote:
> > >> > i'll create a new fork and try again
> > >> >
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > *Contact Details: *
> > >> > Harsha Balasubramanian
> > >> > Graduate Student at University of Toronto
> > >> > Masters in Computer Science
> > >> >
> > >> > Mobile : 647-771-3596
> > >> > Email 1: harshab85@cs.toronto.edu
> > >> > Email 2: harshab85@gmail.com
> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> >
> > >> > On Fri, Dec 26, 2014 at 5:51 PM, Harsha Balasubramanian
> > >> > <harshab85@gmail.com
> > >> > > wrote:
> > >> >
> > >> > > I created a fork from master. Then a new branch for my changes.
> > This
> > >> is
> > >> > > the code i used to build my local release version.
> > >> > > https://github.com/harshab85/storm/tree/autoscaling
> > >> > >
> > >> > > Followed instructions from here:
> > >> > >
> > >>
> >
> http://stackoverflow.com/questions/22559218/how-to-make-a-storm-release-from-the-source-code
> > >> > >
> > >> > > Moved to 'storm-dist/binary' directory. Ran mvn package. Deployed
> > the
> > >> > > release bundle that was created in 'storm-dist/binary/target'
> > >> directory
> > >> > >
> > >> > > Thanks,
> > >> > > Harsha
> > >> > >
> > >> > > *Contact Details: *
> > >> > > Harsha Balasubramanian
> > >> > > Graduate Student at University of Toronto
> > >> > > Masters in Computer Science
> > >> > >
> > >> > > Mobile : 647-771-3596
> > >> > > Email 1: harshab85@cs.toronto.edu
> > >> > > Email 2: harshab85@gmail.com
> > >> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >
> > >> > > On Fri, Dec 26, 2014 at 5:25 PM, Harsha <st...@harsha.io> wrote:
> > >> > >
> > >> > >> your build is older version of storm. Do a git pull and rebuild
> the
> > >> > >> storm.
> > >> > >> I don't see STORM-487 fix in your build.
> > >> > >>
> > >> > >> https://github.com/apache/storm/blob/master/bin/storm
> > >> > >> that is the latest version and I tested in windows 8 don't see
> any
> > >> > >> issues.
> > >> > >> -Harsha
> > >> > >> On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
> > >> > >> > FYI:
> > >> > >> >
> > >> > >> > Running 'storm nimbus', 'storm ui' and 'storm supervisor' works
> > >> fine
> > >> > >> from
> > >> > >> > the STORM_HOME dir.
> > >> > >> >
> > >> > >> > The problem comes only when I try to run using python to run
> > these
> > >> > >> > commands.
> > >> > >> >
> > >> > >> > I want to use python only because I need the CLI tool to
> > >> re-balance the
> > >> > >> > topology that I am using. In windows, storm.cmd apparently has
> a
> > >> bug
> > >> > >> > w.r.t
> > >> > >> > re-balancing.
> > >> > >> >
> > >> > >> > Should I do anything extra to make the storm client work with
> > >> python.
> > >> > >> > Python is in the PATH variable and works fine otherwise. Any
> > extra
> > >> > >> > environment variable needed in windows?
> > >> > >> >
> > >> > >> > Thanks,
> > >> > >> > Harsha
> > >> > >> >
> > >> > >> > *Contact Details: *
> > >> > >> > Harsha Balasubramanian
> > >> > >> > Graduate Student at University of Toronto
> > >> > >> > Masters in Computer Science
> > >> > >> >
> > >> > >> > Mobile : 647-771-3596
> > >> > >> > Email 1: harshab85@cs.toronto.edu
> > >> > >> > Email 2: harshab85@gmail.com
> > >> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> >
> > >> > >> > On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
> > >> > >> > <harshab85@gmail.com
> > >> > >> > > wrote:
> > >> > >> >
> > >> > >> > > I built the storm deliverable from main branch (g drive link
> to
> > >> tar
> > >> > >> and
> > >> > >> > > zip files),
> > >> > >> > >
> > >> > >>
> > >>
> >
> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
> > >> > >> > >
> > >> > >> > > Then I deployed it into my c: drive - c:/storm (which is
> > >> STORM_HOME)
> > >> > >> > >
> > >> > >> > > When I run the python script from STORM_HOME, I get the same
> > >> error as
> > >> > >> > > before. *Is there any setting that distinguishes the build
> as a
> > >> > >> release
> > >> > >> > > bundle*
> > >> > >> > >
> > >> > >> > > H@H-PC /c/storm
> > >> > >> > > $ python ./bin/storm nimbus
> > >> > >> > > ******************************************
> > >> > >> > > The storm client can only be run from within a release. You
> > >> appear to
> > >> > >> be
> > >> > >> > > trying to run the client from a checkout of Storm's source
> > code.
> > >> > >> > >
> > >> > >> > > You can download a Storm release at
> > >> > >> > > http://storm-project.net/downloads.html
> > >> > >> > > ******************************************
> > >> > >> > >
> > >> > >> > > H@H-PC /c/storm
> > >> > >> > > $
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > Thanks,
> > >> > >> > > Harsha
> > >> > >> > >
> > >> > >> > > *Contact Details: *
> > >> > >> > > Harsha Balasubramanian
> > >> > >> > > Graduate Student at University of Toronto
> > >> > >> > > Masters in Computer Science
> > >> > >> > >
> > >> > >> > > Mobile : 647-771-3596
> > >> > >> > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > > Email 2: harshab85@gmail.com
> > >> > >> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >
> > >> > >> > > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io>
> > wrote:
> > >> > >> > >
> > >> > >> > >> Yes he meant "master".
> > >> > >> > >> On your previous mail
> > >> > >> > >> "When I tried, c:\python>python c:\storm\bin\storm list, I
> got
> > >> this
> > >> > >> > >> error"
> > >> > >> > >> you need to build storm and deploy . Running it from source
> > >> doesn't
> > >> > >> > >> work.
> > >> > >> > >> and the correct script is storm\bin\storm like you are using
> > >> above.
> > >> > >> > >>
> > >> > >> > >>
> > >> > >> > >>
> > >> > >> > >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian
> > wrote:
> > >> > >> > >> > Hi Bobby,
> > >> > >> > >> >
> > >> > >> > >> > When you said 'trunk', did you mean the master branch
> from
> > >> > >> > >> > https://github.com/apache/storm?
> > >> > >> > >> >
> > >> > >> > >> > Thanks,
> > >> > >> > >> > Harsha
> > >> > >> > >> >
> > >> > >> > >> > *Contact Details: *
> > >> > >> > >> > Harsha Balasubramanian
> > >> > >> > >> > Graduate Student at University of Toronto
> > >> > >> > >> > Masters in Computer Science
> > >> > >> > >> >
> > >> > >> > >> > Mobile : 647-771-3596
> > >> > >> > >> > Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > Email 2: harshab85@gmail.com
> > >> > >> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> >
> > >> > >> > >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io>
> > >> wrote:
> > >> > >> > >> >
> > >> > >> > >> > > looks like there is a bug storm.cmd in
> > >> make_command_arguments
> > >> > >> method.
> > >> > >> > >> > > Its converting -e count=10 to -e count 10 (removing =)
> > >> which is
> > >> > >> > >> causing
> > >> > >> > >> > > this issue.
> > >> > >> > >> > > If you using trunk you can look STORM-487 which makes
> > python
> > >> > >> storm
> > >> > >> > >> > > command work in windows.
> > >> > >> > >> > >
> > >> > >> > >> > >
> > >> > >> > >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha
> Balasubramanian
> > >> wrote:
> > >> > >> > >> > > > Thanks Bobby.
> > >> > >> > >> > > >
> > >> > >> > >> > > > I setup a local server and CLI, UI and my topology run
> > >> fine.
> > >> > >> However
> > >> > >> > >> > > > there
> > >> > >> > >> > > > is a feature that doesn't work and I need some help to
> > >> figure
> > >> > >> out
> > >> > >> > >> why.
> > >> > >> > >> > > >
> > >> > >> > >> > > > 1. I followed the instructions here to setup the storm
> > >> system,
> > >> > >> > >> > > >
> > >> > >> > >> > >
> > >> > >> > >>
> > >> > >>
> > >>
> >
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> > >> > >> > >> > > >
> > >> > >> > >> > > > 2. My feature requires the usage of the CLI tool to
> > >> re-balance
> > >> > >> the
> > >> > >> > >> > > > topology
> > >> > >> > >> > > > when it needs more resources. For that I thought I
> could
> > >> use
> > >> > >> the
> > >> > >> > >> > > > following
> > >> > >> > >> > > > CLI command,
> > >> > >> > >> > > >
> > >> > >> > >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e
> > >> > >> yellow-bolt=10
> > >> > >> > >> > > >
> > >> > >> > >> > > > I found this in the storm documentation at the bottom
> of
> > >> this
> > >> > >> page:
> > >> > >> > >> > > >
> > >> > >> > >> > >
> > >> > >> > >>
> > >> > >>
> > >>
> >
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> > >> > >> > >> > > >
> > >> > >> > >> > > > 3. When I run this command in my setup, I get the
> error
> > >> pasted
> > >> > >> > >> below. Is
> > >> > >> > >> > > > re-balancing supported at the spouts and bolts level?
> > Am I
> > >> > >> using the
> > >> > >> > >> > > > wrong
> > >> > >> > >> > > > version of the storm jar.
> > >> > >> > >> > > >
> > >> > >> > >> > > > This works:
> > >> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 15*
> > >> > >> > >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting
> to
> > >> > >> Nimbus at
> > >> > >> > >> > > > localhost:6627*
> > >> > >> > >> > > > *3573 [main] INFO backtype.storm.command.rebalance -
> > >> Topology
> > >> > >> > >> WordCount
> > >> > >> > >> > > > is
> > >> > >> > >> > > > rebalancing*
> > >> > >> > >> > > >
> > >> > >> > >> > > > This does not work:
> > >> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> > >> > >> > >> > > > *Exception in thread "main"
> > >> > >> > >> java.lang.StringIndexOutOfBoundsException:
> > >> > >> > >> > > > String index out of range: -1*
> > >> > >> > >> > > > * at java.lang.String.substring(Unknown
> Source)*
> > >> > >> > >> > > > * at
> > >> > >> > >> > > >
> > >> > >> > >>
> > >> > >>
> > >>
> > backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> > >> > >> > >> > > > * at
> > >> clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> > >> > >> > >> > > > * at
> clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> > >> > >> > >> > > > * at
> clojure.lang.RestFn.invoke(RestFn.java:460)*
> > >> > >> > >> > > > * at
> > >> > >> > >> > > >
> > >> > >>
> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> > >> > >> > >> > > > * at
> > clojure.lang.RestFn.applyTo(RestFn.java:137)*
> > >> > >> > >> > > > * at
> > backtype.storm.command.rebalance.main(Unknown
> > >> > >> Source)*
> > >> > >> > >> > > >
> > >> > >> > >> > > >
> > >> > >> > >> > > >
> > >> > >> > >> > > > Thanks,
> > >> > >> > >> > > > Harsha
> > >> > >> > >> > > >
> > >> > >> > >> > > > *Contact Details: *
> > >> > >> > >> > > > Harsha Balasubramanian
> > >> > >> > >> > > > Graduate Student at University of Toronto
> > >> > >> > >> > > > Masters in Computer Science
> > >> > >> > >> > > >
> > >> > >> > >> > > > Mobile : 647-771-3596
> > >> > >> > >> > > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > > > Email 2: harshab85@gmail.com
> > >> > >> > >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> > > >
> > >> > >> > >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> > >> > >> > >> > > > <ev...@yahoo-inc.com.invalid>
> > >> > >> > >> > > > wrote:
> > >> > >> > >> > > >
> > >> > >> > >> > > > > Harsha,
> > >> > >> > >> > > > > Sorry about that, I have been out at a conference.
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > for 1 you can get the number of acks so far, but you
> > are
> > >> > >> going to
> > >> > >> > >> have
> > >> > >> > >> > > to
> > >> > >> > >> > > > > calculate the rate yourself, it keeps track of the
> > >> number of
> > >> > >> acks
> > >> > >> > >> over
> > >> > >> > >> > > > > several time windows, but reports them as whole
> > numbers.
> > >> > >> > >> > > > > What do you mean by a local cluster? Do you mean a
> > >> single
> > >> > >> node
> > >> > >> > >> cluster
> > >> > >> > >> > > > > you launched yourself? then yes. If you mean a
> > >> local-mode
> > >> > >> > >> cluster,
> > >> > >> > >> > > then no.
> > >> > >> > >> > > > > - Bobby
> > >> > >> > >> > > > >
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
> > >> > >> > >> Balasubramanian <
> > >> > >> > >> > > > > harshab85@gmail.com> wrote:
> > >> > >> > >> > > > >
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > Hi,
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > Can someone pls answer my questions from the above
> > >> thread.
> > >> > >> > >> > > > >
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > *I have 2 questions for you;*
> > >> > >> > >> > > > > *1. In one of your earlier emails, you mentioned
> that
> > >> there
> > >> > >> is
> > >> > >> > >> > > > > a limited Metrics implementation which is available
> in
> > >> the
> > >> > >> current
> > >> > >> > >> > > Storm
> > >> > >> > >> > > > > version. *
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > *Will this provide 'acks threshold', i.e, the number
> > of
> > >> acks
> > >> > >> per
> > >> > >> > >> > > second (or
> > >> > >> > >> > > > > minute)? If not, can you please suggest a way in
> which
> > >> I can
> > >> > >> > >> implement
> > >> > >> > >> > > this
> > >> > >> > >> > > > > myself. *
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > *2. Is it possible to use the CLI and UI tools from
> a
> > >> local
> > >> > >> > >> cluster?*
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > Thanks,
> > >> > >> > >> > > > > Harsha
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > *Contact Details: *
> > >> > >> > >> > > > > Harsha Balasubramanian
> > >> > >> > >> > > > > Graduate Student at University of Toronto
> > >> > >> > >> > > > > Masters in Computer Science
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > Mobile : 647-771-3596
> > >> > >> > >> > > > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > > > > Email 2: harshab85@gmail.com
> > >> > >> > >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> > > > >
> > >> > >> > >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha
> > >> Balasubramanian <
> > >> > >> > >> > > > > harshab85@gmail.com> wrote:
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > Hi Bobby,
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > Hope this email finds you well. I.m about to
> start
> > >> > >> designing
> > >> > >> > >> the
> > >> > >> > >> > > > > > auto-scaling system. Created a JIRA for it,
> > >> > >> > >> > > > > > https://issues.apache.org/jira/browse/STORM-594
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > Please take a look and let me know if there are
> any
> > >> > >> concerns.
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > I have 2 questions for you;
> > >> > >> > >> > > > > > 1. In one of your earlier emails, you mentioned
> that
> > >> there
> > >> > >> is a
> > >> > >> > >> > > *limited
> > >> > >> > >> > > > > *Metrics
> > >> > >> > >> > > > > > implementation which is available in the current
> > Storm
> > >> > >> version.
> > >> > >> > >> > > > > > Will this provide 'acks threshold'?
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > 2. Is it possible to use the CLI tool from a local
> > >> cluster?
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > Thanks,
> > >> > >> > >> > > > > > Harsha
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > *Contact Details: *
> > >> > >> > >> > > > > > Harsha Balasubramanian
> > >> > >> > >> > > > > > Graduate Student at University of Toronto
> > >> > >> > >> > > > > > Masters in Computer Science
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > Mobile : 647-771-3596
> > >> > >> > >> > > > > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > > > > > Email 2: harshab85@gmail.com
> > >> > >> > >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> > > > > >
> > >> > >> > >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha
> > >> Balasubramanian <
> > >> > >> > >> > > > > > harshab85@gmail.com> wrote:
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >> Thanks for the detailed explanation Bobby. I will
> > >> keep
> > >> > >> this in
> > >> > >> > >> mind
> > >> > >> > >> > > when
> > >> > >> > >> > > > > >> I create my design document.
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >> I will mostly not deal with adding/removing bolts
> > to
> > >> the
> > >> > >> > >> topology
> > >> > >> > >> > > and
> > >> > >> > >> > > > > >> restrict myself to threads and tasks. This is
> > >> because of
> > >> > >> the
> > >> > >> > >> time I
> > >> > >> > >> > > > > have to
> > >> > >> > >> > > > > >> complete the project. Once I submit my report
> > (early
> > >> > >> January),
> > >> > >> > >> I can
> > >> > >> > >> > > > > extend
> > >> > >> > >> > > > > >> what I have implemented to more use cases.
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >> Thanks,
> > >> > >> > >> > > > > >> Harsha
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >> Thanks,
> > >> > >> > >> > > > > >> Harsha
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >> *Contact Details: *
> > >> > >> > >> > > > > >> Harsha Balasubramanian
> > >> > >> > >> > > > > >> Graduate Student at University of Toronto
> > >> > >> > >> > > > > >> Professional Masters in Computer Science
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >> Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > > > > >> Email 2: harshab85@gmail.com
> > >> > >> > >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > >> > >> > >> > > > > <evans@yahoo-inc.com.invalid
> > >> > >> > >> > > > > >> > wrote:
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >>> Sure,
> > >> > >> > >> > > > > >>> The main thing that storm is missing right now
> is
> > an
> > >> > >> metrics
> > >> > >> > >> > > feedback
> > >> > >> > >> > > > > >>> system to the scheduler (or possibly a separate
> > >> piece of
> > >> > >> code
> > >> > >> > >> that
> > >> > >> > >> > > > > decides
> > >> > >> > >> > > > > >>> when to grow/shrink a topology). We have some
> > basic
> > >> > >> metrics,
> > >> > >> > >> but
> > >> > >> > >> > > they
> > >> > >> > >> > > > > are
> > >> > >> > >> > > > > >>> not exposed to the scheduler. The other
> question
> > >> is how
> > >> > >> do
> > >> > >> > >> we deal
> > >> > >> > >> > > > > with
> > >> > >> > >> > > > > >>> creating/destroying new bolts, especially around
> > >> dealing
> > >> > >> with
> > >> > >> > >> > > different
> > >> > >> > >> > > > > >>> groupings. Some groupings make it difficult.
> > >> There a
> > >> > >> number
> > >> > >> > >> of
> > >> > >> > >> > > ways
> > >> > >> > >> > > > > to
> > >> > >> > >> > > > > >>> get around this, but I think the simplest way is
> > to
> > >> not
> > >> > >> > >> > > create/destroy
> > >> > >> > >> > > > > >>> instances, but to treat it as a resources
> problem,
> > >> and
> > >> > >> give
> > >> > >> > >> them
> > >> > >> > >> > > more
> > >> > >> > >> > > > > or
> > >> > >> > >> > > > > >>> less resources as needed. But that still needs
> to
> > >> be
> > >> > >> > >> discussed. -
> > >> > >> > >> > > > > Bobby
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM,
> > Harsha
> > >> > >> > >> > > Balasubramanian <
> > >> > >> > >> > > > > >>> harshab85@gmail.com> wrote:
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> Hi Bobby,
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> Thanks for getting back to me. It is
> encouraging
> > >> to hear
> > >> > >> > >> that this
> > >> > >> > >> > > > > will
> > >> > >> > >> > > > > >>> be a good addition to Storm if done well.
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> I have not implemented the changes yet. My
> > project
> > >> just
> > >> > >> > >> started.
> > >> > >> > >> > > It
> > >> > >> > >> > > > > >>> spans
> > >> > >> > >> > > > > >>> over the next 2 months. In a few days, I will
> > >> create a
> > >> > >> JIRA
> > >> > >> > >> task
> > >> > >> > >> > > and
> > >> > >> > >> > > > > >>> submit
> > >> > >> > >> > > > > >>> my proposal. It would be great to brainstorm
> with
> > >> the
> > >> > >> Storm
> > >> > >> > >> > > community
> > >> > >> > >> > > > > and
> > >> > >> > >> > > > > >>> iron out my design.
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> Thanks,
> > >> > >> > >> > > > > >>> Harsha
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> Thanks,
> > >> > >> > >> > > > > >>> Harsha
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> *Contact Details: *
> > >> > >> > >> > > > > >>> Harsha Balasubramanian
> > >> > >> > >> > > > > >>> Graduate Student at University of Toronto
> > >> > >> > >> > > > > >>> Professional Masters in Computer Science
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > > > > >>> Email 2: harshab85@gmail.com
> > >> > >> > >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > >> > >> > >> > > > > <evans@yahoo-inc.com.invalid
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> wrote:
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>> > Yes, this type of a feature would be great to
> > >> have. I
> > >> > >> am
> > >> > >> > >> rally
> > >> > >> > >> > > > > curious
> > >> > >> > >> > > > > >>> > how you have done the changes, as there are a
> > lot
> > >> of
> > >> > >> > >> potential
> > >> > >> > >> > > > > pitfalls
> > >> > >> > >> > > > > >>> > here. At a minimum it would just be great to
> > >> have the
> > >> > >> > >> feedback
> > >> > >> > >> > > > > >>> framework
> > >> > >> > >> > > > > >>> > in place so we can iterate on these changes.
> > >> > >> > >> > > > > >>> > - Bobby
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM,
> > >> Harsha <
> > >> > >> > >> > > > > storm@harsha.io>
> > >> > >> > >> > > > > >>> > wrote:
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> > Hi,
> > >> > >> > >> > > > > >>> > It will definitely interesting to the
> storm
> > >> > >> community.
> > >> > >> > >> It
> > >> > >> > >> > > will be
> > >> > >> > >> > > > > >>> > great if you can create a JIRA and post
> your
> > >> code
> > >> > >> as PR
> > >> > >> > >> for
> > >> > >> > >> > > others
> > >> > >> > >> > > > > >>> > to try out and review the code.
> > >> > >> > >> > > > > >>> > Thanks,
> > >> > >> > >> > > > > >>> > Harsha
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha
> > >> > >> Balasubramanian
> > >> > >> > >> wrote:
> > >> > >> > >> > > > > >>> > > Please let me know if my project (outlined
> > >> below)
> > >> > >> will be
> > >> > >> > >> > > useful to
> > >> > >> > >> > > > > >>> > > Storm.
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > Thanks,
> > >> > >> > >> > > > > >>> > > Harsha
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > *Contact Details: *
> > >> > >> > >> > > > > >>> > > Harsha Balasubramanian
> > >> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
> > >> > >> > >> > > > > >>> > > Professional Masters in Computer Science
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> > >> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > ---------- Forwarded message ----------
> > >> > >> > >> > > > > >>> > > From: Harsha Balasubramanian <
> > >> harshab85@gmail.com>
> > >> > >> > >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > >> > >> > >> > > > > >>> > > Subject: Auto-Scaling
> > >> > >> > >> > > > > >>> > > To: dev@storm.apache.org
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > Hi,
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > I am a Graduate student at the University
> of
> > >> > >> Toronto. As
> > >> > >> > >> part
> > >> > >> > >> > > of
> > >> > >> > >> > > > > my
> > >> > >> > >> > > > > >>> > > Advanced Database Systems course, I have
> > >> proposed to
> > >> > >> > >> implement
> > >> > >> > >> > > an
> > >> > >> > >> > > > > >>> Auto
> > >> > >> > >> > > > > >>> > > Scaling mechanism for Storm topologies
> (using
> > a
> > >> > >> Feedback
> > >> > >> > >> > > System).
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > I've gone through the pages on how to
> > >> contribute to
> > >> > >> the
> > >> > >> > >> Storm
> > >> > >> > >> > > > > >>> project and
> > >> > >> > >> > > > > >>> > > have some questions. Please let me know if
> > >> > >> auto-scaling is
> > >> > >> > >> > > being
> > >> > >> > >> > > > > >>> worked
> > >> > >> > >> > > > > >>> > > on
> > >> > >> > >> > > > > >>> > > currently. Also, should this be a project in
> > >> > >> StormCore or
> > >> > >> > >> > > > > >>> StormProcessor
> > >> > >> > >> > > > > >>> > > ?
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > If this project will be a good addition to
> > >> Storm, I
> > >> > >> will
> > >> > >> > >> > > create a
> > >> > >> > >> > > > > new
> > >> > >> > >> > > > > >>> > > JIRA
> > >> > >> > >> > > > > >>> > > task for it and add the details of my
> proposed
> > >> > >> > >> implementation.
> > >> > >> > >> > > > > >>> Please do
> > >> > >> > >> > > > > >>> > > let me know.
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > Thanks,
> > >> > >> > >> > > > > >>> > > Harsha
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > *Contact Details: *
> > >> > >> > >> > > > > >>> > > Harsha Balasubramanian
> > >> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
> > >> > >> > >> > > > > >>> > > Professional Masters in Computer Science
> > >> > >> > >> > > > > >>> > >
> > >> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> > >> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>> >
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>>
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > > >>
> > >> > >> > >> > > > >
> > >> > >> > >> > > > >
> > >> > >> > >> > > > >
> > >> > >> > >> > > > >
> > >> > >> > >> > >
> > >> > >> > >>
> > >> > >> > >
> > >> > >> > >
> > >> > >>
> > >> > >
> > >> > >
> > >>
> > >
> > >
> >
>
Re: Fwd: Auto-Scaling
Posted by Nathan Leung <nc...@gmail.com>.
You can't change the number of tasks. Changing the number of tasks will,
for example, break any fields grouping you have.
On Dec 26, 2014 7:03 PM, "Harsha Balasubramanian" <ha...@gmail.com>
wrote:
> Works Fine Now!!
> Thanks a ton to everyone who helped me get this going. You folks are
> incredibly patient :)
>
> Final questions w.r.t re-balancing topologies with CLI client:
>
> 1. *storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10*
> This changes workers and executors. Is it possible to changes tasks as well
> using the client. If not, what will happen when
> no:of executors > no:tasks for a component?
>
> 2. I'm assuming the CLI client must be invoking some code in
> storm-core to do the re-balancing. Is it possible to call that code
> directly (from another class) with the required parameters?
>
>
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Fri, Dec 26, 2014 at 6:18 PM, Harsha Balasubramanian <
> harshab85@gmail.com
> > wrote:
>
> > ok. I will try again with my new fork now. I compared files from my fork
> > and master branch of storm and they look the same.
> >
> > Quick question: to run the pythin script, do i need to add this env
> > variable in windows
> > PATHEXT = .PY
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Fri, Dec 26, 2014 at 6:08 PM, Harsha <st...@harsha.io> wrote:
> >
> >> your repo looks fine to me. Check your storm dist here
> >>
> >>
> https://drive.google.com/a/hortonworks.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing#
> >> python file bin/storm not the same as the one in your repo.
> >> do a mvn clean package under storm-dist/binary
> >>
> >> On Fri, Dec 26, 2014, at 03:05 PM, Harsha Balasubramanian wrote:
> >> > i'll create a new fork and try again
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > *Contact Details: *
> >> > Harsha Balasubramanian
> >> > Graduate Student at University of Toronto
> >> > Masters in Computer Science
> >> >
> >> > Mobile : 647-771-3596
> >> > Email 1: harshab85@cs.toronto.edu
> >> > Email 2: harshab85@gmail.com
> >> > LinkedIn : ca.linkedin.com/in/harshabala/
> >> >
> >> > On Fri, Dec 26, 2014 at 5:51 PM, Harsha Balasubramanian
> >> > <harshab85@gmail.com
> >> > > wrote:
> >> >
> >> > > I created a fork from master. Then a new branch for my changes.
> This
> >> is
> >> > > the code i used to build my local release version.
> >> > > https://github.com/harshab85/storm/tree/autoscaling
> >> > >
> >> > > Followed instructions from here:
> >> > >
> >>
> http://stackoverflow.com/questions/22559218/how-to-make-a-storm-release-from-the-source-code
> >> > >
> >> > > Moved to 'storm-dist/binary' directory. Ran mvn package. Deployed
> the
> >> > > release bundle that was created in 'storm-dist/binary/target'
> >> directory
> >> > >
> >> > > Thanks,
> >> > > Harsha
> >> > >
> >> > > *Contact Details: *
> >> > > Harsha Balasubramanian
> >> > > Graduate Student at University of Toronto
> >> > > Masters in Computer Science
> >> > >
> >> > > Mobile : 647-771-3596
> >> > > Email 1: harshab85@cs.toronto.edu
> >> > > Email 2: harshab85@gmail.com
> >> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >
> >> > > On Fri, Dec 26, 2014 at 5:25 PM, Harsha <st...@harsha.io> wrote:
> >> > >
> >> > >> your build is older version of storm. Do a git pull and rebuild the
> >> > >> storm.
> >> > >> I don't see STORM-487 fix in your build.
> >> > >>
> >> > >> https://github.com/apache/storm/blob/master/bin/storm
> >> > >> that is the latest version and I tested in windows 8 don't see any
> >> > >> issues.
> >> > >> -Harsha
> >> > >> On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
> >> > >> > FYI:
> >> > >> >
> >> > >> > Running 'storm nimbus', 'storm ui' and 'storm supervisor' works
> >> fine
> >> > >> from
> >> > >> > the STORM_HOME dir.
> >> > >> >
> >> > >> > The problem comes only when I try to run using python to run
> these
> >> > >> > commands.
> >> > >> >
> >> > >> > I want to use python only because I need the CLI tool to
> >> re-balance the
> >> > >> > topology that I am using. In windows, storm.cmd apparently has a
> >> bug
> >> > >> > w.r.t
> >> > >> > re-balancing.
> >> > >> >
> >> > >> > Should I do anything extra to make the storm client work with
> >> python.
> >> > >> > Python is in the PATH variable and works fine otherwise. Any
> extra
> >> > >> > environment variable needed in windows?
> >> > >> >
> >> > >> > Thanks,
> >> > >> > Harsha
> >> > >> >
> >> > >> > *Contact Details: *
> >> > >> > Harsha Balasubramanian
> >> > >> > Graduate Student at University of Toronto
> >> > >> > Masters in Computer Science
> >> > >> >
> >> > >> > Mobile : 647-771-3596
> >> > >> > Email 1: harshab85@cs.toronto.edu
> >> > >> > Email 2: harshab85@gmail.com
> >> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> >
> >> > >> > On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
> >> > >> > <harshab85@gmail.com
> >> > >> > > wrote:
> >> > >> >
> >> > >> > > I built the storm deliverable from main branch (g drive link to
> >> tar
> >> > >> and
> >> > >> > > zip files),
> >> > >> > >
> >> > >>
> >>
> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
> >> > >> > >
> >> > >> > > Then I deployed it into my c: drive - c:/storm (which is
> >> STORM_HOME)
> >> > >> > >
> >> > >> > > When I run the python script from STORM_HOME, I get the same
> >> error as
> >> > >> > > before. *Is there any setting that distinguishes the build as a
> >> > >> release
> >> > >> > > bundle*
> >> > >> > >
> >> > >> > > H@H-PC /c/storm
> >> > >> > > $ python ./bin/storm nimbus
> >> > >> > > ******************************************
> >> > >> > > The storm client can only be run from within a release. You
> >> appear to
> >> > >> be
> >> > >> > > trying to run the client from a checkout of Storm's source
> code.
> >> > >> > >
> >> > >> > > You can download a Storm release at
> >> > >> > > http://storm-project.net/downloads.html
> >> > >> > > ******************************************
> >> > >> > >
> >> > >> > > H@H-PC /c/storm
> >> > >> > > $
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > Thanks,
> >> > >> > > Harsha
> >> > >> > >
> >> > >> > > *Contact Details: *
> >> > >> > > Harsha Balasubramanian
> >> > >> > > Graduate Student at University of Toronto
> >> > >> > > Masters in Computer Science
> >> > >> > >
> >> > >> > > Mobile : 647-771-3596
> >> > >> > > Email 1: harshab85@cs.toronto.edu
> >> > >> > > Email 2: harshab85@gmail.com
> >> > >> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >
> >> > >> > > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io>
> wrote:
> >> > >> > >
> >> > >> > >> Yes he meant "master".
> >> > >> > >> On your previous mail
> >> > >> > >> "When I tried, c:\python>python c:\storm\bin\storm list, I got
> >> this
> >> > >> > >> error"
> >> > >> > >> you need to build storm and deploy . Running it from source
> >> doesn't
> >> > >> > >> work.
> >> > >> > >> and the correct script is storm\bin\storm like you are using
> >> above.
> >> > >> > >>
> >> > >> > >>
> >> > >> > >>
> >> > >> > >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian
> wrote:
> >> > >> > >> > Hi Bobby,
> >> > >> > >> >
> >> > >> > >> > When you said 'trunk', did you mean the master branch from
> >> > >> > >> > https://github.com/apache/storm?
> >> > >> > >> >
> >> > >> > >> > Thanks,
> >> > >> > >> > Harsha
> >> > >> > >> >
> >> > >> > >> > *Contact Details: *
> >> > >> > >> > Harsha Balasubramanian
> >> > >> > >> > Graduate Student at University of Toronto
> >> > >> > >> > Masters in Computer Science
> >> > >> > >> >
> >> > >> > >> > Mobile : 647-771-3596
> >> > >> > >> > Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > Email 2: harshab85@gmail.com
> >> > >> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> >
> >> > >> > >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io>
> >> wrote:
> >> > >> > >> >
> >> > >> > >> > > looks like there is a bug storm.cmd in
> >> make_command_arguments
> >> > >> method.
> >> > >> > >> > > Its converting -e count=10 to -e count 10 (removing =)
> >> which is
> >> > >> > >> causing
> >> > >> > >> > > this issue.
> >> > >> > >> > > If you using trunk you can look STORM-487 which makes
> python
> >> > >> storm
> >> > >> > >> > > command work in windows.
> >> > >> > >> > >
> >> > >> > >> > >
> >> > >> > >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian
> >> wrote:
> >> > >> > >> > > > Thanks Bobby.
> >> > >> > >> > > >
> >> > >> > >> > > > I setup a local server and CLI, UI and my topology run
> >> fine.
> >> > >> However
> >> > >> > >> > > > there
> >> > >> > >> > > > is a feature that doesn't work and I need some help to
> >> figure
> >> > >> out
> >> > >> > >> why.
> >> > >> > >> > > >
> >> > >> > >> > > > 1. I followed the instructions here to setup the storm
> >> system,
> >> > >> > >> > > >
> >> > >> > >> > >
> >> > >> > >>
> >> > >>
> >>
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> >> > >> > >> > > >
> >> > >> > >> > > > 2. My feature requires the usage of the CLI tool to
> >> re-balance
> >> > >> the
> >> > >> > >> > > > topology
> >> > >> > >> > > > when it needs more resources. For that I thought I could
> >> use
> >> > >> the
> >> > >> > >> > > > following
> >> > >> > >> > > > CLI command,
> >> > >> > >> > > >
> >> > >> > >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e
> >> > >> yellow-bolt=10
> >> > >> > >> > > >
> >> > >> > >> > > > I found this in the storm documentation at the bottom of
> >> this
> >> > >> page:
> >> > >> > >> > > >
> >> > >> > >> > >
> >> > >> > >>
> >> > >>
> >>
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> >> > >> > >> > > >
> >> > >> > >> > > > 3. When I run this command in my setup, I get the error
> >> pasted
> >> > >> > >> below. Is
> >> > >> > >> > > > re-balancing supported at the spouts and bolts level?
> Am I
> >> > >> using the
> >> > >> > >> > > > wrong
> >> > >> > >> > > > version of the storm jar.
> >> > >> > >> > > >
> >> > >> > >> > > > This works:
> >> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 15*
> >> > >> > >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to
> >> > >> Nimbus at
> >> > >> > >> > > > localhost:6627*
> >> > >> > >> > > > *3573 [main] INFO backtype.storm.command.rebalance -
> >> Topology
> >> > >> > >> WordCount
> >> > >> > >> > > > is
> >> > >> > >> > > > rebalancing*
> >> > >> > >> > > >
> >> > >> > >> > > > This does not work:
> >> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> >> > >> > >> > > > *Exception in thread "main"
> >> > >> > >> java.lang.StringIndexOutOfBoundsException:
> >> > >> > >> > > > String index out of range: -1*
> >> > >> > >> > > > * at java.lang.String.substring(Unknown Source)*
> >> > >> > >> > > > * at
> >> > >> > >> > > >
> >> > >> > >>
> >> > >>
> >>
> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> >> > >> > >> > > > * at
> >> clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> >> > >> > >> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> >> > >> > >> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> >> > >> > >> > > > * at
> >> > >> > >> > > >
> >> > >> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> >> > >> > >> > > > * at
> clojure.lang.RestFn.applyTo(RestFn.java:137)*
> >> > >> > >> > > > * at
> backtype.storm.command.rebalance.main(Unknown
> >> > >> Source)*
> >> > >> > >> > > >
> >> > >> > >> > > >
> >> > >> > >> > > >
> >> > >> > >> > > > Thanks,
> >> > >> > >> > > > Harsha
> >> > >> > >> > > >
> >> > >> > >> > > > *Contact Details: *
> >> > >> > >> > > > Harsha Balasubramanian
> >> > >> > >> > > > Graduate Student at University of Toronto
> >> > >> > >> > > > Masters in Computer Science
> >> > >> > >> > > >
> >> > >> > >> > > > Mobile : 647-771-3596
> >> > >> > >> > > > Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > > > Email 2: harshab85@gmail.com
> >> > >> > >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> > > >
> >> > >> > >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> >> > >> > >> > > > <ev...@yahoo-inc.com.invalid>
> >> > >> > >> > > > wrote:
> >> > >> > >> > > >
> >> > >> > >> > > > > Harsha,
> >> > >> > >> > > > > Sorry about that, I have been out at a conference.
> >> > >> > >> > > > >
> >> > >> > >> > > > > for 1 you can get the number of acks so far, but you
> are
> >> > >> going to
> >> > >> > >> have
> >> > >> > >> > > to
> >> > >> > >> > > > > calculate the rate yourself, it keeps track of the
> >> number of
> >> > >> acks
> >> > >> > >> over
> >> > >> > >> > > > > several time windows, but reports them as whole
> numbers.
> >> > >> > >> > > > > What do you mean by a local cluster? Do you mean a
> >> single
> >> > >> node
> >> > >> > >> cluster
> >> > >> > >> > > > > you launched yourself? then yes. If you mean a
> >> local-mode
> >> > >> > >> cluster,
> >> > >> > >> > > then no.
> >> > >> > >> > > > > - Bobby
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
> >> > >> > >> Balasubramanian <
> >> > >> > >> > > > > harshab85@gmail.com> wrote:
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > > Hi,
> >> > >> > >> > > > >
> >> > >> > >> > > > > Can someone pls answer my questions from the above
> >> thread.
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > > *I have 2 questions for you;*
> >> > >> > >> > > > > *1. In one of your earlier emails, you mentioned that
> >> there
> >> > >> is
> >> > >> > >> > > > > a limited Metrics implementation which is available in
> >> the
> >> > >> current
> >> > >> > >> > > Storm
> >> > >> > >> > > > > version. *
> >> > >> > >> > > > >
> >> > >> > >> > > > > *Will this provide 'acks threshold', i.e, the number
> of
> >> acks
> >> > >> per
> >> > >> > >> > > second (or
> >> > >> > >> > > > > minute)? If not, can you please suggest a way in which
> >> I can
> >> > >> > >> implement
> >> > >> > >> > > this
> >> > >> > >> > > > > myself. *
> >> > >> > >> > > > >
> >> > >> > >> > > > > *2. Is it possible to use the CLI and UI tools from a
> >> local
> >> > >> > >> cluster?*
> >> > >> > >> > > > >
> >> > >> > >> > > > > Thanks,
> >> > >> > >> > > > > Harsha
> >> > >> > >> > > > >
> >> > >> > >> > > > > *Contact Details: *
> >> > >> > >> > > > > Harsha Balasubramanian
> >> > >> > >> > > > > Graduate Student at University of Toronto
> >> > >> > >> > > > > Masters in Computer Science
> >> > >> > >> > > > >
> >> > >> > >> > > > > Mobile : 647-771-3596
> >> > >> > >> > > > > Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > > > > Email 2: harshab85@gmail.com
> >> > >> > >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> > > > >
> >> > >> > >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha
> >> Balasubramanian <
> >> > >> > >> > > > > harshab85@gmail.com> wrote:
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Hi Bobby,
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Hope this email finds you well. I.m about to start
> >> > >> designing
> >> > >> > >> the
> >> > >> > >> > > > > > auto-scaling system. Created a JIRA for it,
> >> > >> > >> > > > > > https://issues.apache.org/jira/browse/STORM-594
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Please take a look and let me know if there are any
> >> > >> concerns.
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > I have 2 questions for you;
> >> > >> > >> > > > > > 1. In one of your earlier emails, you mentioned that
> >> there
> >> > >> is a
> >> > >> > >> > > *limited
> >> > >> > >> > > > > *Metrics
> >> > >> > >> > > > > > implementation which is available in the current
> Storm
> >> > >> version.
> >> > >> > >> > > > > > Will this provide 'acks threshold'?
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > 2. Is it possible to use the CLI tool from a local
> >> cluster?
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Thanks,
> >> > >> > >> > > > > > Harsha
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > *Contact Details: *
> >> > >> > >> > > > > > Harsha Balasubramanian
> >> > >> > >> > > > > > Graduate Student at University of Toronto
> >> > >> > >> > > > > > Masters in Computer Science
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Mobile : 647-771-3596
> >> > >> > >> > > > > > Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > > > > > Email 2: harshab85@gmail.com
> >> > >> > >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha
> >> Balasubramanian <
> >> > >> > >> > > > > > harshab85@gmail.com> wrote:
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >> Thanks for the detailed explanation Bobby. I will
> >> keep
> >> > >> this in
> >> > >> > >> mind
> >> > >> > >> > > when
> >> > >> > >> > > > > >> I create my design document.
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >> I will mostly not deal with adding/removing bolts
> to
> >> the
> >> > >> > >> topology
> >> > >> > >> > > and
> >> > >> > >> > > > > >> restrict myself to threads and tasks. This is
> >> because of
> >> > >> the
> >> > >> > >> time I
> >> > >> > >> > > > > have to
> >> > >> > >> > > > > >> complete the project. Once I submit my report
> (early
> >> > >> January),
> >> > >> > >> I can
> >> > >> > >> > > > > extend
> >> > >> > >> > > > > >> what I have implemented to more use cases.
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >> Thanks,
> >> > >> > >> > > > > >> Harsha
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >> Thanks,
> >> > >> > >> > > > > >> Harsha
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >> *Contact Details: *
> >> > >> > >> > > > > >> Harsha Balasubramanian
> >> > >> > >> > > > > >> Graduate Student at University of Toronto
> >> > >> > >> > > > > >> Professional Masters in Computer Science
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >> Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > > > > >> Email 2: harshab85@gmail.com
> >> > >> > >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> >> > >> > >> > > > > <evans@yahoo-inc.com.invalid
> >> > >> > >> > > > > >> > wrote:
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >>> Sure,
> >> > >> > >> > > > > >>> The main thing that storm is missing right now is
> an
> >> > >> metrics
> >> > >> > >> > > feedback
> >> > >> > >> > > > > >>> system to the scheduler (or possibly a separate
> >> piece of
> >> > >> code
> >> > >> > >> that
> >> > >> > >> > > > > decides
> >> > >> > >> > > > > >>> when to grow/shrink a topology). We have some
> basic
> >> > >> metrics,
> >> > >> > >> but
> >> > >> > >> > > they
> >> > >> > >> > > > > are
> >> > >> > >> > > > > >>> not exposed to the scheduler. The other question
> >> is how
> >> > >> do
> >> > >> > >> we deal
> >> > >> > >> > > > > with
> >> > >> > >> > > > > >>> creating/destroying new bolts, especially around
> >> dealing
> >> > >> with
> >> > >> > >> > > different
> >> > >> > >> > > > > >>> groupings. Some groupings make it difficult.
> >> There a
> >> > >> number
> >> > >> > >> of
> >> > >> > >> > > ways
> >> > >> > >> > > > > to
> >> > >> > >> > > > > >>> get around this, but I think the simplest way is
> to
> >> not
> >> > >> > >> > > create/destroy
> >> > >> > >> > > > > >>> instances, but to treat it as a resources problem,
> >> and
> >> > >> give
> >> > >> > >> them
> >> > >> > >> > > more
> >> > >> > >> > > > > or
> >> > >> > >> > > > > >>> less resources as needed. But that still needs to
> >> be
> >> > >> > >> discussed. -
> >> > >> > >> > > > > Bobby
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM,
> Harsha
> >> > >> > >> > > Balasubramanian <
> >> > >> > >> > > > > >>> harshab85@gmail.com> wrote:
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> Hi Bobby,
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> Thanks for getting back to me. It is encouraging
> >> to hear
> >> > >> > >> that this
> >> > >> > >> > > > > will
> >> > >> > >> > > > > >>> be a good addition to Storm if done well.
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> I have not implemented the changes yet. My
> project
> >> just
> >> > >> > >> started.
> >> > >> > >> > > It
> >> > >> > >> > > > > >>> spans
> >> > >> > >> > > > > >>> over the next 2 months. In a few days, I will
> >> create a
> >> > >> JIRA
> >> > >> > >> task
> >> > >> > >> > > and
> >> > >> > >> > > > > >>> submit
> >> > >> > >> > > > > >>> my proposal. It would be great to brainstorm with
> >> the
> >> > >> Storm
> >> > >> > >> > > community
> >> > >> > >> > > > > and
> >> > >> > >> > > > > >>> iron out my design.
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> Thanks,
> >> > >> > >> > > > > >>> Harsha
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> Thanks,
> >> > >> > >> > > > > >>> Harsha
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> *Contact Details: *
> >> > >> > >> > > > > >>> Harsha Balasubramanian
> >> > >> > >> > > > > >>> Graduate Student at University of Toronto
> >> > >> > >> > > > > >>> Professional Masters in Computer Science
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > > > > >>> Email 2: harshab85@gmail.com
> >> > >> > >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> >> > >> > >> > > > > <evans@yahoo-inc.com.invalid
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> wrote:
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>> > Yes, this type of a feature would be great to
> >> have. I
> >> > >> am
> >> > >> > >> rally
> >> > >> > >> > > > > curious
> >> > >> > >> > > > > >>> > how you have done the changes, as there are a
> lot
> >> of
> >> > >> > >> potential
> >> > >> > >> > > > > pitfalls
> >> > >> > >> > > > > >>> > here. At a minimum it would just be great to
> >> have the
> >> > >> > >> feedback
> >> > >> > >> > > > > >>> framework
> >> > >> > >> > > > > >>> > in place so we can iterate on these changes.
> >> > >> > >> > > > > >>> > - Bobby
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM,
> >> Harsha <
> >> > >> > >> > > > > storm@harsha.io>
> >> > >> > >> > > > > >>> > wrote:
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> > Hi,
> >> > >> > >> > > > > >>> > It will definitely interesting to the storm
> >> > >> community.
> >> > >> > >> It
> >> > >> > >> > > will be
> >> > >> > >> > > > > >>> > great if you can create a JIRA and post your
> >> code
> >> > >> as PR
> >> > >> > >> for
> >> > >> > >> > > others
> >> > >> > >> > > > > >>> > to try out and review the code.
> >> > >> > >> > > > > >>> > Thanks,
> >> > >> > >> > > > > >>> > Harsha
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha
> >> > >> Balasubramanian
> >> > >> > >> wrote:
> >> > >> > >> > > > > >>> > > Please let me know if my project (outlined
> >> below)
> >> > >> will be
> >> > >> > >> > > useful to
> >> > >> > >> > > > > >>> > > Storm.
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > Thanks,
> >> > >> > >> > > > > >>> > > Harsha
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > *Contact Details: *
> >> > >> > >> > > > > >>> > > Harsha Balasubramanian
> >> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
> >> > >> > >> > > > > >>> > > Professional Masters in Computer Science
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> >> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > ---------- Forwarded message ----------
> >> > >> > >> > > > > >>> > > From: Harsha Balasubramanian <
> >> harshab85@gmail.com>
> >> > >> > >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> >> > >> > >> > > > > >>> > > Subject: Auto-Scaling
> >> > >> > >> > > > > >>> > > To: dev@storm.apache.org
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > Hi,
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > I am a Graduate student at the University of
> >> > >> Toronto. As
> >> > >> > >> part
> >> > >> > >> > > of
> >> > >> > >> > > > > my
> >> > >> > >> > > > > >>> > > Advanced Database Systems course, I have
> >> proposed to
> >> > >> > >> implement
> >> > >> > >> > > an
> >> > >> > >> > > > > >>> Auto
> >> > >> > >> > > > > >>> > > Scaling mechanism for Storm topologies (using
> a
> >> > >> Feedback
> >> > >> > >> > > System).
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > I've gone through the pages on how to
> >> contribute to
> >> > >> the
> >> > >> > >> Storm
> >> > >> > >> > > > > >>> project and
> >> > >> > >> > > > > >>> > > have some questions. Please let me know if
> >> > >> auto-scaling is
> >> > >> > >> > > being
> >> > >> > >> > > > > >>> worked
> >> > >> > >> > > > > >>> > > on
> >> > >> > >> > > > > >>> > > currently. Also, should this be a project in
> >> > >> StormCore or
> >> > >> > >> > > > > >>> StormProcessor
> >> > >> > >> > > > > >>> > > ?
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > If this project will be a good addition to
> >> Storm, I
> >> > >> will
> >> > >> > >> > > create a
> >> > >> > >> > > > > new
> >> > >> > >> > > > > >>> > > JIRA
> >> > >> > >> > > > > >>> > > task for it and add the details of my proposed
> >> > >> > >> implementation.
> >> > >> > >> > > > > >>> Please do
> >> > >> > >> > > > > >>> > > let me know.
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > Thanks,
> >> > >> > >> > > > > >>> > > Harsha
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > *Contact Details: *
> >> > >> > >> > > > > >>> > > Harsha Balasubramanian
> >> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
> >> > >> > >> > > > > >>> > > Professional Masters in Computer Science
> >> > >> > >> > > > > >>> > >
> >> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> >> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> >> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>> >
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>>
> >> > >> > >> > > > > >>
> >> > >> > >> > > > > >>
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > >
> >> > >> > >>
> >> > >> > >
> >> > >> > >
> >> > >>
> >> > >
> >> > >
> >>
> >
> >
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Works Fine Now!!
Thanks a ton to everyone who helped me get this going. You folks are
incredibly patient :)
Final questions w.r.t re-balancing topologies with CLI client:
1. *storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10*
This changes workers and executors. Is it possible to changes tasks as well
using the client. If not, what will happen when
no:of executors > no:tasks for a component?
2. I'm assuming the CLI client must be invoking some code in
storm-core to do the re-balancing. Is it possible to call that code
directly (from another class) with the required parameters?
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Fri, Dec 26, 2014 at 6:18 PM, Harsha Balasubramanian <harshab85@gmail.com
> wrote:
> ok. I will try again with my new fork now. I compared files from my fork
> and master branch of storm and they look the same.
>
> Quick question: to run the pythin script, do i need to add this env
> variable in windows
> PATHEXT = .PY
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Fri, Dec 26, 2014 at 6:08 PM, Harsha <st...@harsha.io> wrote:
>
>> your repo looks fine to me. Check your storm dist here
>>
>> https://drive.google.com/a/hortonworks.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing#
>> python file bin/storm not the same as the one in your repo.
>> do a mvn clean package under storm-dist/binary
>>
>> On Fri, Dec 26, 2014, at 03:05 PM, Harsha Balasubramanian wrote:
>> > i'll create a new fork and try again
>> >
>> > Thanks,
>> > Harsha
>> >
>> > *Contact Details: *
>> > Harsha Balasubramanian
>> > Graduate Student at University of Toronto
>> > Masters in Computer Science
>> >
>> > Mobile : 647-771-3596
>> > Email 1: harshab85@cs.toronto.edu
>> > Email 2: harshab85@gmail.com
>> > LinkedIn : ca.linkedin.com/in/harshabala/
>> >
>> > On Fri, Dec 26, 2014 at 5:51 PM, Harsha Balasubramanian
>> > <harshab85@gmail.com
>> > > wrote:
>> >
>> > > I created a fork from master. Then a new branch for my changes. This
>> is
>> > > the code i used to build my local release version.
>> > > https://github.com/harshab85/storm/tree/autoscaling
>> > >
>> > > Followed instructions from here:
>> > >
>> http://stackoverflow.com/questions/22559218/how-to-make-a-storm-release-from-the-source-code
>> > >
>> > > Moved to 'storm-dist/binary' directory. Ran mvn package. Deployed the
>> > > release bundle that was created in 'storm-dist/binary/target'
>> directory
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > > *Contact Details: *
>> > > Harsha Balasubramanian
>> > > Graduate Student at University of Toronto
>> > > Masters in Computer Science
>> > >
>> > > Mobile : 647-771-3596
>> > > Email 1: harshab85@cs.toronto.edu
>> > > Email 2: harshab85@gmail.com
>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >
>> > > On Fri, Dec 26, 2014 at 5:25 PM, Harsha <st...@harsha.io> wrote:
>> > >
>> > >> your build is older version of storm. Do a git pull and rebuild the
>> > >> storm.
>> > >> I don't see STORM-487 fix in your build.
>> > >>
>> > >> https://github.com/apache/storm/blob/master/bin/storm
>> > >> that is the latest version and I tested in windows 8 don't see any
>> > >> issues.
>> > >> -Harsha
>> > >> On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
>> > >> > FYI:
>> > >> >
>> > >> > Running 'storm nimbus', 'storm ui' and 'storm supervisor' works
>> fine
>> > >> from
>> > >> > the STORM_HOME dir.
>> > >> >
>> > >> > The problem comes only when I try to run using python to run these
>> > >> > commands.
>> > >> >
>> > >> > I want to use python only because I need the CLI tool to
>> re-balance the
>> > >> > topology that I am using. In windows, storm.cmd apparently has a
>> bug
>> > >> > w.r.t
>> > >> > re-balancing.
>> > >> >
>> > >> > Should I do anything extra to make the storm client work with
>> python.
>> > >> > Python is in the PATH variable and works fine otherwise. Any extra
>> > >> > environment variable needed in windows?
>> > >> >
>> > >> > Thanks,
>> > >> > Harsha
>> > >> >
>> > >> > *Contact Details: *
>> > >> > Harsha Balasubramanian
>> > >> > Graduate Student at University of Toronto
>> > >> > Masters in Computer Science
>> > >> >
>> > >> > Mobile : 647-771-3596
>> > >> > Email 1: harshab85@cs.toronto.edu
>> > >> > Email 2: harshab85@gmail.com
>> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> >
>> > >> > On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
>> > >> > <harshab85@gmail.com
>> > >> > > wrote:
>> > >> >
>> > >> > > I built the storm deliverable from main branch (g drive link to
>> tar
>> > >> and
>> > >> > > zip files),
>> > >> > >
>> > >>
>> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
>> > >> > >
>> > >> > > Then I deployed it into my c: drive - c:/storm (which is
>> STORM_HOME)
>> > >> > >
>> > >> > > When I run the python script from STORM_HOME, I get the same
>> error as
>> > >> > > before. *Is there any setting that distinguishes the build as a
>> > >> release
>> > >> > > bundle*
>> > >> > >
>> > >> > > H@H-PC /c/storm
>> > >> > > $ python ./bin/storm nimbus
>> > >> > > ******************************************
>> > >> > > The storm client can only be run from within a release. You
>> appear to
>> > >> be
>> > >> > > trying to run the client from a checkout of Storm's source code.
>> > >> > >
>> > >> > > You can download a Storm release at
>> > >> > > http://storm-project.net/downloads.html
>> > >> > > ******************************************
>> > >> > >
>> > >> > > H@H-PC /c/storm
>> > >> > > $
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Harsha
>> > >> > >
>> > >> > > *Contact Details: *
>> > >> > > Harsha Balasubramanian
>> > >> > > Graduate Student at University of Toronto
>> > >> > > Masters in Computer Science
>> > >> > >
>> > >> > > Mobile : 647-771-3596
>> > >> > > Email 1: harshab85@cs.toronto.edu
>> > >> > > Email 2: harshab85@gmail.com
>> > >> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >
>> > >> > > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
>> > >> > >
>> > >> > >> Yes he meant "master".
>> > >> > >> On your previous mail
>> > >> > >> "When I tried, c:\python>python c:\storm\bin\storm list, I got
>> this
>> > >> > >> error"
>> > >> > >> you need to build storm and deploy . Running it from source
>> doesn't
>> > >> > >> work.
>> > >> > >> and the correct script is storm\bin\storm like you are using
>> above.
>> > >> > >>
>> > >> > >>
>> > >> > >>
>> > >> > >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
>> > >> > >> > Hi Bobby,
>> > >> > >> >
>> > >> > >> > When you said 'trunk', did you mean the master branch from
>> > >> > >> > https://github.com/apache/storm?
>> > >> > >> >
>> > >> > >> > Thanks,
>> > >> > >> > Harsha
>> > >> > >> >
>> > >> > >> > *Contact Details: *
>> > >> > >> > Harsha Balasubramanian
>> > >> > >> > Graduate Student at University of Toronto
>> > >> > >> > Masters in Computer Science
>> > >> > >> >
>> > >> > >> > Mobile : 647-771-3596
>> > >> > >> > Email 1: harshab85@cs.toronto.edu
>> > >> > >> > Email 2: harshab85@gmail.com
>> > >> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> >
>> > >> > >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io>
>> wrote:
>> > >> > >> >
>> > >> > >> > > looks like there is a bug storm.cmd in
>> make_command_arguments
>> > >> method.
>> > >> > >> > > Its converting -e count=10 to -e count 10 (removing =)
>> which is
>> > >> > >> causing
>> > >> > >> > > this issue.
>> > >> > >> > > If you using trunk you can look STORM-487 which makes python
>> > >> storm
>> > >> > >> > > command work in windows.
>> > >> > >> > >
>> > >> > >> > >
>> > >> > >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian
>> wrote:
>> > >> > >> > > > Thanks Bobby.
>> > >> > >> > > >
>> > >> > >> > > > I setup a local server and CLI, UI and my topology run
>> fine.
>> > >> However
>> > >> > >> > > > there
>> > >> > >> > > > is a feature that doesn't work and I need some help to
>> figure
>> > >> out
>> > >> > >> why.
>> > >> > >> > > >
>> > >> > >> > > > 1. I followed the instructions here to setup the storm
>> system,
>> > >> > >> > > >
>> > >> > >> > >
>> > >> > >>
>> > >>
>> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
>> > >> > >> > > >
>> > >> > >> > > > 2. My feature requires the usage of the CLI tool to
>> re-balance
>> > >> the
>> > >> > >> > > > topology
>> > >> > >> > > > when it needs more resources. For that I thought I could
>> use
>> > >> the
>> > >> > >> > > > following
>> > >> > >> > > > CLI command,
>> > >> > >> > > >
>> > >> > >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e
>> > >> yellow-bolt=10
>> > >> > >> > > >
>> > >> > >> > > > I found this in the storm documentation at the bottom of
>> this
>> > >> page:
>> > >> > >> > > >
>> > >> > >> > >
>> > >> > >>
>> > >>
>> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>> > >> > >> > > >
>> > >> > >> > > > 3. When I run this command in my setup, I get the error
>> pasted
>> > >> > >> below. Is
>> > >> > >> > > > re-balancing supported at the spouts and bolts level? Am I
>> > >> using the
>> > >> > >> > > > wrong
>> > >> > >> > > > version of the storm jar.
>> > >> > >> > > >
>> > >> > >> > > > This works:
>> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 15*
>> > >> > >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to
>> > >> Nimbus at
>> > >> > >> > > > localhost:6627*
>> > >> > >> > > > *3573 [main] INFO backtype.storm.command.rebalance -
>> Topology
>> > >> > >> WordCount
>> > >> > >> > > > is
>> > >> > >> > > > rebalancing*
>> > >> > >> > > >
>> > >> > >> > > > This does not work:
>> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
>> > >> > >> > > > *Exception in thread "main"
>> > >> > >> java.lang.StringIndexOutOfBoundsException:
>> > >> > >> > > > String index out of range: -1*
>> > >> > >> > > > * at java.lang.String.substring(Unknown Source)*
>> > >> > >> > > > * at
>> > >> > >> > > >
>> > >> > >>
>> > >>
>> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
>> > >> > >> > > > * at
>> clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
>> > >> > >> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
>> > >> > >> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
>> > >> > >> > > > * at
>> > >> > >> > > >
>> > >> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
>> > >> > >> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
>> > >> > >> > > > * at backtype.storm.command.rebalance.main(Unknown
>> > >> Source)*
>> > >> > >> > > >
>> > >> > >> > > >
>> > >> > >> > > >
>> > >> > >> > > > Thanks,
>> > >> > >> > > > Harsha
>> > >> > >> > > >
>> > >> > >> > > > *Contact Details: *
>> > >> > >> > > > Harsha Balasubramanian
>> > >> > >> > > > Graduate Student at University of Toronto
>> > >> > >> > > > Masters in Computer Science
>> > >> > >> > > >
>> > >> > >> > > > Mobile : 647-771-3596
>> > >> > >> > > > Email 1: harshab85@cs.toronto.edu
>> > >> > >> > > > Email 2: harshab85@gmail.com
>> > >> > >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> > > >
>> > >> > >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
>> > >> > >> > > > <ev...@yahoo-inc.com.invalid>
>> > >> > >> > > > wrote:
>> > >> > >> > > >
>> > >> > >> > > > > Harsha,
>> > >> > >> > > > > Sorry about that, I have been out at a conference.
>> > >> > >> > > > >
>> > >> > >> > > > > for 1 you can get the number of acks so far, but you are
>> > >> going to
>> > >> > >> have
>> > >> > >> > > to
>> > >> > >> > > > > calculate the rate yourself, it keeps track of the
>> number of
>> > >> acks
>> > >> > >> over
>> > >> > >> > > > > several time windows, but reports them as whole numbers.
>> > >> > >> > > > > What do you mean by a local cluster? Do you mean a
>> single
>> > >> node
>> > >> > >> cluster
>> > >> > >> > > > > you launched yourself? then yes. If you mean a
>> local-mode
>> > >> > >> cluster,
>> > >> > >> > > then no.
>> > >> > >> > > > > - Bobby
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
>> > >> > >> Balasubramanian <
>> > >> > >> > > > > harshab85@gmail.com> wrote:
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > > > Hi,
>> > >> > >> > > > >
>> > >> > >> > > > > Can someone pls answer my questions from the above
>> thread.
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > > > *I have 2 questions for you;*
>> > >> > >> > > > > *1. In one of your earlier emails, you mentioned that
>> there
>> > >> is
>> > >> > >> > > > > a limited Metrics implementation which is available in
>> the
>> > >> current
>> > >> > >> > > Storm
>> > >> > >> > > > > version. *
>> > >> > >> > > > >
>> > >> > >> > > > > *Will this provide 'acks threshold', i.e, the number of
>> acks
>> > >> per
>> > >> > >> > > second (or
>> > >> > >> > > > > minute)? If not, can you please suggest a way in which
>> I can
>> > >> > >> implement
>> > >> > >> > > this
>> > >> > >> > > > > myself. *
>> > >> > >> > > > >
>> > >> > >> > > > > *2. Is it possible to use the CLI and UI tools from a
>> local
>> > >> > >> cluster?*
>> > >> > >> > > > >
>> > >> > >> > > > > Thanks,
>> > >> > >> > > > > Harsha
>> > >> > >> > > > >
>> > >> > >> > > > > *Contact Details: *
>> > >> > >> > > > > Harsha Balasubramanian
>> > >> > >> > > > > Graduate Student at University of Toronto
>> > >> > >> > > > > Masters in Computer Science
>> > >> > >> > > > >
>> > >> > >> > > > > Mobile : 647-771-3596
>> > >> > >> > > > > Email 1: harshab85@cs.toronto.edu
>> > >> > >> > > > > Email 2: harshab85@gmail.com
>> > >> > >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> > > > >
>> > >> > >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha
>> Balasubramanian <
>> > >> > >> > > > > harshab85@gmail.com> wrote:
>> > >> > >> > > > > >
>> > >> > >> > > > > > Hi Bobby,
>> > >> > >> > > > > >
>> > >> > >> > > > > > Hope this email finds you well. I.m about to start
>> > >> designing
>> > >> > >> the
>> > >> > >> > > > > > auto-scaling system. Created a JIRA for it,
>> > >> > >> > > > > > https://issues.apache.org/jira/browse/STORM-594
>> > >> > >> > > > > >
>> > >> > >> > > > > > Please take a look and let me know if there are any
>> > >> concerns.
>> > >> > >> > > > > >
>> > >> > >> > > > > > I have 2 questions for you;
>> > >> > >> > > > > > 1. In one of your earlier emails, you mentioned that
>> there
>> > >> is a
>> > >> > >> > > *limited
>> > >> > >> > > > > *Metrics
>> > >> > >> > > > > > implementation which is available in the current Storm
>> > >> version.
>> > >> > >> > > > > > Will this provide 'acks threshold'?
>> > >> > >> > > > > >
>> > >> > >> > > > > > 2. Is it possible to use the CLI tool from a local
>> cluster?
>> > >> > >> > > > > >
>> > >> > >> > > > > > Thanks,
>> > >> > >> > > > > > Harsha
>> > >> > >> > > > > >
>> > >> > >> > > > > > *Contact Details: *
>> > >> > >> > > > > > Harsha Balasubramanian
>> > >> > >> > > > > > Graduate Student at University of Toronto
>> > >> > >> > > > > > Masters in Computer Science
>> > >> > >> > > > > >
>> > >> > >> > > > > > Mobile : 647-771-3596
>> > >> > >> > > > > > Email 1: harshab85@cs.toronto.edu
>> > >> > >> > > > > > Email 2: harshab85@gmail.com
>> > >> > >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> > > > > >
>> > >> > >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha
>> Balasubramanian <
>> > >> > >> > > > > > harshab85@gmail.com> wrote:
>> > >> > >> > > > > >>
>> > >> > >> > > > > >> Thanks for the detailed explanation Bobby. I will
>> keep
>> > >> this in
>> > >> > >> mind
>> > >> > >> > > when
>> > >> > >> > > > > >> I create my design document.
>> > >> > >> > > > > >>
>> > >> > >> > > > > >> I will mostly not deal with adding/removing bolts to
>> the
>> > >> > >> topology
>> > >> > >> > > and
>> > >> > >> > > > > >> restrict myself to threads and tasks. This is
>> because of
>> > >> the
>> > >> > >> time I
>> > >> > >> > > > > have to
>> > >> > >> > > > > >> complete the project. Once I submit my report (early
>> > >> January),
>> > >> > >> I can
>> > >> > >> > > > > extend
>> > >> > >> > > > > >> what I have implemented to more use cases.
>> > >> > >> > > > > >>
>> > >> > >> > > > > >> Thanks,
>> > >> > >> > > > > >> Harsha
>> > >> > >> > > > > >>
>> > >> > >> > > > > >> Thanks,
>> > >> > >> > > > > >> Harsha
>> > >> > >> > > > > >>
>> > >> > >> > > > > >> *Contact Details: *
>> > >> > >> > > > > >> Harsha Balasubramanian
>> > >> > >> > > > > >> Graduate Student at University of Toronto
>> > >> > >> > > > > >> Professional Masters in Computer Science
>> > >> > >> > > > > >>
>> > >> > >> > > > > >> Email 1: harshab85@cs.toronto.edu
>> > >> > >> > > > > >> Email 2: harshab85@gmail.com
>> > >> > >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> > > > > >>
>> > >> > >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
>> > >> > >> > > > > <evans@yahoo-inc.com.invalid
>> > >> > >> > > > > >> > wrote:
>> > >> > >> > > > > >>
>> > >> > >> > > > > >>> Sure,
>> > >> > >> > > > > >>> The main thing that storm is missing right now is an
>> > >> metrics
>> > >> > >> > > feedback
>> > >> > >> > > > > >>> system to the scheduler (or possibly a separate
>> piece of
>> > >> code
>> > >> > >> that
>> > >> > >> > > > > decides
>> > >> > >> > > > > >>> when to grow/shrink a topology). We have some basic
>> > >> metrics,
>> > >> > >> but
>> > >> > >> > > they
>> > >> > >> > > > > are
>> > >> > >> > > > > >>> not exposed to the scheduler. The other question
>> is how
>> > >> do
>> > >> > >> we deal
>> > >> > >> > > > > with
>> > >> > >> > > > > >>> creating/destroying new bolts, especially around
>> dealing
>> > >> with
>> > >> > >> > > different
>> > >> > >> > > > > >>> groupings. Some groupings make it difficult.
>> There a
>> > >> number
>> > >> > >> of
>> > >> > >> > > ways
>> > >> > >> > > > > to
>> > >> > >> > > > > >>> get around this, but I think the simplest way is to
>> not
>> > >> > >> > > create/destroy
>> > >> > >> > > > > >>> instances, but to treat it as a resources problem,
>> and
>> > >> give
>> > >> > >> them
>> > >> > >> > > more
>> > >> > >> > > > > or
>> > >> > >> > > > > >>> less resources as needed. But that still needs to
>> be
>> > >> > >> discussed. -
>> > >> > >> > > > > Bobby
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
>> > >> > >> > > Balasubramanian <
>> > >> > >> > > > > >>> harshab85@gmail.com> wrote:
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> Hi Bobby,
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> Thanks for getting back to me. It is encouraging
>> to hear
>> > >> > >> that this
>> > >> > >> > > > > will
>> > >> > >> > > > > >>> be a good addition to Storm if done well.
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> I have not implemented the changes yet. My project
>> just
>> > >> > >> started.
>> > >> > >> > > It
>> > >> > >> > > > > >>> spans
>> > >> > >> > > > > >>> over the next 2 months. In a few days, I will
>> create a
>> > >> JIRA
>> > >> > >> task
>> > >> > >> > > and
>> > >> > >> > > > > >>> submit
>> > >> > >> > > > > >>> my proposal. It would be great to brainstorm with
>> the
>> > >> Storm
>> > >> > >> > > community
>> > >> > >> > > > > and
>> > >> > >> > > > > >>> iron out my design.
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> Thanks,
>> > >> > >> > > > > >>> Harsha
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> Thanks,
>> > >> > >> > > > > >>> Harsha
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> *Contact Details: *
>> > >> > >> > > > > >>> Harsha Balasubramanian
>> > >> > >> > > > > >>> Graduate Student at University of Toronto
>> > >> > >> > > > > >>> Professional Masters in Computer Science
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> Email 1: harshab85@cs.toronto.edu
>> > >> > >> > > > > >>> Email 2: harshab85@gmail.com
>> > >> > >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
>> > >> > >> > > > > <evans@yahoo-inc.com.invalid
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> wrote:
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>> > Yes, this type of a feature would be great to
>> have. I
>> > >> am
>> > >> > >> rally
>> > >> > >> > > > > curious
>> > >> > >> > > > > >>> > how you have done the changes, as there are a lot
>> of
>> > >> > >> potential
>> > >> > >> > > > > pitfalls
>> > >> > >> > > > > >>> > here. At a minimum it would just be great to
>> have the
>> > >> > >> feedback
>> > >> > >> > > > > >>> framework
>> > >> > >> > > > > >>> > in place so we can iterate on these changes.
>> > >> > >> > > > > >>> > - Bobby
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM,
>> Harsha <
>> > >> > >> > > > > storm@harsha.io>
>> > >> > >> > > > > >>> > wrote:
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> > Hi,
>> > >> > >> > > > > >>> > It will definitely interesting to the storm
>> > >> community.
>> > >> > >> It
>> > >> > >> > > will be
>> > >> > >> > > > > >>> > great if you can create a JIRA and post your
>> code
>> > >> as PR
>> > >> > >> for
>> > >> > >> > > others
>> > >> > >> > > > > >>> > to try out and review the code.
>> > >> > >> > > > > >>> > Thanks,
>> > >> > >> > > > > >>> > Harsha
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha
>> > >> Balasubramanian
>> > >> > >> wrote:
>> > >> > >> > > > > >>> > > Please let me know if my project (outlined
>> below)
>> > >> will be
>> > >> > >> > > useful to
>> > >> > >> > > > > >>> > > Storm.
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > Thanks,
>> > >> > >> > > > > >>> > > Harsha
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > *Contact Details: *
>> > >> > >> > > > > >>> > > Harsha Balasubramanian
>> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
>> > >> > >> > > > > >>> > > Professional Masters in Computer Science
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
>> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > ---------- Forwarded message ----------
>> > >> > >> > > > > >>> > > From: Harsha Balasubramanian <
>> harshab85@gmail.com>
>> > >> > >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
>> > >> > >> > > > > >>> > > Subject: Auto-Scaling
>> > >> > >> > > > > >>> > > To: dev@storm.apache.org
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > Hi,
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > I am a Graduate student at the University of
>> > >> Toronto. As
>> > >> > >> part
>> > >> > >> > > of
>> > >> > >> > > > > my
>> > >> > >> > > > > >>> > > Advanced Database Systems course, I have
>> proposed to
>> > >> > >> implement
>> > >> > >> > > an
>> > >> > >> > > > > >>> Auto
>> > >> > >> > > > > >>> > > Scaling mechanism for Storm topologies (using a
>> > >> Feedback
>> > >> > >> > > System).
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > I've gone through the pages on how to
>> contribute to
>> > >> the
>> > >> > >> Storm
>> > >> > >> > > > > >>> project and
>> > >> > >> > > > > >>> > > have some questions. Please let me know if
>> > >> auto-scaling is
>> > >> > >> > > being
>> > >> > >> > > > > >>> worked
>> > >> > >> > > > > >>> > > on
>> > >> > >> > > > > >>> > > currently. Also, should this be a project in
>> > >> StormCore or
>> > >> > >> > > > > >>> StormProcessor
>> > >> > >> > > > > >>> > > ?
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > If this project will be a good addition to
>> Storm, I
>> > >> will
>> > >> > >> > > create a
>> > >> > >> > > > > new
>> > >> > >> > > > > >>> > > JIRA
>> > >> > >> > > > > >>> > > task for it and add the details of my proposed
>> > >> > >> implementation.
>> > >> > >> > > > > >>> Please do
>> > >> > >> > > > > >>> > > let me know.
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > Thanks,
>> > >> > >> > > > > >>> > > Harsha
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > *Contact Details: *
>> > >> > >> > > > > >>> > > Harsha Balasubramanian
>> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
>> > >> > >> > > > > >>> > > Professional Masters in Computer Science
>> > >> > >> > > > > >>> > >
>> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
>> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>> >
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>>
>> > >> > >> > > > > >>
>> > >> > >> > > > > >>
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > >
>> > >> > >>
>> > >> > >
>> > >> > >
>> > >>
>> > >
>> > >
>>
>
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
ok. I will try again with my new fork now. I compared files from my fork
and master branch of storm and they look the same.
Quick question: to run the pythin script, do i need to add this env
variable in windows
PATHEXT = .PY
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Fri, Dec 26, 2014 at 6:08 PM, Harsha <st...@harsha.io> wrote:
> your repo looks fine to me. Check your storm dist here
>
> https://drive.google.com/a/hortonworks.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing#
> python file bin/storm not the same as the one in your repo.
> do a mvn clean package under storm-dist/binary
>
> On Fri, Dec 26, 2014, at 03:05 PM, Harsha Balasubramanian wrote:
> > i'll create a new fork and try again
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Fri, Dec 26, 2014 at 5:51 PM, Harsha Balasubramanian
> > <harshab85@gmail.com
> > > wrote:
> >
> > > I created a fork from master. Then a new branch for my changes. This
> is
> > > the code i used to build my local release version.
> > > https://github.com/harshab85/storm/tree/autoscaling
> > >
> > > Followed instructions from here:
> > >
> http://stackoverflow.com/questions/22559218/how-to-make-a-storm-release-from-the-source-code
> > >
> > > Moved to 'storm-dist/binary' directory. Ran mvn package. Deployed the
> > > release bundle that was created in 'storm-dist/binary/target'
> directory
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Masters in Computer Science
> > >
> > > Mobile : 647-771-3596
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > On Fri, Dec 26, 2014 at 5:25 PM, Harsha <st...@harsha.io> wrote:
> > >
> > >> your build is older version of storm. Do a git pull and rebuild the
> > >> storm.
> > >> I don't see STORM-487 fix in your build.
> > >>
> > >> https://github.com/apache/storm/blob/master/bin/storm
> > >> that is the latest version and I tested in windows 8 don't see any
> > >> issues.
> > >> -Harsha
> > >> On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
> > >> > FYI:
> > >> >
> > >> > Running 'storm nimbus', 'storm ui' and 'storm supervisor' works fine
> > >> from
> > >> > the STORM_HOME dir.
> > >> >
> > >> > The problem comes only when I try to run using python to run these
> > >> > commands.
> > >> >
> > >> > I want to use python only because I need the CLI tool to re-balance
> the
> > >> > topology that I am using. In windows, storm.cmd apparently has a bug
> > >> > w.r.t
> > >> > re-balancing.
> > >> >
> > >> > Should I do anything extra to make the storm client work with
> python.
> > >> > Python is in the PATH variable and works fine otherwise. Any extra
> > >> > environment variable needed in windows?
> > >> >
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > *Contact Details: *
> > >> > Harsha Balasubramanian
> > >> > Graduate Student at University of Toronto
> > >> > Masters in Computer Science
> > >> >
> > >> > Mobile : 647-771-3596
> > >> > Email 1: harshab85@cs.toronto.edu
> > >> > Email 2: harshab85@gmail.com
> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> >
> > >> > On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
> > >> > <harshab85@gmail.com
> > >> > > wrote:
> > >> >
> > >> > > I built the storm deliverable from main branch (g drive link to
> tar
> > >> and
> > >> > > zip files),
> > >> > >
> > >>
> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
> > >> > >
> > >> > > Then I deployed it into my c: drive - c:/storm (which is
> STORM_HOME)
> > >> > >
> > >> > > When I run the python script from STORM_HOME, I get the same
> error as
> > >> > > before. *Is there any setting that distinguishes the build as a
> > >> release
> > >> > > bundle*
> > >> > >
> > >> > > H@H-PC /c/storm
> > >> > > $ python ./bin/storm nimbus
> > >> > > ******************************************
> > >> > > The storm client can only be run from within a release. You
> appear to
> > >> be
> > >> > > trying to run the client from a checkout of Storm's source code.
> > >> > >
> > >> > > You can download a Storm release at
> > >> > > http://storm-project.net/downloads.html
> > >> > > ******************************************
> > >> > >
> > >> > > H@H-PC /c/storm
> > >> > > $
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > Thanks,
> > >> > > Harsha
> > >> > >
> > >> > > *Contact Details: *
> > >> > > Harsha Balasubramanian
> > >> > > Graduate Student at University of Toronto
> > >> > > Masters in Computer Science
> > >> > >
> > >> > > Mobile : 647-771-3596
> > >> > > Email 1: harshab85@cs.toronto.edu
> > >> > > Email 2: harshab85@gmail.com
> > >> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >
> > >> > > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
> > >> > >
> > >> > >> Yes he meant "master".
> > >> > >> On your previous mail
> > >> > >> "When I tried, c:\python>python c:\storm\bin\storm list, I got
> this
> > >> > >> error"
> > >> > >> you need to build storm and deploy . Running it from source
> doesn't
> > >> > >> work.
> > >> > >> and the correct script is storm\bin\storm like you are using
> above.
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
> > >> > >> > Hi Bobby,
> > >> > >> >
> > >> > >> > When you said 'trunk', did you mean the master branch from
> > >> > >> > https://github.com/apache/storm?
> > >> > >> >
> > >> > >> > Thanks,
> > >> > >> > Harsha
> > >> > >> >
> > >> > >> > *Contact Details: *
> > >> > >> > Harsha Balasubramanian
> > >> > >> > Graduate Student at University of Toronto
> > >> > >> > Masters in Computer Science
> > >> > >> >
> > >> > >> > Mobile : 647-771-3596
> > >> > >> > Email 1: harshab85@cs.toronto.edu
> > >> > >> > Email 2: harshab85@gmail.com
> > >> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> >
> > >> > >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io>
> wrote:
> > >> > >> >
> > >> > >> > > looks like there is a bug storm.cmd in make_command_arguments
> > >> method.
> > >> > >> > > Its converting -e count=10 to -e count 10 (removing =) which
> is
> > >> > >> causing
> > >> > >> > > this issue.
> > >> > >> > > If you using trunk you can look STORM-487 which makes python
> > >> storm
> > >> > >> > > command work in windows.
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian
> wrote:
> > >> > >> > > > Thanks Bobby.
> > >> > >> > > >
> > >> > >> > > > I setup a local server and CLI, UI and my topology run
> fine.
> > >> However
> > >> > >> > > > there
> > >> > >> > > > is a feature that doesn't work and I need some help to
> figure
> > >> out
> > >> > >> why.
> > >> > >> > > >
> > >> > >> > > > 1. I followed the instructions here to setup the storm
> system,
> > >> > >> > > >
> > >> > >> > >
> > >> > >>
> > >>
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> > >> > >> > > >
> > >> > >> > > > 2. My feature requires the usage of the CLI tool to
> re-balance
> > >> the
> > >> > >> > > > topology
> > >> > >> > > > when it needs more resources. For that I thought I could
> use
> > >> the
> > >> > >> > > > following
> > >> > >> > > > CLI command,
> > >> > >> > > >
> > >> > >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e
> > >> yellow-bolt=10
> > >> > >> > > >
> > >> > >> > > > I found this in the storm documentation at the bottom of
> this
> > >> page:
> > >> > >> > > >
> > >> > >> > >
> > >> > >>
> > >>
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> > >> > >> > > >
> > >> > >> > > > 3. When I run this command in my setup, I get the error
> pasted
> > >> > >> below. Is
> > >> > >> > > > re-balancing supported at the spouts and bolts level? Am I
> > >> using the
> > >> > >> > > > wrong
> > >> > >> > > > version of the storm jar.
> > >> > >> > > >
> > >> > >> > > > This works:
> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 15*
> > >> > >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to
> > >> Nimbus at
> > >> > >> > > > localhost:6627*
> > >> > >> > > > *3573 [main] INFO backtype.storm.command.rebalance -
> Topology
> > >> > >> WordCount
> > >> > >> > > > is
> > >> > >> > > > rebalancing*
> > >> > >> > > >
> > >> > >> > > > This does not work:
> > >> > >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> > >> > >> > > > *Exception in thread "main"
> > >> > >> java.lang.StringIndexOutOfBoundsException:
> > >> > >> > > > String index out of range: -1*
> > >> > >> > > > * at java.lang.String.substring(Unknown Source)*
> > >> > >> > > > * at
> > >> > >> > > >
> > >> > >>
> > >>
> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> > >> > >> > > > * at
> clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> > >> > >> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> > >> > >> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> > >> > >> > > > * at
> > >> > >> > > >
> > >> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> > >> > >> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> > >> > >> > > > * at backtype.storm.command.rebalance.main(Unknown
> > >> Source)*
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > Thanks,
> > >> > >> > > > Harsha
> > >> > >> > > >
> > >> > >> > > > *Contact Details: *
> > >> > >> > > > Harsha Balasubramanian
> > >> > >> > > > Graduate Student at University of Toronto
> > >> > >> > > > Masters in Computer Science
> > >> > >> > > >
> > >> > >> > > > Mobile : 647-771-3596
> > >> > >> > > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > > > Email 2: harshab85@gmail.com
> > >> > >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > > >
> > >> > >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> > >> > >> > > > <ev...@yahoo-inc.com.invalid>
> > >> > >> > > > wrote:
> > >> > >> > > >
> > >> > >> > > > > Harsha,
> > >> > >> > > > > Sorry about that, I have been out at a conference.
> > >> > >> > > > >
> > >> > >> > > > > for 1 you can get the number of acks so far, but you are
> > >> going to
> > >> > >> have
> > >> > >> > > to
> > >> > >> > > > > calculate the rate yourself, it keeps track of the
> number of
> > >> acks
> > >> > >> over
> > >> > >> > > > > several time windows, but reports them as whole numbers.
> > >> > >> > > > > What do you mean by a local cluster? Do you mean a
> single
> > >> node
> > >> > >> cluster
> > >> > >> > > > > you launched yourself? then yes. If you mean a
> local-mode
> > >> > >> cluster,
> > >> > >> > > then no.
> > >> > >> > > > > - Bobby
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
> > >> > >> Balasubramanian <
> > >> > >> > > > > harshab85@gmail.com> wrote:
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > Hi,
> > >> > >> > > > >
> > >> > >> > > > > Can someone pls answer my questions from the above
> thread.
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > *I have 2 questions for you;*
> > >> > >> > > > > *1. In one of your earlier emails, you mentioned that
> there
> > >> is
> > >> > >> > > > > a limited Metrics implementation which is available in
> the
> > >> current
> > >> > >> > > Storm
> > >> > >> > > > > version. *
> > >> > >> > > > >
> > >> > >> > > > > *Will this provide 'acks threshold', i.e, the number of
> acks
> > >> per
> > >> > >> > > second (or
> > >> > >> > > > > minute)? If not, can you please suggest a way in which I
> can
> > >> > >> implement
> > >> > >> > > this
> > >> > >> > > > > myself. *
> > >> > >> > > > >
> > >> > >> > > > > *2. Is it possible to use the CLI and UI tools from a
> local
> > >> > >> cluster?*
> > >> > >> > > > >
> > >> > >> > > > > Thanks,
> > >> > >> > > > > Harsha
> > >> > >> > > > >
> > >> > >> > > > > *Contact Details: *
> > >> > >> > > > > Harsha Balasubramanian
> > >> > >> > > > > Graduate Student at University of Toronto
> > >> > >> > > > > Masters in Computer Science
> > >> > >> > > > >
> > >> > >> > > > > Mobile : 647-771-3596
> > >> > >> > > > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > > > > Email 2: harshab85@gmail.com
> > >> > >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > > > >
> > >> > >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian
> <
> > >> > >> > > > > harshab85@gmail.com> wrote:
> > >> > >> > > > > >
> > >> > >> > > > > > Hi Bobby,
> > >> > >> > > > > >
> > >> > >> > > > > > Hope this email finds you well. I.m about to start
> > >> designing
> > >> > >> the
> > >> > >> > > > > > auto-scaling system. Created a JIRA for it,
> > >> > >> > > > > > https://issues.apache.org/jira/browse/STORM-594
> > >> > >> > > > > >
> > >> > >> > > > > > Please take a look and let me know if there are any
> > >> concerns.
> > >> > >> > > > > >
> > >> > >> > > > > > I have 2 questions for you;
> > >> > >> > > > > > 1. In one of your earlier emails, you mentioned that
> there
> > >> is a
> > >> > >> > > *limited
> > >> > >> > > > > *Metrics
> > >> > >> > > > > > implementation which is available in the current Storm
> > >> version.
> > >> > >> > > > > > Will this provide 'acks threshold'?
> > >> > >> > > > > >
> > >> > >> > > > > > 2. Is it possible to use the CLI tool from a local
> cluster?
> > >> > >> > > > > >
> > >> > >> > > > > > Thanks,
> > >> > >> > > > > > Harsha
> > >> > >> > > > > >
> > >> > >> > > > > > *Contact Details: *
> > >> > >> > > > > > Harsha Balasubramanian
> > >> > >> > > > > > Graduate Student at University of Toronto
> > >> > >> > > > > > Masters in Computer Science
> > >> > >> > > > > >
> > >> > >> > > > > > Mobile : 647-771-3596
> > >> > >> > > > > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > > > > > Email 2: harshab85@gmail.com
> > >> > >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > > > > >
> > >> > >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha
> Balasubramanian <
> > >> > >> > > > > > harshab85@gmail.com> wrote:
> > >> > >> > > > > >>
> > >> > >> > > > > >> Thanks for the detailed explanation Bobby. I will keep
> > >> this in
> > >> > >> mind
> > >> > >> > > when
> > >> > >> > > > > >> I create my design document.
> > >> > >> > > > > >>
> > >> > >> > > > > >> I will mostly not deal with adding/removing bolts to
> the
> > >> > >> topology
> > >> > >> > > and
> > >> > >> > > > > >> restrict myself to threads and tasks. This is because
> of
> > >> the
> > >> > >> time I
> > >> > >> > > > > have to
> > >> > >> > > > > >> complete the project. Once I submit my report (early
> > >> January),
> > >> > >> I can
> > >> > >> > > > > extend
> > >> > >> > > > > >> what I have implemented to more use cases.
> > >> > >> > > > > >>
> > >> > >> > > > > >> Thanks,
> > >> > >> > > > > >> Harsha
> > >> > >> > > > > >>
> > >> > >> > > > > >> Thanks,
> > >> > >> > > > > >> Harsha
> > >> > >> > > > > >>
> > >> > >> > > > > >> *Contact Details: *
> > >> > >> > > > > >> Harsha Balasubramanian
> > >> > >> > > > > >> Graduate Student at University of Toronto
> > >> > >> > > > > >> Professional Masters in Computer Science
> > >> > >> > > > > >>
> > >> > >> > > > > >> Email 1: harshab85@cs.toronto.edu
> > >> > >> > > > > >> Email 2: harshab85@gmail.com
> > >> > >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > > > > >>
> > >> > >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > >> > >> > > > > <evans@yahoo-inc.com.invalid
> > >> > >> > > > > >> > wrote:
> > >> > >> > > > > >>
> > >> > >> > > > > >>> Sure,
> > >> > >> > > > > >>> The main thing that storm is missing right now is an
> > >> metrics
> > >> > >> > > feedback
> > >> > >> > > > > >>> system to the scheduler (or possibly a separate
> piece of
> > >> code
> > >> > >> that
> > >> > >> > > > > decides
> > >> > >> > > > > >>> when to grow/shrink a topology). We have some basic
> > >> metrics,
> > >> > >> but
> > >> > >> > > they
> > >> > >> > > > > are
> > >> > >> > > > > >>> not exposed to the scheduler. The other question is
> how
> > >> do
> > >> > >> we deal
> > >> > >> > > > > with
> > >> > >> > > > > >>> creating/destroying new bolts, especially around
> dealing
> > >> with
> > >> > >> > > different
> > >> > >> > > > > >>> groupings. Some groupings make it difficult. There
> a
> > >> number
> > >> > >> of
> > >> > >> > > ways
> > >> > >> > > > > to
> > >> > >> > > > > >>> get around this, but I think the simplest way is to
> not
> > >> > >> > > create/destroy
> > >> > >> > > > > >>> instances, but to treat it as a resources problem,
> and
> > >> give
> > >> > >> them
> > >> > >> > > more
> > >> > >> > > > > or
> > >> > >> > > > > >>> less resources as needed. But that still needs to be
> > >> > >> discussed. -
> > >> > >> > > > > Bobby
> > >> > >> > > > > >>>
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> > >> > >> > > Balasubramanian <
> > >> > >> > > > > >>> harshab85@gmail.com> wrote:
> > >> > >> > > > > >>>
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> Hi Bobby,
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> Thanks for getting back to me. It is encouraging to
> hear
> > >> > >> that this
> > >> > >> > > > > will
> > >> > >> > > > > >>> be a good addition to Storm if done well.
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> I have not implemented the changes yet. My project
> just
> > >> > >> started.
> > >> > >> > > It
> > >> > >> > > > > >>> spans
> > >> > >> > > > > >>> over the next 2 months. In a few days, I will create
> a
> > >> JIRA
> > >> > >> task
> > >> > >> > > and
> > >> > >> > > > > >>> submit
> > >> > >> > > > > >>> my proposal. It would be great to brainstorm with the
> > >> Storm
> > >> > >> > > community
> > >> > >> > > > > and
> > >> > >> > > > > >>> iron out my design.
> > >> > >> > > > > >>>
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> Thanks,
> > >> > >> > > > > >>> Harsha
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> Thanks,
> > >> > >> > > > > >>> Harsha
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> *Contact Details: *
> > >> > >> > > > > >>> Harsha Balasubramanian
> > >> > >> > > > > >>> Graduate Student at University of Toronto
> > >> > >> > > > > >>> Professional Masters in Computer Science
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> Email 1: harshab85@cs.toronto.edu
> > >> > >> > > > > >>> Email 2: harshab85@gmail.com
> > >> > >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > >> > >> > > > > <evans@yahoo-inc.com.invalid
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> wrote:
> > >> > >> > > > > >>>
> > >> > >> > > > > >>> > Yes, this type of a feature would be great to
> have. I
> > >> am
> > >> > >> rally
> > >> > >> > > > > curious
> > >> > >> > > > > >>> > how you have done the changes, as there are a lot
> of
> > >> > >> potential
> > >> > >> > > > > pitfalls
> > >> > >> > > > > >>> > here. At a minimum it would just be great to have
> the
> > >> > >> feedback
> > >> > >> > > > > >>> framework
> > >> > >> > > > > >>> > in place so we can iterate on these changes.
> > >> > >> > > > > >>> > - Bobby
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM,
> Harsha <
> > >> > >> > > > > storm@harsha.io>
> > >> > >> > > > > >>> > wrote:
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> > Hi,
> > >> > >> > > > > >>> > It will definitely interesting to the storm
> > >> community.
> > >> > >> It
> > >> > >> > > will be
> > >> > >> > > > > >>> > great if you can create a JIRA and post your
> code
> > >> as PR
> > >> > >> for
> > >> > >> > > others
> > >> > >> > > > > >>> > to try out and review the code.
> > >> > >> > > > > >>> > Thanks,
> > >> > >> > > > > >>> > Harsha
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha
> > >> Balasubramanian
> > >> > >> wrote:
> > >> > >> > > > > >>> > > Please let me know if my project (outlined below)
> > >> will be
> > >> > >> > > useful to
> > >> > >> > > > > >>> > > Storm.
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > Thanks,
> > >> > >> > > > > >>> > > Harsha
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > *Contact Details: *
> > >> > >> > > > > >>> > > Harsha Balasubramanian
> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
> > >> > >> > > > > >>> > > Professional Masters in Computer Science
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > ---------- Forwarded message ----------
> > >> > >> > > > > >>> > > From: Harsha Balasubramanian <
> harshab85@gmail.com>
> > >> > >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > >> > >> > > > > >>> > > Subject: Auto-Scaling
> > >> > >> > > > > >>> > > To: dev@storm.apache.org
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > Hi,
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > I am a Graduate student at the University of
> > >> Toronto. As
> > >> > >> part
> > >> > >> > > of
> > >> > >> > > > > my
> > >> > >> > > > > >>> > > Advanced Database Systems course, I have
> proposed to
> > >> > >> implement
> > >> > >> > > an
> > >> > >> > > > > >>> Auto
> > >> > >> > > > > >>> > > Scaling mechanism for Storm topologies (using a
> > >> Feedback
> > >> > >> > > System).
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > I've gone through the pages on how to contribute
> to
> > >> the
> > >> > >> Storm
> > >> > >> > > > > >>> project and
> > >> > >> > > > > >>> > > have some questions. Please let me know if
> > >> auto-scaling is
> > >> > >> > > being
> > >> > >> > > > > >>> worked
> > >> > >> > > > > >>> > > on
> > >> > >> > > > > >>> > > currently. Also, should this be a project in
> > >> StormCore or
> > >> > >> > > > > >>> StormProcessor
> > >> > >> > > > > >>> > > ?
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > If this project will be a good addition to
> Storm, I
> > >> will
> > >> > >> > > create a
> > >> > >> > > > > new
> > >> > >> > > > > >>> > > JIRA
> > >> > >> > > > > >>> > > task for it and add the details of my proposed
> > >> > >> implementation.
> > >> > >> > > > > >>> Please do
> > >> > >> > > > > >>> > > let me know.
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > Thanks,
> > >> > >> > > > > >>> > > Harsha
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > *Contact Details: *
> > >> > >> > > > > >>> > > Harsha Balasubramanian
> > >> > >> > > > > >>> > > Graduate Student at University of Toronto
> > >> > >> > > > > >>> > > Professional Masters in Computer Science
> > >> > >> > > > > >>> > >
> > >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> > >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>> >
> > >> > >> > > > > >>>
> > >> > >> > > > > >>>
> > >> > >> > > > > >>>
> > >> > >> > > > > >>>
> > >> > >> > > > > >>
> > >> > >> > > > > >>
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > >
> > >> > >>
> > >> > >
> > >> > >
> > >>
> > >
> > >
>
Re: Fwd: Auto-Scaling
Posted by Harsha <st...@harsha.io>.
your repo looks fine to me. Check your storm dist here
https://drive.google.com/a/hortonworks.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing#
python file bin/storm not the same as the one in your repo.
do a mvn clean package under storm-dist/binary
On Fri, Dec 26, 2014, at 03:05 PM, Harsha Balasubramanian wrote:
> i'll create a new fork and try again
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Fri, Dec 26, 2014 at 5:51 PM, Harsha Balasubramanian
> <harshab85@gmail.com
> > wrote:
>
> > I created a fork from master. Then a new branch for my changes. This is
> > the code i used to build my local release version.
> > https://github.com/harshab85/storm/tree/autoscaling
> >
> > Followed instructions from here:
> > http://stackoverflow.com/questions/22559218/how-to-make-a-storm-release-from-the-source-code
> >
> > Moved to 'storm-dist/binary' directory. Ran mvn package. Deployed the
> > release bundle that was created in 'storm-dist/binary/target' directory
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Fri, Dec 26, 2014 at 5:25 PM, Harsha <st...@harsha.io> wrote:
> >
> >> your build is older version of storm. Do a git pull and rebuild the
> >> storm.
> >> I don't see STORM-487 fix in your build.
> >>
> >> https://github.com/apache/storm/blob/master/bin/storm
> >> that is the latest version and I tested in windows 8 don't see any
> >> issues.
> >> -Harsha
> >> On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
> >> > FYI:
> >> >
> >> > Running 'storm nimbus', 'storm ui' and 'storm supervisor' works fine
> >> from
> >> > the STORM_HOME dir.
> >> >
> >> > The problem comes only when I try to run using python to run these
> >> > commands.
> >> >
> >> > I want to use python only because I need the CLI tool to re-balance the
> >> > topology that I am using. In windows, storm.cmd apparently has a bug
> >> > w.r.t
> >> > re-balancing.
> >> >
> >> > Should I do anything extra to make the storm client work with python.
> >> > Python is in the PATH variable and works fine otherwise. Any extra
> >> > environment variable needed in windows?
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > *Contact Details: *
> >> > Harsha Balasubramanian
> >> > Graduate Student at University of Toronto
> >> > Masters in Computer Science
> >> >
> >> > Mobile : 647-771-3596
> >> > Email 1: harshab85@cs.toronto.edu
> >> > Email 2: harshab85@gmail.com
> >> > LinkedIn : ca.linkedin.com/in/harshabala/
> >> >
> >> > On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
> >> > <harshab85@gmail.com
> >> > > wrote:
> >> >
> >> > > I built the storm deliverable from main branch (g drive link to tar
> >> and
> >> > > zip files),
> >> > >
> >> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
> >> > >
> >> > > Then I deployed it into my c: drive - c:/storm (which is STORM_HOME)
> >> > >
> >> > > When I run the python script from STORM_HOME, I get the same error as
> >> > > before. *Is there any setting that distinguishes the build as a
> >> release
> >> > > bundle*
> >> > >
> >> > > H@H-PC /c/storm
> >> > > $ python ./bin/storm nimbus
> >> > > ******************************************
> >> > > The storm client can only be run from within a release. You appear to
> >> be
> >> > > trying to run the client from a checkout of Storm's source code.
> >> > >
> >> > > You can download a Storm release at
> >> > > http://storm-project.net/downloads.html
> >> > > ******************************************
> >> > >
> >> > > H@H-PC /c/storm
> >> > > $
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Thanks,
> >> > > Harsha
> >> > >
> >> > > *Contact Details: *
> >> > > Harsha Balasubramanian
> >> > > Graduate Student at University of Toronto
> >> > > Masters in Computer Science
> >> > >
> >> > > Mobile : 647-771-3596
> >> > > Email 1: harshab85@cs.toronto.edu
> >> > > Email 2: harshab85@gmail.com
> >> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >
> >> > > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
> >> > >
> >> > >> Yes he meant "master".
> >> > >> On your previous mail
> >> > >> "When I tried, c:\python>python c:\storm\bin\storm list, I got this
> >> > >> error"
> >> > >> you need to build storm and deploy . Running it from source doesn't
> >> > >> work.
> >> > >> and the correct script is storm\bin\storm like you are using above.
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
> >> > >> > Hi Bobby,
> >> > >> >
> >> > >> > When you said 'trunk', did you mean the master branch from
> >> > >> > https://github.com/apache/storm?
> >> > >> >
> >> > >> > Thanks,
> >> > >> > Harsha
> >> > >> >
> >> > >> > *Contact Details: *
> >> > >> > Harsha Balasubramanian
> >> > >> > Graduate Student at University of Toronto
> >> > >> > Masters in Computer Science
> >> > >> >
> >> > >> > Mobile : 647-771-3596
> >> > >> > Email 1: harshab85@cs.toronto.edu
> >> > >> > Email 2: harshab85@gmail.com
> >> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> >
> >> > >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
> >> > >> >
> >> > >> > > looks like there is a bug storm.cmd in make_command_arguments
> >> method.
> >> > >> > > Its converting -e count=10 to -e count 10 (removing =) which is
> >> > >> causing
> >> > >> > > this issue.
> >> > >> > > If you using trunk you can look STORM-487 which makes python
> >> storm
> >> > >> > > command work in windows.
> >> > >> > >
> >> > >> > >
> >> > >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> >> > >> > > > Thanks Bobby.
> >> > >> > > >
> >> > >> > > > I setup a local server and CLI, UI and my topology run fine.
> >> However
> >> > >> > > > there
> >> > >> > > > is a feature that doesn't work and I need some help to figure
> >> out
> >> > >> why.
> >> > >> > > >
> >> > >> > > > 1. I followed the instructions here to setup the storm system,
> >> > >> > > >
> >> > >> > >
> >> > >>
> >> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> >> > >> > > >
> >> > >> > > > 2. My feature requires the usage of the CLI tool to re-balance
> >> the
> >> > >> > > > topology
> >> > >> > > > when it needs more resources. For that I thought I could use
> >> the
> >> > >> > > > following
> >> > >> > > > CLI command,
> >> > >> > > >
> >> > >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e
> >> yellow-bolt=10
> >> > >> > > >
> >> > >> > > > I found this in the storm documentation at the bottom of this
> >> page:
> >> > >> > > >
> >> > >> > >
> >> > >>
> >> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> >> > >> > > >
> >> > >> > > > 3. When I run this command in my setup, I get the error pasted
> >> > >> below. Is
> >> > >> > > > re-balancing supported at the spouts and bolts level? Am I
> >> using the
> >> > >> > > > wrong
> >> > >> > > > version of the storm jar.
> >> > >> > > >
> >> > >> > > > This works:
> >> > >> > > > *C:\storm>storm rebalance WordCount -n 15*
> >> > >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to
> >> Nimbus at
> >> > >> > > > localhost:6627*
> >> > >> > > > *3573 [main] INFO backtype.storm.command.rebalance - Topology
> >> > >> WordCount
> >> > >> > > > is
> >> > >> > > > rebalancing*
> >> > >> > > >
> >> > >> > > > This does not work:
> >> > >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> >> > >> > > > *Exception in thread "main"
> >> > >> java.lang.StringIndexOutOfBoundsException:
> >> > >> > > > String index out of range: -1*
> >> > >> > > > * at java.lang.String.substring(Unknown Source)*
> >> > >> > > > * at
> >> > >> > > >
> >> > >>
> >> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> >> > >> > > > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> >> > >> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> >> > >> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> >> > >> > > > * at
> >> > >> > > >
> >> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> >> > >> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> >> > >> > > > * at backtype.storm.command.rebalance.main(Unknown
> >> Source)*
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > Thanks,
> >> > >> > > > Harsha
> >> > >> > > >
> >> > >> > > > *Contact Details: *
> >> > >> > > > Harsha Balasubramanian
> >> > >> > > > Graduate Student at University of Toronto
> >> > >> > > > Masters in Computer Science
> >> > >> > > >
> >> > >> > > > Mobile : 647-771-3596
> >> > >> > > > Email 1: harshab85@cs.toronto.edu
> >> > >> > > > Email 2: harshab85@gmail.com
> >> > >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > > >
> >> > >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> >> > >> > > > <ev...@yahoo-inc.com.invalid>
> >> > >> > > > wrote:
> >> > >> > > >
> >> > >> > > > > Harsha,
> >> > >> > > > > Sorry about that, I have been out at a conference.
> >> > >> > > > >
> >> > >> > > > > for 1 you can get the number of acks so far, but you are
> >> going to
> >> > >> have
> >> > >> > > to
> >> > >> > > > > calculate the rate yourself, it keeps track of the number of
> >> acks
> >> > >> over
> >> > >> > > > > several time windows, but reports them as whole numbers.
> >> > >> > > > > What do you mean by a local cluster? Do you mean a single
> >> node
> >> > >> cluster
> >> > >> > > > > you launched yourself? then yes. If you mean a local-mode
> >> > >> cluster,
> >> > >> > > then no.
> >> > >> > > > > - Bobby
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
> >> > >> Balasubramanian <
> >> > >> > > > > harshab85@gmail.com> wrote:
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > Hi,
> >> > >> > > > >
> >> > >> > > > > Can someone pls answer my questions from the above thread.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > *I have 2 questions for you;*
> >> > >> > > > > *1. In one of your earlier emails, you mentioned that there
> >> is
> >> > >> > > > > a limited Metrics implementation which is available in the
> >> current
> >> > >> > > Storm
> >> > >> > > > > version. *
> >> > >> > > > >
> >> > >> > > > > *Will this provide 'acks threshold', i.e, the number of acks
> >> per
> >> > >> > > second (or
> >> > >> > > > > minute)? If not, can you please suggest a way in which I can
> >> > >> implement
> >> > >> > > this
> >> > >> > > > > myself. *
> >> > >> > > > >
> >> > >> > > > > *2. Is it possible to use the CLI and UI tools from a local
> >> > >> cluster?*
> >> > >> > > > >
> >> > >> > > > > Thanks,
> >> > >> > > > > Harsha
> >> > >> > > > >
> >> > >> > > > > *Contact Details: *
> >> > >> > > > > Harsha Balasubramanian
> >> > >> > > > > Graduate Student at University of Toronto
> >> > >> > > > > Masters in Computer Science
> >> > >> > > > >
> >> > >> > > > > Mobile : 647-771-3596
> >> > >> > > > > Email 1: harshab85@cs.toronto.edu
> >> > >> > > > > Email 2: harshab85@gmail.com
> >> > >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > > > >
> >> > >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> >> > >> > > > > harshab85@gmail.com> wrote:
> >> > >> > > > > >
> >> > >> > > > > > Hi Bobby,
> >> > >> > > > > >
> >> > >> > > > > > Hope this email finds you well. I.m about to start
> >> designing
> >> > >> the
> >> > >> > > > > > auto-scaling system. Created a JIRA for it,
> >> > >> > > > > > https://issues.apache.org/jira/browse/STORM-594
> >> > >> > > > > >
> >> > >> > > > > > Please take a look and let me know if there are any
> >> concerns.
> >> > >> > > > > >
> >> > >> > > > > > I have 2 questions for you;
> >> > >> > > > > > 1. In one of your earlier emails, you mentioned that there
> >> is a
> >> > >> > > *limited
> >> > >> > > > > *Metrics
> >> > >> > > > > > implementation which is available in the current Storm
> >> version.
> >> > >> > > > > > Will this provide 'acks threshold'?
> >> > >> > > > > >
> >> > >> > > > > > 2. Is it possible to use the CLI tool from a local cluster?
> >> > >> > > > > >
> >> > >> > > > > > Thanks,
> >> > >> > > > > > Harsha
> >> > >> > > > > >
> >> > >> > > > > > *Contact Details: *
> >> > >> > > > > > Harsha Balasubramanian
> >> > >> > > > > > Graduate Student at University of Toronto
> >> > >> > > > > > Masters in Computer Science
> >> > >> > > > > >
> >> > >> > > > > > Mobile : 647-771-3596
> >> > >> > > > > > Email 1: harshab85@cs.toronto.edu
> >> > >> > > > > > Email 2: harshab85@gmail.com
> >> > >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > > > > >
> >> > >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> >> > >> > > > > > harshab85@gmail.com> wrote:
> >> > >> > > > > >>
> >> > >> > > > > >> Thanks for the detailed explanation Bobby. I will keep
> >> this in
> >> > >> mind
> >> > >> > > when
> >> > >> > > > > >> I create my design document.
> >> > >> > > > > >>
> >> > >> > > > > >> I will mostly not deal with adding/removing bolts to the
> >> > >> topology
> >> > >> > > and
> >> > >> > > > > >> restrict myself to threads and tasks. This is because of
> >> the
> >> > >> time I
> >> > >> > > > > have to
> >> > >> > > > > >> complete the project. Once I submit my report (early
> >> January),
> >> > >> I can
> >> > >> > > > > extend
> >> > >> > > > > >> what I have implemented to more use cases.
> >> > >> > > > > >>
> >> > >> > > > > >> Thanks,
> >> > >> > > > > >> Harsha
> >> > >> > > > > >>
> >> > >> > > > > >> Thanks,
> >> > >> > > > > >> Harsha
> >> > >> > > > > >>
> >> > >> > > > > >> *Contact Details: *
> >> > >> > > > > >> Harsha Balasubramanian
> >> > >> > > > > >> Graduate Student at University of Toronto
> >> > >> > > > > >> Professional Masters in Computer Science
> >> > >> > > > > >>
> >> > >> > > > > >> Email 1: harshab85@cs.toronto.edu
> >> > >> > > > > >> Email 2: harshab85@gmail.com
> >> > >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > > > > >>
> >> > >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> >> > >> > > > > <evans@yahoo-inc.com.invalid
> >> > >> > > > > >> > wrote:
> >> > >> > > > > >>
> >> > >> > > > > >>> Sure,
> >> > >> > > > > >>> The main thing that storm is missing right now is an
> >> metrics
> >> > >> > > feedback
> >> > >> > > > > >>> system to the scheduler (or possibly a separate piece of
> >> code
> >> > >> that
> >> > >> > > > > decides
> >> > >> > > > > >>> when to grow/shrink a topology). We have some basic
> >> metrics,
> >> > >> but
> >> > >> > > they
> >> > >> > > > > are
> >> > >> > > > > >>> not exposed to the scheduler. The other question is how
> >> do
> >> > >> we deal
> >> > >> > > > > with
> >> > >> > > > > >>> creating/destroying new bolts, especially around dealing
> >> with
> >> > >> > > different
> >> > >> > > > > >>> groupings. Some groupings make it difficult. There a
> >> number
> >> > >> of
> >> > >> > > ways
> >> > >> > > > > to
> >> > >> > > > > >>> get around this, but I think the simplest way is to not
> >> > >> > > create/destroy
> >> > >> > > > > >>> instances, but to treat it as a resources problem, and
> >> give
> >> > >> them
> >> > >> > > more
> >> > >> > > > > or
> >> > >> > > > > >>> less resources as needed. But that still needs to be
> >> > >> discussed. -
> >> > >> > > > > Bobby
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> >> > >> > > Balasubramanian <
> >> > >> > > > > >>> harshab85@gmail.com> wrote:
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>> Hi Bobby,
> >> > >> > > > > >>>
> >> > >> > > > > >>> Thanks for getting back to me. It is encouraging to hear
> >> > >> that this
> >> > >> > > > > will
> >> > >> > > > > >>> be a good addition to Storm if done well.
> >> > >> > > > > >>>
> >> > >> > > > > >>> I have not implemented the changes yet. My project just
> >> > >> started.
> >> > >> > > It
> >> > >> > > > > >>> spans
> >> > >> > > > > >>> over the next 2 months. In a few days, I will create a
> >> JIRA
> >> > >> task
> >> > >> > > and
> >> > >> > > > > >>> submit
> >> > >> > > > > >>> my proposal. It would be great to brainstorm with the
> >> Storm
> >> > >> > > community
> >> > >> > > > > and
> >> > >> > > > > >>> iron out my design.
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>> Thanks,
> >> > >> > > > > >>> Harsha
> >> > >> > > > > >>>
> >> > >> > > > > >>> Thanks,
> >> > >> > > > > >>> Harsha
> >> > >> > > > > >>>
> >> > >> > > > > >>> *Contact Details: *
> >> > >> > > > > >>> Harsha Balasubramanian
> >> > >> > > > > >>> Graduate Student at University of Toronto
> >> > >> > > > > >>> Professional Masters in Computer Science
> >> > >> > > > > >>>
> >> > >> > > > > >>> Email 1: harshab85@cs.toronto.edu
> >> > >> > > > > >>> Email 2: harshab85@gmail.com
> >> > >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > > > > >>>
> >> > >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> >> > >> > > > > <evans@yahoo-inc.com.invalid
> >> > >> > > > > >>> >
> >> > >> > > > > >>> wrote:
> >> > >> > > > > >>>
> >> > >> > > > > >>> > Yes, this type of a feature would be great to have. I
> >> am
> >> > >> rally
> >> > >> > > > > curious
> >> > >> > > > > >>> > how you have done the changes, as there are a lot of
> >> > >> potential
> >> > >> > > > > pitfalls
> >> > >> > > > > >>> > here. At a minimum it would just be great to have the
> >> > >> feedback
> >> > >> > > > > >>> framework
> >> > >> > > > > >>> > in place so we can iterate on these changes.
> >> > >> > > > > >>> > - Bobby
> >> > >> > > > > >>> >
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> >> > >> > > > > storm@harsha.io>
> >> > >> > > > > >>> > wrote:
> >> > >> > > > > >>> >
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > Hi,
> >> > >> > > > > >>> > It will definitely interesting to the storm
> >> community.
> >> > >> It
> >> > >> > > will be
> >> > >> > > > > >>> > great if you can create a JIRA and post your code
> >> as PR
> >> > >> for
> >> > >> > > others
> >> > >> > > > > >>> > to try out and review the code.
> >> > >> > > > > >>> > Thanks,
> >> > >> > > > > >>> > Harsha
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha
> >> Balasubramanian
> >> > >> wrote:
> >> > >> > > > > >>> > > Please let me know if my project (outlined below)
> >> will be
> >> > >> > > useful to
> >> > >> > > > > >>> > > Storm.
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > Thanks,
> >> > >> > > > > >>> > > Harsha
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > *Contact Details: *
> >> > >> > > > > >>> > > Harsha Balasubramanian
> >> > >> > > > > >>> > > Graduate Student at University of Toronto
> >> > >> > > > > >>> > > Professional Masters in Computer Science
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > ---------- Forwarded message ----------
> >> > >> > > > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> >> > >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> >> > >> > > > > >>> > > Subject: Auto-Scaling
> >> > >> > > > > >>> > > To: dev@storm.apache.org
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > Hi,
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > I am a Graduate student at the University of
> >> Toronto. As
> >> > >> part
> >> > >> > > of
> >> > >> > > > > my
> >> > >> > > > > >>> > > Advanced Database Systems course, I have proposed to
> >> > >> implement
> >> > >> > > an
> >> > >> > > > > >>> Auto
> >> > >> > > > > >>> > > Scaling mechanism for Storm topologies (using a
> >> Feedback
> >> > >> > > System).
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > I've gone through the pages on how to contribute to
> >> the
> >> > >> Storm
> >> > >> > > > > >>> project and
> >> > >> > > > > >>> > > have some questions. Please let me know if
> >> auto-scaling is
> >> > >> > > being
> >> > >> > > > > >>> worked
> >> > >> > > > > >>> > > on
> >> > >> > > > > >>> > > currently. Also, should this be a project in
> >> StormCore or
> >> > >> > > > > >>> StormProcessor
> >> > >> > > > > >>> > > ?
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > If this project will be a good addition to Storm, I
> >> will
> >> > >> > > create a
> >> > >> > > > > new
> >> > >> > > > > >>> > > JIRA
> >> > >> > > > > >>> > > task for it and add the details of my proposed
> >> > >> implementation.
> >> > >> > > > > >>> Please do
> >> > >> > > > > >>> > > let me know.
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > Thanks,
> >> > >> > > > > >>> > > Harsha
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > *Contact Details: *
> >> > >> > > > > >>> > > Harsha Balasubramanian
> >> > >> > > > > >>> > > Graduate Student at University of Toronto
> >> > >> > > > > >>> > > Professional Masters in Computer Science
> >> > >> > > > > >>> > >
> >> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> >> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> >> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > >> > > > > >>> >
> >> > >> > > > > >>> >
> >> > >> > > > > >>> >
> >> > >> > > > > >>> >
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>
> >> > >> > > > > >>
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > >
> >> > >>
> >> > >
> >> > >
> >>
> >
> >
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
i'll create a new fork and try again
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Fri, Dec 26, 2014 at 5:51 PM, Harsha Balasubramanian <harshab85@gmail.com
> wrote:
> I created a fork from master. Then a new branch for my changes. This is
> the code i used to build my local release version.
> https://github.com/harshab85/storm/tree/autoscaling
>
> Followed instructions from here:
> http://stackoverflow.com/questions/22559218/how-to-make-a-storm-release-from-the-source-code
>
> Moved to 'storm-dist/binary' directory. Ran mvn package. Deployed the
> release bundle that was created in 'storm-dist/binary/target' directory
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Fri, Dec 26, 2014 at 5:25 PM, Harsha <st...@harsha.io> wrote:
>
>> your build is older version of storm. Do a git pull and rebuild the
>> storm.
>> I don't see STORM-487 fix in your build.
>>
>> https://github.com/apache/storm/blob/master/bin/storm
>> that is the latest version and I tested in windows 8 don't see any
>> issues.
>> -Harsha
>> On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
>> > FYI:
>> >
>> > Running 'storm nimbus', 'storm ui' and 'storm supervisor' works fine
>> from
>> > the STORM_HOME dir.
>> >
>> > The problem comes only when I try to run using python to run these
>> > commands.
>> >
>> > I want to use python only because I need the CLI tool to re-balance the
>> > topology that I am using. In windows, storm.cmd apparently has a bug
>> > w.r.t
>> > re-balancing.
>> >
>> > Should I do anything extra to make the storm client work with python.
>> > Python is in the PATH variable and works fine otherwise. Any extra
>> > environment variable needed in windows?
>> >
>> > Thanks,
>> > Harsha
>> >
>> > *Contact Details: *
>> > Harsha Balasubramanian
>> > Graduate Student at University of Toronto
>> > Masters in Computer Science
>> >
>> > Mobile : 647-771-3596
>> > Email 1: harshab85@cs.toronto.edu
>> > Email 2: harshab85@gmail.com
>> > LinkedIn : ca.linkedin.com/in/harshabala/
>> >
>> > On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
>> > <harshab85@gmail.com
>> > > wrote:
>> >
>> > > I built the storm deliverable from main branch (g drive link to tar
>> and
>> > > zip files),
>> > >
>> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
>> > >
>> > > Then I deployed it into my c: drive - c:/storm (which is STORM_HOME)
>> > >
>> > > When I run the python script from STORM_HOME, I get the same error as
>> > > before. *Is there any setting that distinguishes the build as a
>> release
>> > > bundle*
>> > >
>> > > H@H-PC /c/storm
>> > > $ python ./bin/storm nimbus
>> > > ******************************************
>> > > The storm client can only be run from within a release. You appear to
>> be
>> > > trying to run the client from a checkout of Storm's source code.
>> > >
>> > > You can download a Storm release at
>> > > http://storm-project.net/downloads.html
>> > > ******************************************
>> > >
>> > > H@H-PC /c/storm
>> > > $
>> > >
>> > >
>> > >
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > > *Contact Details: *
>> > > Harsha Balasubramanian
>> > > Graduate Student at University of Toronto
>> > > Masters in Computer Science
>> > >
>> > > Mobile : 647-771-3596
>> > > Email 1: harshab85@cs.toronto.edu
>> > > Email 2: harshab85@gmail.com
>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >
>> > > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
>> > >
>> > >> Yes he meant "master".
>> > >> On your previous mail
>> > >> "When I tried, c:\python>python c:\storm\bin\storm list, I got this
>> > >> error"
>> > >> you need to build storm and deploy . Running it from source doesn't
>> > >> work.
>> > >> and the correct script is storm\bin\storm like you are using above.
>> > >>
>> > >>
>> > >>
>> > >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
>> > >> > Hi Bobby,
>> > >> >
>> > >> > When you said 'trunk', did you mean the master branch from
>> > >> > https://github.com/apache/storm?
>> > >> >
>> > >> > Thanks,
>> > >> > Harsha
>> > >> >
>> > >> > *Contact Details: *
>> > >> > Harsha Balasubramanian
>> > >> > Graduate Student at University of Toronto
>> > >> > Masters in Computer Science
>> > >> >
>> > >> > Mobile : 647-771-3596
>> > >> > Email 1: harshab85@cs.toronto.edu
>> > >> > Email 2: harshab85@gmail.com
>> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> >
>> > >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
>> > >> >
>> > >> > > looks like there is a bug storm.cmd in make_command_arguments
>> method.
>> > >> > > Its converting -e count=10 to -e count 10 (removing =) which is
>> > >> causing
>> > >> > > this issue.
>> > >> > > If you using trunk you can look STORM-487 which makes python
>> storm
>> > >> > > command work in windows.
>> > >> > >
>> > >> > >
>> > >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
>> > >> > > > Thanks Bobby.
>> > >> > > >
>> > >> > > > I setup a local server and CLI, UI and my topology run fine.
>> However
>> > >> > > > there
>> > >> > > > is a feature that doesn't work and I need some help to figure
>> out
>> > >> why.
>> > >> > > >
>> > >> > > > 1. I followed the instructions here to setup the storm system,
>> > >> > > >
>> > >> > >
>> > >>
>> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
>> > >> > > >
>> > >> > > > 2. My feature requires the usage of the CLI tool to re-balance
>> the
>> > >> > > > topology
>> > >> > > > when it needs more resources. For that I thought I could use
>> the
>> > >> > > > following
>> > >> > > > CLI command,
>> > >> > > >
>> > >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e
>> yellow-bolt=10
>> > >> > > >
>> > >> > > > I found this in the storm documentation at the bottom of this
>> page:
>> > >> > > >
>> > >> > >
>> > >>
>> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>> > >> > > >
>> > >> > > > 3. When I run this command in my setup, I get the error pasted
>> > >> below. Is
>> > >> > > > re-balancing supported at the spouts and bolts level? Am I
>> using the
>> > >> > > > wrong
>> > >> > > > version of the storm jar.
>> > >> > > >
>> > >> > > > This works:
>> > >> > > > *C:\storm>storm rebalance WordCount -n 15*
>> > >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to
>> Nimbus at
>> > >> > > > localhost:6627*
>> > >> > > > *3573 [main] INFO backtype.storm.command.rebalance - Topology
>> > >> WordCount
>> > >> > > > is
>> > >> > > > rebalancing*
>> > >> > > >
>> > >> > > > This does not work:
>> > >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
>> > >> > > > *Exception in thread "main"
>> > >> java.lang.StringIndexOutOfBoundsException:
>> > >> > > > String index out of range: -1*
>> > >> > > > * at java.lang.String.substring(Unknown Source)*
>> > >> > > > * at
>> > >> > > >
>> > >>
>> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
>> > >> > > > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
>> > >> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
>> > >> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
>> > >> > > > * at
>> > >> > > >
>> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
>> > >> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
>> > >> > > > * at backtype.storm.command.rebalance.main(Unknown
>> Source)*
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > > Harsha
>> > >> > > >
>> > >> > > > *Contact Details: *
>> > >> > > > Harsha Balasubramanian
>> > >> > > > Graduate Student at University of Toronto
>> > >> > > > Masters in Computer Science
>> > >> > > >
>> > >> > > > Mobile : 647-771-3596
>> > >> > > > Email 1: harshab85@cs.toronto.edu
>> > >> > > > Email 2: harshab85@gmail.com
>> > >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > > >
>> > >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
>> > >> > > > <ev...@yahoo-inc.com.invalid>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Harsha,
>> > >> > > > > Sorry about that, I have been out at a conference.
>> > >> > > > >
>> > >> > > > > for 1 you can get the number of acks so far, but you are
>> going to
>> > >> have
>> > >> > > to
>> > >> > > > > calculate the rate yourself, it keeps track of the number of
>> acks
>> > >> over
>> > >> > > > > several time windows, but reports them as whole numbers.
>> > >> > > > > What do you mean by a local cluster? Do you mean a single
>> node
>> > >> cluster
>> > >> > > > > you launched yourself? then yes. If you mean a local-mode
>> > >> cluster,
>> > >> > > then no.
>> > >> > > > > - Bobby
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
>> > >> Balasubramanian <
>> > >> > > > > harshab85@gmail.com> wrote:
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > Hi,
>> > >> > > > >
>> > >> > > > > Can someone pls answer my questions from the above thread.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > *I have 2 questions for you;*
>> > >> > > > > *1. In one of your earlier emails, you mentioned that there
>> is
>> > >> > > > > a limited Metrics implementation which is available in the
>> current
>> > >> > > Storm
>> > >> > > > > version. *
>> > >> > > > >
>> > >> > > > > *Will this provide 'acks threshold', i.e, the number of acks
>> per
>> > >> > > second (or
>> > >> > > > > minute)? If not, can you please suggest a way in which I can
>> > >> implement
>> > >> > > this
>> > >> > > > > myself. *
>> > >> > > > >
>> > >> > > > > *2. Is it possible to use the CLI and UI tools from a local
>> > >> cluster?*
>> > >> > > > >
>> > >> > > > > Thanks,
>> > >> > > > > Harsha
>> > >> > > > >
>> > >> > > > > *Contact Details: *
>> > >> > > > > Harsha Balasubramanian
>> > >> > > > > Graduate Student at University of Toronto
>> > >> > > > > Masters in Computer Science
>> > >> > > > >
>> > >> > > > > Mobile : 647-771-3596
>> > >> > > > > Email 1: harshab85@cs.toronto.edu
>> > >> > > > > Email 2: harshab85@gmail.com
>> > >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > > > >
>> > >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
>> > >> > > > > harshab85@gmail.com> wrote:
>> > >> > > > > >
>> > >> > > > > > Hi Bobby,
>> > >> > > > > >
>> > >> > > > > > Hope this email finds you well. I.m about to start
>> designing
>> > >> the
>> > >> > > > > > auto-scaling system. Created a JIRA for it,
>> > >> > > > > > https://issues.apache.org/jira/browse/STORM-594
>> > >> > > > > >
>> > >> > > > > > Please take a look and let me know if there are any
>> concerns.
>> > >> > > > > >
>> > >> > > > > > I have 2 questions for you;
>> > >> > > > > > 1. In one of your earlier emails, you mentioned that there
>> is a
>> > >> > > *limited
>> > >> > > > > *Metrics
>> > >> > > > > > implementation which is available in the current Storm
>> version.
>> > >> > > > > > Will this provide 'acks threshold'?
>> > >> > > > > >
>> > >> > > > > > 2. Is it possible to use the CLI tool from a local cluster?
>> > >> > > > > >
>> > >> > > > > > Thanks,
>> > >> > > > > > Harsha
>> > >> > > > > >
>> > >> > > > > > *Contact Details: *
>> > >> > > > > > Harsha Balasubramanian
>> > >> > > > > > Graduate Student at University of Toronto
>> > >> > > > > > Masters in Computer Science
>> > >> > > > > >
>> > >> > > > > > Mobile : 647-771-3596
>> > >> > > > > > Email 1: harshab85@cs.toronto.edu
>> > >> > > > > > Email 2: harshab85@gmail.com
>> > >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > > > > >
>> > >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
>> > >> > > > > > harshab85@gmail.com> wrote:
>> > >> > > > > >>
>> > >> > > > > >> Thanks for the detailed explanation Bobby. I will keep
>> this in
>> > >> mind
>> > >> > > when
>> > >> > > > > >> I create my design document.
>> > >> > > > > >>
>> > >> > > > > >> I will mostly not deal with adding/removing bolts to the
>> > >> topology
>> > >> > > and
>> > >> > > > > >> restrict myself to threads and tasks. This is because of
>> the
>> > >> time I
>> > >> > > > > have to
>> > >> > > > > >> complete the project. Once I submit my report (early
>> January),
>> > >> I can
>> > >> > > > > extend
>> > >> > > > > >> what I have implemented to more use cases.
>> > >> > > > > >>
>> > >> > > > > >> Thanks,
>> > >> > > > > >> Harsha
>> > >> > > > > >>
>> > >> > > > > >> Thanks,
>> > >> > > > > >> Harsha
>> > >> > > > > >>
>> > >> > > > > >> *Contact Details: *
>> > >> > > > > >> Harsha Balasubramanian
>> > >> > > > > >> Graduate Student at University of Toronto
>> > >> > > > > >> Professional Masters in Computer Science
>> > >> > > > > >>
>> > >> > > > > >> Email 1: harshab85@cs.toronto.edu
>> > >> > > > > >> Email 2: harshab85@gmail.com
>> > >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > > > > >>
>> > >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
>> > >> > > > > <evans@yahoo-inc.com.invalid
>> > >> > > > > >> > wrote:
>> > >> > > > > >>
>> > >> > > > > >>> Sure,
>> > >> > > > > >>> The main thing that storm is missing right now is an
>> metrics
>> > >> > > feedback
>> > >> > > > > >>> system to the scheduler (or possibly a separate piece of
>> code
>> > >> that
>> > >> > > > > decides
>> > >> > > > > >>> when to grow/shrink a topology). We have some basic
>> metrics,
>> > >> but
>> > >> > > they
>> > >> > > > > are
>> > >> > > > > >>> not exposed to the scheduler. The other question is how
>> do
>> > >> we deal
>> > >> > > > > with
>> > >> > > > > >>> creating/destroying new bolts, especially around dealing
>> with
>> > >> > > different
>> > >> > > > > >>> groupings. Some groupings make it difficult. There a
>> number
>> > >> of
>> > >> > > ways
>> > >> > > > > to
>> > >> > > > > >>> get around this, but I think the simplest way is to not
>> > >> > > create/destroy
>> > >> > > > > >>> instances, but to treat it as a resources problem, and
>> give
>> > >> them
>> > >> > > more
>> > >> > > > > or
>> > >> > > > > >>> less resources as needed. But that still needs to be
>> > >> discussed. -
>> > >> > > > > Bobby
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
>> > >> > > Balasubramanian <
>> > >> > > > > >>> harshab85@gmail.com> wrote:
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>> Hi Bobby,
>> > >> > > > > >>>
>> > >> > > > > >>> Thanks for getting back to me. It is encouraging to hear
>> > >> that this
>> > >> > > > > will
>> > >> > > > > >>> be a good addition to Storm if done well.
>> > >> > > > > >>>
>> > >> > > > > >>> I have not implemented the changes yet. My project just
>> > >> started.
>> > >> > > It
>> > >> > > > > >>> spans
>> > >> > > > > >>> over the next 2 months. In a few days, I will create a
>> JIRA
>> > >> task
>> > >> > > and
>> > >> > > > > >>> submit
>> > >> > > > > >>> my proposal. It would be great to brainstorm with the
>> Storm
>> > >> > > community
>> > >> > > > > and
>> > >> > > > > >>> iron out my design.
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>> Thanks,
>> > >> > > > > >>> Harsha
>> > >> > > > > >>>
>> > >> > > > > >>> Thanks,
>> > >> > > > > >>> Harsha
>> > >> > > > > >>>
>> > >> > > > > >>> *Contact Details: *
>> > >> > > > > >>> Harsha Balasubramanian
>> > >> > > > > >>> Graduate Student at University of Toronto
>> > >> > > > > >>> Professional Masters in Computer Science
>> > >> > > > > >>>
>> > >> > > > > >>> Email 1: harshab85@cs.toronto.edu
>> > >> > > > > >>> Email 2: harshab85@gmail.com
>> > >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > > > > >>>
>> > >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
>> > >> > > > > <evans@yahoo-inc.com.invalid
>> > >> > > > > >>> >
>> > >> > > > > >>> wrote:
>> > >> > > > > >>>
>> > >> > > > > >>> > Yes, this type of a feature would be great to have. I
>> am
>> > >> rally
>> > >> > > > > curious
>> > >> > > > > >>> > how you have done the changes, as there are a lot of
>> > >> potential
>> > >> > > > > pitfalls
>> > >> > > > > >>> > here. At a minimum it would just be great to have the
>> > >> feedback
>> > >> > > > > >>> framework
>> > >> > > > > >>> > in place so we can iterate on these changes.
>> > >> > > > > >>> > - Bobby
>> > >> > > > > >>> >
>> > >> > > > > >>> >
>> > >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
>> > >> > > > > storm@harsha.io>
>> > >> > > > > >>> > wrote:
>> > >> > > > > >>> >
>> > >> > > > > >>> >
>> > >> > > > > >>> > Hi,
>> > >> > > > > >>> > It will definitely interesting to the storm
>> community.
>> > >> It
>> > >> > > will be
>> > >> > > > > >>> > great if you can create a JIRA and post your code
>> as PR
>> > >> for
>> > >> > > others
>> > >> > > > > >>> > to try out and review the code.
>> > >> > > > > >>> > Thanks,
>> > >> > > > > >>> > Harsha
>> > >> > > > > >>> >
>> > >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha
>> Balasubramanian
>> > >> wrote:
>> > >> > > > > >>> > > Please let me know if my project (outlined below)
>> will be
>> > >> > > useful to
>> > >> > > > > >>> > > Storm.
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > Thanks,
>> > >> > > > > >>> > > Harsha
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > *Contact Details: *
>> > >> > > > > >>> > > Harsha Balasubramanian
>> > >> > > > > >>> > > Graduate Student at University of Toronto
>> > >> > > > > >>> > > Professional Masters in Computer Science
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > >> > > > > >>> > > Email 2: harshab85@gmail.com
>> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > ---------- Forwarded message ----------
>> > >> > > > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
>> > >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
>> > >> > > > > >>> > > Subject: Auto-Scaling
>> > >> > > > > >>> > > To: dev@storm.apache.org
>> > >> > > > > >>> > >
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > Hi,
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > I am a Graduate student at the University of
>> Toronto. As
>> > >> part
>> > >> > > of
>> > >> > > > > my
>> > >> > > > > >>> > > Advanced Database Systems course, I have proposed to
>> > >> implement
>> > >> > > an
>> > >> > > > > >>> Auto
>> > >> > > > > >>> > > Scaling mechanism for Storm topologies (using a
>> Feedback
>> > >> > > System).
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > I've gone through the pages on how to contribute to
>> the
>> > >> Storm
>> > >> > > > > >>> project and
>> > >> > > > > >>> > > have some questions. Please let me know if
>> auto-scaling is
>> > >> > > being
>> > >> > > > > >>> worked
>> > >> > > > > >>> > > on
>> > >> > > > > >>> > > currently. Also, should this be a project in
>> StormCore or
>> > >> > > > > >>> StormProcessor
>> > >> > > > > >>> > > ?
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > If this project will be a good addition to Storm, I
>> will
>> > >> > > create a
>> > >> > > > > new
>> > >> > > > > >>> > > JIRA
>> > >> > > > > >>> > > task for it and add the details of my proposed
>> > >> implementation.
>> > >> > > > > >>> Please do
>> > >> > > > > >>> > > let me know.
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > Thanks,
>> > >> > > > > >>> > > Harsha
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > *Contact Details: *
>> > >> > > > > >>> > > Harsha Balasubramanian
>> > >> > > > > >>> > > Graduate Student at University of Toronto
>> > >> > > > > >>> > > Professional Masters in Computer Science
>> > >> > > > > >>> > >
>> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > >> > > > > >>> > > Email 2: harshab85@gmail.com
>> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >> > > > > >>> >
>> > >> > > > > >>> >
>> > >> > > > > >>> >
>> > >> > > > > >>> >
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>
>> > >> > > > > >>
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > >
>> > >>
>> > >
>> > >
>>
>
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
I created a fork from master. Then a new branch for my changes. This is
the code i used to build my local release version.
https://github.com/harshab85/storm/tree/autoscaling
Followed instructions from here:
http://stackoverflow.com/questions/22559218/how-to-make-a-storm-release-from-the-source-code
Moved to 'storm-dist/binary' directory. Ran mvn package. Deployed the
release bundle that was created in 'storm-dist/binary/target' directory
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Fri, Dec 26, 2014 at 5:25 PM, Harsha <st...@harsha.io> wrote:
> your build is older version of storm. Do a git pull and rebuild the
> storm.
> I don't see STORM-487 fix in your build.
>
> https://github.com/apache/storm/blob/master/bin/storm
> that is the latest version and I tested in windows 8 don't see any
> issues.
> -Harsha
> On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
> > FYI:
> >
> > Running 'storm nimbus', 'storm ui' and 'storm supervisor' works fine from
> > the STORM_HOME dir.
> >
> > The problem comes only when I try to run using python to run these
> > commands.
> >
> > I want to use python only because I need the CLI tool to re-balance the
> > topology that I am using. In windows, storm.cmd apparently has a bug
> > w.r.t
> > re-balancing.
> >
> > Should I do anything extra to make the storm client work with python.
> > Python is in the PATH variable and works fine otherwise. Any extra
> > environment variable needed in windows?
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
> > <harshab85@gmail.com
> > > wrote:
> >
> > > I built the storm deliverable from main branch (g drive link to tar and
> > > zip files),
> > >
> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
> > >
> > > Then I deployed it into my c: drive - c:/storm (which is STORM_HOME)
> > >
> > > When I run the python script from STORM_HOME, I get the same error as
> > > before. *Is there any setting that distinguishes the build as a release
> > > bundle*
> > >
> > > H@H-PC /c/storm
> > > $ python ./bin/storm nimbus
> > > ******************************************
> > > The storm client can only be run from within a release. You appear to
> be
> > > trying to run the client from a checkout of Storm's source code.
> > >
> > > You can download a Storm release at
> > > http://storm-project.net/downloads.html
> > > ******************************************
> > >
> > > H@H-PC /c/storm
> > > $
> > >
> > >
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Masters in Computer Science
> > >
> > > Mobile : 647-771-3596
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
> > >
> > >> Yes he meant "master".
> > >> On your previous mail
> > >> "When I tried, c:\python>python c:\storm\bin\storm list, I got this
> > >> error"
> > >> you need to build storm and deploy . Running it from source doesn't
> > >> work.
> > >> and the correct script is storm\bin\storm like you are using above.
> > >>
> > >>
> > >>
> > >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
> > >> > Hi Bobby,
> > >> >
> > >> > When you said 'trunk', did you mean the master branch from
> > >> > https://github.com/apache/storm?
> > >> >
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > *Contact Details: *
> > >> > Harsha Balasubramanian
> > >> > Graduate Student at University of Toronto
> > >> > Masters in Computer Science
> > >> >
> > >> > Mobile : 647-771-3596
> > >> > Email 1: harshab85@cs.toronto.edu
> > >> > Email 2: harshab85@gmail.com
> > >> > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> >
> > >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
> > >> >
> > >> > > looks like there is a bug storm.cmd in make_command_arguments
> method.
> > >> > > Its converting -e count=10 to -e count 10 (removing =) which is
> > >> causing
> > >> > > this issue.
> > >> > > If you using trunk you can look STORM-487 which makes python storm
> > >> > > command work in windows.
> > >> > >
> > >> > >
> > >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> > >> > > > Thanks Bobby.
> > >> > > >
> > >> > > > I setup a local server and CLI, UI and my topology run fine.
> However
> > >> > > > there
> > >> > > > is a feature that doesn't work and I need some help to figure
> out
> > >> why.
> > >> > > >
> > >> > > > 1. I followed the instructions here to setup the storm system,
> > >> > > >
> > >> > >
> > >>
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> > >> > > >
> > >> > > > 2. My feature requires the usage of the CLI tool to re-balance
> the
> > >> > > > topology
> > >> > > > when it needs more resources. For that I thought I could use the
> > >> > > > following
> > >> > > > CLI command,
> > >> > > >
> > >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e
> yellow-bolt=10
> > >> > > >
> > >> > > > I found this in the storm documentation at the bottom of this
> page:
> > >> > > >
> > >> > >
> > >>
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> > >> > > >
> > >> > > > 3. When I run this command in my setup, I get the error pasted
> > >> below. Is
> > >> > > > re-balancing supported at the spouts and bolts level? Am I
> using the
> > >> > > > wrong
> > >> > > > version of the storm jar.
> > >> > > >
> > >> > > > This works:
> > >> > > > *C:\storm>storm rebalance WordCount -n 15*
> > >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus
> at
> > >> > > > localhost:6627*
> > >> > > > *3573 [main] INFO backtype.storm.command.rebalance - Topology
> > >> WordCount
> > >> > > > is
> > >> > > > rebalancing*
> > >> > > >
> > >> > > > This does not work:
> > >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> > >> > > > *Exception in thread "main"
> > >> java.lang.StringIndexOutOfBoundsException:
> > >> > > > String index out of range: -1*
> > >> > > > * at java.lang.String.substring(Unknown Source)*
> > >> > > > * at
> > >> > > >
> > >>
> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> > >> > > > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> > >> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> > >> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> > >> > > > * at
> > >> > > >
> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> > >> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> > >> > > > * at backtype.storm.command.rebalance.main(Unknown
> Source)*
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Harsha
> > >> > > >
> > >> > > > *Contact Details: *
> > >> > > > Harsha Balasubramanian
> > >> > > > Graduate Student at University of Toronto
> > >> > > > Masters in Computer Science
> > >> > > >
> > >> > > > Mobile : 647-771-3596
> > >> > > > Email 1: harshab85@cs.toronto.edu
> > >> > > > Email 2: harshab85@gmail.com
> > >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > > >
> > >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> > >> > > > <ev...@yahoo-inc.com.invalid>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Harsha,
> > >> > > > > Sorry about that, I have been out at a conference.
> > >> > > > >
> > >> > > > > for 1 you can get the number of acks so far, but you are
> going to
> > >> have
> > >> > > to
> > >> > > > > calculate the rate yourself, it keeps track of the number of
> acks
> > >> over
> > >> > > > > several time windows, but reports them as whole numbers.
> > >> > > > > What do you mean by a local cluster? Do you mean a single
> node
> > >> cluster
> > >> > > > > you launched yourself? then yes. If you mean a local-mode
> > >> cluster,
> > >> > > then no.
> > >> > > > > - Bobby
> > >> > > > >
> > >> > > > >
> > >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
> > >> Balasubramanian <
> > >> > > > > harshab85@gmail.com> wrote:
> > >> > > > >
> > >> > > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > Can someone pls answer my questions from the above thread.
> > >> > > > >
> > >> > > > >
> > >> > > > > *I have 2 questions for you;*
> > >> > > > > *1. In one of your earlier emails, you mentioned that there is
> > >> > > > > a limited Metrics implementation which is available in the
> current
> > >> > > Storm
> > >> > > > > version. *
> > >> > > > >
> > >> > > > > *Will this provide 'acks threshold', i.e, the number of acks
> per
> > >> > > second (or
> > >> > > > > minute)? If not, can you please suggest a way in which I can
> > >> implement
> > >> > > this
> > >> > > > > myself. *
> > >> > > > >
> > >> > > > > *2. Is it possible to use the CLI and UI tools from a local
> > >> cluster?*
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Harsha
> > >> > > > >
> > >> > > > > *Contact Details: *
> > >> > > > > Harsha Balasubramanian
> > >> > > > > Graduate Student at University of Toronto
> > >> > > > > Masters in Computer Science
> > >> > > > >
> > >> > > > > Mobile : 647-771-3596
> > >> > > > > Email 1: harshab85@cs.toronto.edu
> > >> > > > > Email 2: harshab85@gmail.com
> > >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > > > >
> > >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> > >> > > > > harshab85@gmail.com> wrote:
> > >> > > > > >
> > >> > > > > > Hi Bobby,
> > >> > > > > >
> > >> > > > > > Hope this email finds you well. I.m about to start
> designing
> > >> the
> > >> > > > > > auto-scaling system. Created a JIRA for it,
> > >> > > > > > https://issues.apache.org/jira/browse/STORM-594
> > >> > > > > >
> > >> > > > > > Please take a look and let me know if there are any
> concerns.
> > >> > > > > >
> > >> > > > > > I have 2 questions for you;
> > >> > > > > > 1. In one of your earlier emails, you mentioned that there
> is a
> > >> > > *limited
> > >> > > > > *Metrics
> > >> > > > > > implementation which is available in the current Storm
> version.
> > >> > > > > > Will this provide 'acks threshold'?
> > >> > > > > >
> > >> > > > > > 2. Is it possible to use the CLI tool from a local cluster?
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Harsha
> > >> > > > > >
> > >> > > > > > *Contact Details: *
> > >> > > > > > Harsha Balasubramanian
> > >> > > > > > Graduate Student at University of Toronto
> > >> > > > > > Masters in Computer Science
> > >> > > > > >
> > >> > > > > > Mobile : 647-771-3596
> > >> > > > > > Email 1: harshab85@cs.toronto.edu
> > >> > > > > > Email 2: harshab85@gmail.com
> > >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > > > > >
> > >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> > >> > > > > > harshab85@gmail.com> wrote:
> > >> > > > > >>
> > >> > > > > >> Thanks for the detailed explanation Bobby. I will keep
> this in
> > >> mind
> > >> > > when
> > >> > > > > >> I create my design document.
> > >> > > > > >>
> > >> > > > > >> I will mostly not deal with adding/removing bolts to the
> > >> topology
> > >> > > and
> > >> > > > > >> restrict myself to threads and tasks. This is because of
> the
> > >> time I
> > >> > > > > have to
> > >> > > > > >> complete the project. Once I submit my report (early
> January),
> > >> I can
> > >> > > > > extend
> > >> > > > > >> what I have implemented to more use cases.
> > >> > > > > >>
> > >> > > > > >> Thanks,
> > >> > > > > >> Harsha
> > >> > > > > >>
> > >> > > > > >> Thanks,
> > >> > > > > >> Harsha
> > >> > > > > >>
> > >> > > > > >> *Contact Details: *
> > >> > > > > >> Harsha Balasubramanian
> > >> > > > > >> Graduate Student at University of Toronto
> > >> > > > > >> Professional Masters in Computer Science
> > >> > > > > >>
> > >> > > > > >> Email 1: harshab85@cs.toronto.edu
> > >> > > > > >> Email 2: harshab85@gmail.com
> > >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > > > > >>
> > >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > >> > > > > <evans@yahoo-inc.com.invalid
> > >> > > > > >> > wrote:
> > >> > > > > >>
> > >> > > > > >>> Sure,
> > >> > > > > >>> The main thing that storm is missing right now is an
> metrics
> > >> > > feedback
> > >> > > > > >>> system to the scheduler (or possibly a separate piece of
> code
> > >> that
> > >> > > > > decides
> > >> > > > > >>> when to grow/shrink a topology). We have some basic
> metrics,
> > >> but
> > >> > > they
> > >> > > > > are
> > >> > > > > >>> not exposed to the scheduler. The other question is how
> do
> > >> we deal
> > >> > > > > with
> > >> > > > > >>> creating/destroying new bolts, especially around dealing
> with
> > >> > > different
> > >> > > > > >>> groupings. Some groupings make it difficult. There a
> number
> > >> of
> > >> > > ways
> > >> > > > > to
> > >> > > > > >>> get around this, but I think the simplest way is to not
> > >> > > create/destroy
> > >> > > > > >>> instances, but to treat it as a resources problem, and
> give
> > >> them
> > >> > > more
> > >> > > > > or
> > >> > > > > >>> less resources as needed. But that still needs to be
> > >> discussed. -
> > >> > > > > Bobby
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> > >> > > Balasubramanian <
> > >> > > > > >>> harshab85@gmail.com> wrote:
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>> Hi Bobby,
> > >> > > > > >>>
> > >> > > > > >>> Thanks for getting back to me. It is encouraging to hear
> > >> that this
> > >> > > > > will
> > >> > > > > >>> be a good addition to Storm if done well.
> > >> > > > > >>>
> > >> > > > > >>> I have not implemented the changes yet. My project just
> > >> started.
> > >> > > It
> > >> > > > > >>> spans
> > >> > > > > >>> over the next 2 months. In a few days, I will create a
> JIRA
> > >> task
> > >> > > and
> > >> > > > > >>> submit
> > >> > > > > >>> my proposal. It would be great to brainstorm with the
> Storm
> > >> > > community
> > >> > > > > and
> > >> > > > > >>> iron out my design.
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>> Thanks,
> > >> > > > > >>> Harsha
> > >> > > > > >>>
> > >> > > > > >>> Thanks,
> > >> > > > > >>> Harsha
> > >> > > > > >>>
> > >> > > > > >>> *Contact Details: *
> > >> > > > > >>> Harsha Balasubramanian
> > >> > > > > >>> Graduate Student at University of Toronto
> > >> > > > > >>> Professional Masters in Computer Science
> > >> > > > > >>>
> > >> > > > > >>> Email 1: harshab85@cs.toronto.edu
> > >> > > > > >>> Email 2: harshab85@gmail.com
> > >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > > > > >>>
> > >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > >> > > > > <evans@yahoo-inc.com.invalid
> > >> > > > > >>> >
> > >> > > > > >>> wrote:
> > >> > > > > >>>
> > >> > > > > >>> > Yes, this type of a feature would be great to have. I
> am
> > >> rally
> > >> > > > > curious
> > >> > > > > >>> > how you have done the changes, as there are a lot of
> > >> potential
> > >> > > > > pitfalls
> > >> > > > > >>> > here. At a minimum it would just be great to have the
> > >> feedback
> > >> > > > > >>> framework
> > >> > > > > >>> > in place so we can iterate on these changes.
> > >> > > > > >>> > - Bobby
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> > >> > > > > storm@harsha.io>
> > >> > > > > >>> > wrote:
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> > Hi,
> > >> > > > > >>> > It will definitely interesting to the storm
> community.
> > >> It
> > >> > > will be
> > >> > > > > >>> > great if you can create a JIRA and post your code as
> PR
> > >> for
> > >> > > others
> > >> > > > > >>> > to try out and review the code.
> > >> > > > > >>> > Thanks,
> > >> > > > > >>> > Harsha
> > >> > > > > >>> >
> > >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha
> Balasubramanian
> > >> wrote:
> > >> > > > > >>> > > Please let me know if my project (outlined below)
> will be
> > >> > > useful to
> > >> > > > > >>> > > Storm.
> > >> > > > > >>> > >
> > >> > > > > >>> > > Thanks,
> > >> > > > > >>> > > Harsha
> > >> > > > > >>> > >
> > >> > > > > >>> > > *Contact Details: *
> > >> > > > > >>> > > Harsha Balasubramanian
> > >> > > > > >>> > > Graduate Student at University of Toronto
> > >> > > > > >>> > > Professional Masters in Computer Science
> > >> > > > > >>> > >
> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > > > > >>> > >
> > >> > > > > >>> > > ---------- Forwarded message ----------
> > >> > > > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> > >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > >> > > > > >>> > > Subject: Auto-Scaling
> > >> > > > > >>> > > To: dev@storm.apache.org
> > >> > > > > >>> > >
> > >> > > > > >>> > >
> > >> > > > > >>> > > Hi,
> > >> > > > > >>> > >
> > >> > > > > >>> > > I am a Graduate student at the University of
> Toronto. As
> > >> part
> > >> > > of
> > >> > > > > my
> > >> > > > > >>> > > Advanced Database Systems course, I have proposed to
> > >> implement
> > >> > > an
> > >> > > > > >>> Auto
> > >> > > > > >>> > > Scaling mechanism for Storm topologies (using a
> Feedback
> > >> > > System).
> > >> > > > > >>> > >
> > >> > > > > >>> > > I've gone through the pages on how to contribute to
> the
> > >> Storm
> > >> > > > > >>> project and
> > >> > > > > >>> > > have some questions. Please let me know if
> auto-scaling is
> > >> > > being
> > >> > > > > >>> worked
> > >> > > > > >>> > > on
> > >> > > > > >>> > > currently. Also, should this be a project in
> StormCore or
> > >> > > > > >>> StormProcessor
> > >> > > > > >>> > > ?
> > >> > > > > >>> > >
> > >> > > > > >>> > > If this project will be a good addition to Storm, I
> will
> > >> > > create a
> > >> > > > > new
> > >> > > > > >>> > > JIRA
> > >> > > > > >>> > > task for it and add the details of my proposed
> > >> implementation.
> > >> > > > > >>> Please do
> > >> > > > > >>> > > let me know.
> > >> > > > > >>> > >
> > >> > > > > >>> > > Thanks,
> > >> > > > > >>> > > Harsha
> > >> > > > > >>> > >
> > >> > > > > >>> > > *Contact Details: *
> > >> > > > > >>> > > Harsha Balasubramanian
> > >> > > > > >>> > > Graduate Student at University of Toronto
> > >> > > > > >>> > > Professional Masters in Computer Science
> > >> > > > > >>> > >
> > >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > >> > > > > >>> > > Email 2: harshab85@gmail.com
> > >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > >
> > >>
> > >
> > >
>
Re: Fwd: Auto-Scaling
Posted by Harsha <st...@harsha.io>.
your build is older version of storm. Do a git pull and rebuild the
storm.
I don't see STORM-487 fix in your build.
https://github.com/apache/storm/blob/master/bin/storm
that is the latest version and I tested in windows 8 don't see any
issues.
-Harsha
On Fri, Dec 26, 2014, at 02:16 PM, Harsha Balasubramanian wrote:
> FYI:
>
> Running 'storm nimbus', 'storm ui' and 'storm supervisor' works fine from
> the STORM_HOME dir.
>
> The problem comes only when I try to run using python to run these
> commands.
>
> I want to use python only because I need the CLI tool to re-balance the
> topology that I am using. In windows, storm.cmd apparently has a bug
> w.r.t
> re-balancing.
>
> Should I do anything extra to make the storm client work with python.
> Python is in the PATH variable and works fine otherwise. Any extra
> environment variable needed in windows?
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian
> <harshab85@gmail.com
> > wrote:
>
> > I built the storm deliverable from main branch (g drive link to tar and
> > zip files),
> > https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
> >
> > Then I deployed it into my c: drive - c:/storm (which is STORM_HOME)
> >
> > When I run the python script from STORM_HOME, I get the same error as
> > before. *Is there any setting that distinguishes the build as a release
> > bundle*
> >
> > H@H-PC /c/storm
> > $ python ./bin/storm nimbus
> > ******************************************
> > The storm client can only be run from within a release. You appear to be
> > trying to run the client from a checkout of Storm's source code.
> >
> > You can download a Storm release at
> > http://storm-project.net/downloads.html
> > ******************************************
> >
> > H@H-PC /c/storm
> > $
> >
> >
> >
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
> >
> >> Yes he meant "master".
> >> On your previous mail
> >> "When I tried, c:\python>python c:\storm\bin\storm list, I got this
> >> error"
> >> you need to build storm and deploy . Running it from source doesn't
> >> work.
> >> and the correct script is storm\bin\storm like you are using above.
> >>
> >>
> >>
> >> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
> >> > Hi Bobby,
> >> >
> >> > When you said 'trunk', did you mean the master branch from
> >> > https://github.com/apache/storm?
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > *Contact Details: *
> >> > Harsha Balasubramanian
> >> > Graduate Student at University of Toronto
> >> > Masters in Computer Science
> >> >
> >> > Mobile : 647-771-3596
> >> > Email 1: harshab85@cs.toronto.edu
> >> > Email 2: harshab85@gmail.com
> >> > LinkedIn : ca.linkedin.com/in/harshabala/
> >> >
> >> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
> >> >
> >> > > looks like there is a bug storm.cmd in make_command_arguments method.
> >> > > Its converting -e count=10 to -e count 10 (removing =) which is
> >> causing
> >> > > this issue.
> >> > > If you using trunk you can look STORM-487 which makes python storm
> >> > > command work in windows.
> >> > >
> >> > >
> >> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> >> > > > Thanks Bobby.
> >> > > >
> >> > > > I setup a local server and CLI, UI and my topology run fine. However
> >> > > > there
> >> > > > is a feature that doesn't work and I need some help to figure out
> >> why.
> >> > > >
> >> > > > 1. I followed the instructions here to setup the storm system,
> >> > > >
> >> > >
> >> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> >> > > >
> >> > > > 2. My feature requires the usage of the CLI tool to re-balance the
> >> > > > topology
> >> > > > when it needs more resources. For that I thought I could use the
> >> > > > following
> >> > > > CLI command,
> >> > > >
> >> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
> >> > > >
> >> > > > I found this in the storm documentation at the bottom of this page:
> >> > > >
> >> > >
> >> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> >> > > >
> >> > > > 3. When I run this command in my setup, I get the error pasted
> >> below. Is
> >> > > > re-balancing supported at the spouts and bolts level? Am I using the
> >> > > > wrong
> >> > > > version of the storm jar.
> >> > > >
> >> > > > This works:
> >> > > > *C:\storm>storm rebalance WordCount -n 15*
> >> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
> >> > > > localhost:6627*
> >> > > > *3573 [main] INFO backtype.storm.command.rebalance - Topology
> >> WordCount
> >> > > > is
> >> > > > rebalancing*
> >> > > >
> >> > > > This does not work:
> >> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> >> > > > *Exception in thread "main"
> >> java.lang.StringIndexOutOfBoundsException:
> >> > > > String index out of range: -1*
> >> > > > * at java.lang.String.substring(Unknown Source)*
> >> > > > * at
> >> > > >
> >> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> >> > > > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> >> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> >> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> >> > > > * at
> >> > > > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> >> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> >> > > > * at backtype.storm.command.rebalance.main(Unknown Source)*
> >> > > >
> >> > > >
> >> > > >
> >> > > > Thanks,
> >> > > > Harsha
> >> > > >
> >> > > > *Contact Details: *
> >> > > > Harsha Balasubramanian
> >> > > > Graduate Student at University of Toronto
> >> > > > Masters in Computer Science
> >> > > >
> >> > > > Mobile : 647-771-3596
> >> > > > Email 1: harshab85@cs.toronto.edu
> >> > > > Email 2: harshab85@gmail.com
> >> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > > >
> >> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> >> > > > <ev...@yahoo-inc.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > > > Harsha,
> >> > > > > Sorry about that, I have been out at a conference.
> >> > > > >
> >> > > > > for 1 you can get the number of acks so far, but you are going to
> >> have
> >> > > to
> >> > > > > calculate the rate yourself, it keeps track of the number of acks
> >> over
> >> > > > > several time windows, but reports them as whole numbers.
> >> > > > > What do you mean by a local cluster? Do you mean a single node
> >> cluster
> >> > > > > you launched yourself? then yes. If you mean a local-mode
> >> cluster,
> >> > > then no.
> >> > > > > - Bobby
> >> > > > >
> >> > > > >
> >> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
> >> Balasubramanian <
> >> > > > > harshab85@gmail.com> wrote:
> >> > > > >
> >> > > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > Can someone pls answer my questions from the above thread.
> >> > > > >
> >> > > > >
> >> > > > > *I have 2 questions for you;*
> >> > > > > *1. In one of your earlier emails, you mentioned that there is
> >> > > > > a limited Metrics implementation which is available in the current
> >> > > Storm
> >> > > > > version. *
> >> > > > >
> >> > > > > *Will this provide 'acks threshold', i.e, the number of acks per
> >> > > second (or
> >> > > > > minute)? If not, can you please suggest a way in which I can
> >> implement
> >> > > this
> >> > > > > myself. *
> >> > > > >
> >> > > > > *2. Is it possible to use the CLI and UI tools from a local
> >> cluster?*
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Harsha
> >> > > > >
> >> > > > > *Contact Details: *
> >> > > > > Harsha Balasubramanian
> >> > > > > Graduate Student at University of Toronto
> >> > > > > Masters in Computer Science
> >> > > > >
> >> > > > > Mobile : 647-771-3596
> >> > > > > Email 1: harshab85@cs.toronto.edu
> >> > > > > Email 2: harshab85@gmail.com
> >> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > > > >
> >> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> >> > > > > harshab85@gmail.com> wrote:
> >> > > > > >
> >> > > > > > Hi Bobby,
> >> > > > > >
> >> > > > > > Hope this email finds you well. I.m about to start designing
> >> the
> >> > > > > > auto-scaling system. Created a JIRA for it,
> >> > > > > > https://issues.apache.org/jira/browse/STORM-594
> >> > > > > >
> >> > > > > > Please take a look and let me know if there are any concerns.
> >> > > > > >
> >> > > > > > I have 2 questions for you;
> >> > > > > > 1. In one of your earlier emails, you mentioned that there is a
> >> > > *limited
> >> > > > > *Metrics
> >> > > > > > implementation which is available in the current Storm version.
> >> > > > > > Will this provide 'acks threshold'?
> >> > > > > >
> >> > > > > > 2. Is it possible to use the CLI tool from a local cluster?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Harsha
> >> > > > > >
> >> > > > > > *Contact Details: *
> >> > > > > > Harsha Balasubramanian
> >> > > > > > Graduate Student at University of Toronto
> >> > > > > > Masters in Computer Science
> >> > > > > >
> >> > > > > > Mobile : 647-771-3596
> >> > > > > > Email 1: harshab85@cs.toronto.edu
> >> > > > > > Email 2: harshab85@gmail.com
> >> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > > > > >
> >> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> >> > > > > > harshab85@gmail.com> wrote:
> >> > > > > >>
> >> > > > > >> Thanks for the detailed explanation Bobby. I will keep this in
> >> mind
> >> > > when
> >> > > > > >> I create my design document.
> >> > > > > >>
> >> > > > > >> I will mostly not deal with adding/removing bolts to the
> >> topology
> >> > > and
> >> > > > > >> restrict myself to threads and tasks. This is because of the
> >> time I
> >> > > > > have to
> >> > > > > >> complete the project. Once I submit my report (early January),
> >> I can
> >> > > > > extend
> >> > > > > >> what I have implemented to more use cases.
> >> > > > > >>
> >> > > > > >> Thanks,
> >> > > > > >> Harsha
> >> > > > > >>
> >> > > > > >> Thanks,
> >> > > > > >> Harsha
> >> > > > > >>
> >> > > > > >> *Contact Details: *
> >> > > > > >> Harsha Balasubramanian
> >> > > > > >> Graduate Student at University of Toronto
> >> > > > > >> Professional Masters in Computer Science
> >> > > > > >>
> >> > > > > >> Email 1: harshab85@cs.toronto.edu
> >> > > > > >> Email 2: harshab85@gmail.com
> >> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> >> > > > > >>
> >> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> >> > > > > <evans@yahoo-inc.com.invalid
> >> > > > > >> > wrote:
> >> > > > > >>
> >> > > > > >>> Sure,
> >> > > > > >>> The main thing that storm is missing right now is an metrics
> >> > > feedback
> >> > > > > >>> system to the scheduler (or possibly a separate piece of code
> >> that
> >> > > > > decides
> >> > > > > >>> when to grow/shrink a topology). We have some basic metrics,
> >> but
> >> > > they
> >> > > > > are
> >> > > > > >>> not exposed to the scheduler. The other question is how do
> >> we deal
> >> > > > > with
> >> > > > > >>> creating/destroying new bolts, especially around dealing with
> >> > > different
> >> > > > > >>> groupings. Some groupings make it difficult. There a number
> >> of
> >> > > ways
> >> > > > > to
> >> > > > > >>> get around this, but I think the simplest way is to not
> >> > > create/destroy
> >> > > > > >>> instances, but to treat it as a resources problem, and give
> >> them
> >> > > more
> >> > > > > or
> >> > > > > >>> less resources as needed. But that still needs to be
> >> discussed. -
> >> > > > > Bobby
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> >> > > Balasubramanian <
> >> > > > > >>> harshab85@gmail.com> wrote:
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>> Hi Bobby,
> >> > > > > >>>
> >> > > > > >>> Thanks for getting back to me. It is encouraging to hear
> >> that this
> >> > > > > will
> >> > > > > >>> be a good addition to Storm if done well.
> >> > > > > >>>
> >> > > > > >>> I have not implemented the changes yet. My project just
> >> started.
> >> > > It
> >> > > > > >>> spans
> >> > > > > >>> over the next 2 months. In a few days, I will create a JIRA
> >> task
> >> > > and
> >> > > > > >>> submit
> >> > > > > >>> my proposal. It would be great to brainstorm with the Storm
> >> > > community
> >> > > > > and
> >> > > > > >>> iron out my design.
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>> Thanks,
> >> > > > > >>> Harsha
> >> > > > > >>>
> >> > > > > >>> Thanks,
> >> > > > > >>> Harsha
> >> > > > > >>>
> >> > > > > >>> *Contact Details: *
> >> > > > > >>> Harsha Balasubramanian
> >> > > > > >>> Graduate Student at University of Toronto
> >> > > > > >>> Professional Masters in Computer Science
> >> > > > > >>>
> >> > > > > >>> Email 1: harshab85@cs.toronto.edu
> >> > > > > >>> Email 2: harshab85@gmail.com
> >> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> >> > > > > >>>
> >> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> >> > > > > <evans@yahoo-inc.com.invalid
> >> > > > > >>> >
> >> > > > > >>> wrote:
> >> > > > > >>>
> >> > > > > >>> > Yes, this type of a feature would be great to have. I am
> >> rally
> >> > > > > curious
> >> > > > > >>> > how you have done the changes, as there are a lot of
> >> potential
> >> > > > > pitfalls
> >> > > > > >>> > here. At a minimum it would just be great to have the
> >> feedback
> >> > > > > >>> framework
> >> > > > > >>> > in place so we can iterate on these changes.
> >> > > > > >>> > - Bobby
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> >> > > > > storm@harsha.io>
> >> > > > > >>> > wrote:
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> > Hi,
> >> > > > > >>> > It will definitely interesting to the storm community.
> >> It
> >> > > will be
> >> > > > > >>> > great if you can create a JIRA and post your code as PR
> >> for
> >> > > others
> >> > > > > >>> > to try out and review the code.
> >> > > > > >>> > Thanks,
> >> > > > > >>> > Harsha
> >> > > > > >>> >
> >> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian
> >> wrote:
> >> > > > > >>> > > Please let me know if my project (outlined below) will be
> >> > > useful to
> >> > > > > >>> > > Storm.
> >> > > > > >>> > >
> >> > > > > >>> > > Thanks,
> >> > > > > >>> > > Harsha
> >> > > > > >>> > >
> >> > > > > >>> > > *Contact Details: *
> >> > > > > >>> > > Harsha Balasubramanian
> >> > > > > >>> > > Graduate Student at University of Toronto
> >> > > > > >>> > > Professional Masters in Computer Science
> >> > > > > >>> > >
> >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> >> > > > > >>> > > Email 2: harshab85@gmail.com
> >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > > > > >>> > >
> >> > > > > >>> > > ---------- Forwarded message ----------
> >> > > > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> >> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> >> > > > > >>> > > Subject: Auto-Scaling
> >> > > > > >>> > > To: dev@storm.apache.org
> >> > > > > >>> > >
> >> > > > > >>> > >
> >> > > > > >>> > > Hi,
> >> > > > > >>> > >
> >> > > > > >>> > > I am a Graduate student at the University of Toronto. As
> >> part
> >> > > of
> >> > > > > my
> >> > > > > >>> > > Advanced Database Systems course, I have proposed to
> >> implement
> >> > > an
> >> > > > > >>> Auto
> >> > > > > >>> > > Scaling mechanism for Storm topologies (using a Feedback
> >> > > System).
> >> > > > > >>> > >
> >> > > > > >>> > > I've gone through the pages on how to contribute to the
> >> Storm
> >> > > > > >>> project and
> >> > > > > >>> > > have some questions. Please let me know if auto-scaling is
> >> > > being
> >> > > > > >>> worked
> >> > > > > >>> > > on
> >> > > > > >>> > > currently. Also, should this be a project in StormCore or
> >> > > > > >>> StormProcessor
> >> > > > > >>> > > ?
> >> > > > > >>> > >
> >> > > > > >>> > > If this project will be a good addition to Storm, I will
> >> > > create a
> >> > > > > new
> >> > > > > >>> > > JIRA
> >> > > > > >>> > > task for it and add the details of my proposed
> >> implementation.
> >> > > > > >>> Please do
> >> > > > > >>> > > let me know.
> >> > > > > >>> > >
> >> > > > > >>> > > Thanks,
> >> > > > > >>> > > Harsha
> >> > > > > >>> > >
> >> > > > > >>> > > *Contact Details: *
> >> > > > > >>> > > Harsha Balasubramanian
> >> > > > > >>> > > Graduate Student at University of Toronto
> >> > > > > >>> > > Professional Masters in Computer Science
> >> > > > > >>> > >
> >> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> >> > > > > >>> > > Email 2: harshab85@gmail.com
> >> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>
> >> > > > > >>
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > >
> >>
> >
> >
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
FYI:
Running 'storm nimbus', 'storm ui' and 'storm supervisor' works fine from
the STORM_HOME dir.
The problem comes only when I try to run using python to run these commands.
I want to use python only because I need the CLI tool to re-balance the
topology that I am using. In windows, storm.cmd apparently has a bug w.r.t
re-balancing.
Should I do anything extra to make the storm client work with python.
Python is in the PATH variable and works fine otherwise. Any extra
environment variable needed in windows?
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Fri, Dec 26, 2014 at 3:51 PM, Harsha Balasubramanian <harshab85@gmail.com
> wrote:
> I built the storm deliverable from main branch (g drive link to tar and
> zip files),
> https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
>
> Then I deployed it into my c: drive - c:/storm (which is STORM_HOME)
>
> When I run the python script from STORM_HOME, I get the same error as
> before. *Is there any setting that distinguishes the build as a release
> bundle*
>
> H@H-PC /c/storm
> $ python ./bin/storm nimbus
> ******************************************
> The storm client can only be run from within a release. You appear to be
> trying to run the client from a checkout of Storm's source code.
>
> You can download a Storm release at
> http://storm-project.net/downloads.html
> ******************************************
>
> H@H-PC /c/storm
> $
>
>
>
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
>
>> Yes he meant "master".
>> On your previous mail
>> "When I tried, c:\python>python c:\storm\bin\storm list, I got this
>> error"
>> you need to build storm and deploy . Running it from source doesn't
>> work.
>> and the correct script is storm\bin\storm like you are using above.
>>
>>
>>
>> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
>> > Hi Bobby,
>> >
>> > When you said 'trunk', did you mean the master branch from
>> > https://github.com/apache/storm?
>> >
>> > Thanks,
>> > Harsha
>> >
>> > *Contact Details: *
>> > Harsha Balasubramanian
>> > Graduate Student at University of Toronto
>> > Masters in Computer Science
>> >
>> > Mobile : 647-771-3596
>> > Email 1: harshab85@cs.toronto.edu
>> > Email 2: harshab85@gmail.com
>> > LinkedIn : ca.linkedin.com/in/harshabala/
>> >
>> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
>> >
>> > > looks like there is a bug storm.cmd in make_command_arguments method.
>> > > Its converting -e count=10 to -e count 10 (removing =) which is
>> causing
>> > > this issue.
>> > > If you using trunk you can look STORM-487 which makes python storm
>> > > command work in windows.
>> > >
>> > >
>> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
>> > > > Thanks Bobby.
>> > > >
>> > > > I setup a local server and CLI, UI and my topology run fine. However
>> > > > there
>> > > > is a feature that doesn't work and I need some help to figure out
>> why.
>> > > >
>> > > > 1. I followed the instructions here to setup the storm system,
>> > > >
>> > >
>> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
>> > > >
>> > > > 2. My feature requires the usage of the CLI tool to re-balance the
>> > > > topology
>> > > > when it needs more resources. For that I thought I could use the
>> > > > following
>> > > > CLI command,
>> > > >
>> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
>> > > >
>> > > > I found this in the storm documentation at the bottom of this page:
>> > > >
>> > >
>> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>> > > >
>> > > > 3. When I run this command in my setup, I get the error pasted
>> below. Is
>> > > > re-balancing supported at the spouts and bolts level? Am I using the
>> > > > wrong
>> > > > version of the storm jar.
>> > > >
>> > > > This works:
>> > > > *C:\storm>storm rebalance WordCount -n 15*
>> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
>> > > > localhost:6627*
>> > > > *3573 [main] INFO backtype.storm.command.rebalance - Topology
>> WordCount
>> > > > is
>> > > > rebalancing*
>> > > >
>> > > > This does not work:
>> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
>> > > > *Exception in thread "main"
>> java.lang.StringIndexOutOfBoundsException:
>> > > > String index out of range: -1*
>> > > > * at java.lang.String.substring(Unknown Source)*
>> > > > * at
>> > > >
>> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
>> > > > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
>> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
>> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
>> > > > * at
>> > > > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
>> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
>> > > > * at backtype.storm.command.rebalance.main(Unknown Source)*
>> > > >
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > > *Contact Details: *
>> > > > Harsha Balasubramanian
>> > > > Graduate Student at University of Toronto
>> > > > Masters in Computer Science
>> > > >
>> > > > Mobile : 647-771-3596
>> > > > Email 1: harshab85@cs.toronto.edu
>> > > > Email 2: harshab85@gmail.com
>> > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > >
>> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
>> > > > <ev...@yahoo-inc.com.invalid>
>> > > > wrote:
>> > > >
>> > > > > Harsha,
>> > > > > Sorry about that, I have been out at a conference.
>> > > > >
>> > > > > for 1 you can get the number of acks so far, but you are going to
>> have
>> > > to
>> > > > > calculate the rate yourself, it keeps track of the number of acks
>> over
>> > > > > several time windows, but reports them as whole numbers.
>> > > > > What do you mean by a local cluster? Do you mean a single node
>> cluster
>> > > > > you launched yourself? then yes. If you mean a local-mode
>> cluster,
>> > > then no.
>> > > > > - Bobby
>> > > > >
>> > > > >
>> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha
>> Balasubramanian <
>> > > > > harshab85@gmail.com> wrote:
>> > > > >
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > Can someone pls answer my questions from the above thread.
>> > > > >
>> > > > >
>> > > > > *I have 2 questions for you;*
>> > > > > *1. In one of your earlier emails, you mentioned that there is
>> > > > > a limited Metrics implementation which is available in the current
>> > > Storm
>> > > > > version. *
>> > > > >
>> > > > > *Will this provide 'acks threshold', i.e, the number of acks per
>> > > second (or
>> > > > > minute)? If not, can you please suggest a way in which I can
>> implement
>> > > this
>> > > > > myself. *
>> > > > >
>> > > > > *2. Is it possible to use the CLI and UI tools from a local
>> cluster?*
>> > > > >
>> > > > > Thanks,
>> > > > > Harsha
>> > > > >
>> > > > > *Contact Details: *
>> > > > > Harsha Balasubramanian
>> > > > > Graduate Student at University of Toronto
>> > > > > Masters in Computer Science
>> > > > >
>> > > > > Mobile : 647-771-3596
>> > > > > Email 1: harshab85@cs.toronto.edu
>> > > > > Email 2: harshab85@gmail.com
>> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > > >
>> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
>> > > > > harshab85@gmail.com> wrote:
>> > > > > >
>> > > > > > Hi Bobby,
>> > > > > >
>> > > > > > Hope this email finds you well. I.m about to start designing
>> the
>> > > > > > auto-scaling system. Created a JIRA for it,
>> > > > > > https://issues.apache.org/jira/browse/STORM-594
>> > > > > >
>> > > > > > Please take a look and let me know if there are any concerns.
>> > > > > >
>> > > > > > I have 2 questions for you;
>> > > > > > 1. In one of your earlier emails, you mentioned that there is a
>> > > *limited
>> > > > > *Metrics
>> > > > > > implementation which is available in the current Storm version.
>> > > > > > Will this provide 'acks threshold'?
>> > > > > >
>> > > > > > 2. Is it possible to use the CLI tool from a local cluster?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Harsha
>> > > > > >
>> > > > > > *Contact Details: *
>> > > > > > Harsha Balasubramanian
>> > > > > > Graduate Student at University of Toronto
>> > > > > > Masters in Computer Science
>> > > > > >
>> > > > > > Mobile : 647-771-3596
>> > > > > > Email 1: harshab85@cs.toronto.edu
>> > > > > > Email 2: harshab85@gmail.com
>> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > > > >
>> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
>> > > > > > harshab85@gmail.com> wrote:
>> > > > > >>
>> > > > > >> Thanks for the detailed explanation Bobby. I will keep this in
>> mind
>> > > when
>> > > > > >> I create my design document.
>> > > > > >>
>> > > > > >> I will mostly not deal with adding/removing bolts to the
>> topology
>> > > and
>> > > > > >> restrict myself to threads and tasks. This is because of the
>> time I
>> > > > > have to
>> > > > > >> complete the project. Once I submit my report (early January),
>> I can
>> > > > > extend
>> > > > > >> what I have implemented to more use cases.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> Harsha
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> Harsha
>> > > > > >>
>> > > > > >> *Contact Details: *
>> > > > > >> Harsha Balasubramanian
>> > > > > >> Graduate Student at University of Toronto
>> > > > > >> Professional Masters in Computer Science
>> > > > > >>
>> > > > > >> Email 1: harshab85@cs.toronto.edu
>> > > > > >> Email 2: harshab85@gmail.com
>> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
>> > > > > >>
>> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
>> > > > > <evans@yahoo-inc.com.invalid
>> > > > > >> > wrote:
>> > > > > >>
>> > > > > >>> Sure,
>> > > > > >>> The main thing that storm is missing right now is an metrics
>> > > feedback
>> > > > > >>> system to the scheduler (or possibly a separate piece of code
>> that
>> > > > > decides
>> > > > > >>> when to grow/shrink a topology). We have some basic metrics,
>> but
>> > > they
>> > > > > are
>> > > > > >>> not exposed to the scheduler. The other question is how do
>> we deal
>> > > > > with
>> > > > > >>> creating/destroying new bolts, especially around dealing with
>> > > different
>> > > > > >>> groupings. Some groupings make it difficult. There a number
>> of
>> > > ways
>> > > > > to
>> > > > > >>> get around this, but I think the simplest way is to not
>> > > create/destroy
>> > > > > >>> instances, but to treat it as a resources problem, and give
>> them
>> > > more
>> > > > > or
>> > > > > >>> less resources as needed. But that still needs to be
>> discussed. -
>> > > > > Bobby
>> > > > > >>>
>> > > > > >>>
>> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
>> > > Balasubramanian <
>> > > > > >>> harshab85@gmail.com> wrote:
>> > > > > >>>
>> > > > > >>>
>> > > > > >>> Hi Bobby,
>> > > > > >>>
>> > > > > >>> Thanks for getting back to me. It is encouraging to hear
>> that this
>> > > > > will
>> > > > > >>> be a good addition to Storm if done well.
>> > > > > >>>
>> > > > > >>> I have not implemented the changes yet. My project just
>> started.
>> > > It
>> > > > > >>> spans
>> > > > > >>> over the next 2 months. In a few days, I will create a JIRA
>> task
>> > > and
>> > > > > >>> submit
>> > > > > >>> my proposal. It would be great to brainstorm with the Storm
>> > > community
>> > > > > and
>> > > > > >>> iron out my design.
>> > > > > >>>
>> > > > > >>>
>> > > > > >>> Thanks,
>> > > > > >>> Harsha
>> > > > > >>>
>> > > > > >>> Thanks,
>> > > > > >>> Harsha
>> > > > > >>>
>> > > > > >>> *Contact Details: *
>> > > > > >>> Harsha Balasubramanian
>> > > > > >>> Graduate Student at University of Toronto
>> > > > > >>> Professional Masters in Computer Science
>> > > > > >>>
>> > > > > >>> Email 1: harshab85@cs.toronto.edu
>> > > > > >>> Email 2: harshab85@gmail.com
>> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
>> > > > > >>>
>> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
>> > > > > <evans@yahoo-inc.com.invalid
>> > > > > >>> >
>> > > > > >>> wrote:
>> > > > > >>>
>> > > > > >>> > Yes, this type of a feature would be great to have. I am
>> rally
>> > > > > curious
>> > > > > >>> > how you have done the changes, as there are a lot of
>> potential
>> > > > > pitfalls
>> > > > > >>> > here. At a minimum it would just be great to have the
>> feedback
>> > > > > >>> framework
>> > > > > >>> > in place so we can iterate on these changes.
>> > > > > >>> > - Bobby
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
>> > > > > storm@harsha.io>
>> > > > > >>> > wrote:
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> > Hi,
>> > > > > >>> > It will definitely interesting to the storm community.
>> It
>> > > will be
>> > > > > >>> > great if you can create a JIRA and post your code as PR
>> for
>> > > others
>> > > > > >>> > to try out and review the code.
>> > > > > >>> > Thanks,
>> > > > > >>> > Harsha
>> > > > > >>> >
>> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian
>> wrote:
>> > > > > >>> > > Please let me know if my project (outlined below) will be
>> > > useful to
>> > > > > >>> > > Storm.
>> > > > > >>> > >
>> > > > > >>> > > Thanks,
>> > > > > >>> > > Harsha
>> > > > > >>> > >
>> > > > > >>> > > *Contact Details: *
>> > > > > >>> > > Harsha Balasubramanian
>> > > > > >>> > > Graduate Student at University of Toronto
>> > > > > >>> > > Professional Masters in Computer Science
>> > > > > >>> > >
>> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > > > > >>> > > Email 2: harshab85@gmail.com
>> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > > > >>> > >
>> > > > > >>> > > ---------- Forwarded message ----------
>> > > > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
>> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
>> > > > > >>> > > Subject: Auto-Scaling
>> > > > > >>> > > To: dev@storm.apache.org
>> > > > > >>> > >
>> > > > > >>> > >
>> > > > > >>> > > Hi,
>> > > > > >>> > >
>> > > > > >>> > > I am a Graduate student at the University of Toronto. As
>> part
>> > > of
>> > > > > my
>> > > > > >>> > > Advanced Database Systems course, I have proposed to
>> implement
>> > > an
>> > > > > >>> Auto
>> > > > > >>> > > Scaling mechanism for Storm topologies (using a Feedback
>> > > System).
>> > > > > >>> > >
>> > > > > >>> > > I've gone through the pages on how to contribute to the
>> Storm
>> > > > > >>> project and
>> > > > > >>> > > have some questions. Please let me know if auto-scaling is
>> > > being
>> > > > > >>> worked
>> > > > > >>> > > on
>> > > > > >>> > > currently. Also, should this be a project in StormCore or
>> > > > > >>> StormProcessor
>> > > > > >>> > > ?
>> > > > > >>> > >
>> > > > > >>> > > If this project will be a good addition to Storm, I will
>> > > create a
>> > > > > new
>> > > > > >>> > > JIRA
>> > > > > >>> > > task for it and add the details of my proposed
>> implementation.
>> > > > > >>> Please do
>> > > > > >>> > > let me know.
>> > > > > >>> > >
>> > > > > >>> > > Thanks,
>> > > > > >>> > > Harsha
>> > > > > >>> > >
>> > > > > >>> > > *Contact Details: *
>> > > > > >>> > > Harsha Balasubramanian
>> > > > > >>> > > Graduate Student at University of Toronto
>> > > > > >>> > > Professional Masters in Computer Science
>> > > > > >>> > >
>> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
>> > > > > >>> > > Email 2: harshab85@gmail.com
>> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > >
>>
>
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
I built the storm deliverable from main branch (g drive link to tar and zip
files),
https://drive.google.com/folderview?id=0B9th7RvhGLuiMHc5LWpobWY1czg&usp=sharing
Then I deployed it into my c: drive - c:/storm (which is STORM_HOME)
When I run the python script from STORM_HOME, I get the same error as
before. *Is there any setting that distinguishes the build as a release
bundle*
H@H-PC /c/storm
$ python ./bin/storm nimbus
******************************************
The storm client can only be run from within a release. You appear to be
trying to run the client from a checkout of Storm's source code.
You can download a Storm release at http://storm-project.net/downloads.html
******************************************
H@H-PC /c/storm
$
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Fri, Dec 26, 2014 at 12:40 PM, Harsha <ma...@harsha.io> wrote:
> Yes he meant "master".
> On your previous mail
> "When I tried, c:\python>python c:\storm\bin\storm list, I got this
> error"
> you need to build storm and deploy . Running it from source doesn't
> work.
> and the correct script is storm\bin\storm like you are using above.
>
>
>
> On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
> > Hi Bobby,
> >
> > When you said 'trunk', did you mean the master branch from
> > https://github.com/apache/storm?
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
> >
> > > looks like there is a bug storm.cmd in make_command_arguments method.
> > > Its converting -e count=10 to -e count 10 (removing =) which is causing
> > > this issue.
> > > If you using trunk you can look STORM-487 which makes python storm
> > > command work in windows.
> > >
> > >
> > > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> > > > Thanks Bobby.
> > > >
> > > > I setup a local server and CLI, UI and my topology run fine. However
> > > > there
> > > > is a feature that doesn't work and I need some help to figure out
> why.
> > > >
> > > > 1. I followed the instructions here to setup the storm system,
> > > >
> > >
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> > > >
> > > > 2. My feature requires the usage of the CLI tool to re-balance the
> > > > topology
> > > > when it needs more resources. For that I thought I could use the
> > > > following
> > > > CLI command,
> > > >
> > > > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
> > > >
> > > > I found this in the storm documentation at the bottom of this page:
> > > >
> > >
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> > > >
> > > > 3. When I run this command in my setup, I get the error pasted
> below. Is
> > > > re-balancing supported at the spouts and bolts level? Am I using the
> > > > wrong
> > > > version of the storm jar.
> > > >
> > > > This works:
> > > > *C:\storm>storm rebalance WordCount -n 15*
> > > > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
> > > > localhost:6627*
> > > > *3573 [main] INFO backtype.storm.command.rebalance - Topology
> WordCount
> > > > is
> > > > rebalancing*
> > > >
> > > > This does not work:
> > > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> > > > *Exception in thread "main"
> java.lang.StringIndexOutOfBoundsException:
> > > > String index out of range: -1*
> > > > * at java.lang.String.substring(Unknown Source)*
> > > > * at
> > > >
> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> > > > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> > > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> > > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> > > > * at
> > > > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> > > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> > > > * at backtype.storm.command.rebalance.main(Unknown Source)*
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > *Contact Details: *
> > > > Harsha Balasubramanian
> > > > Graduate Student at University of Toronto
> > > > Masters in Computer Science
> > > >
> > > > Mobile : 647-771-3596
> > > > Email 1: harshab85@cs.toronto.edu
> > > > Email 2: harshab85@gmail.com
> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >
> > > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> > > > <ev...@yahoo-inc.com.invalid>
> > > > wrote:
> > > >
> > > > > Harsha,
> > > > > Sorry about that, I have been out at a conference.
> > > > >
> > > > > for 1 you can get the number of acks so far, but you are going to
> have
> > > to
> > > > > calculate the rate yourself, it keeps track of the number of acks
> over
> > > > > several time windows, but reports them as whole numbers.
> > > > > What do you mean by a local cluster? Do you mean a single node
> cluster
> > > > > you launched yourself? then yes. If you mean a local-mode cluster,
> > > then no.
> > > > > - Bobby
> > > > >
> > > > >
> > > > > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian
> <
> > > > > harshab85@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > Can someone pls answer my questions from the above thread.
> > > > >
> > > > >
> > > > > *I have 2 questions for you;*
> > > > > *1. In one of your earlier emails, you mentioned that there is
> > > > > a limited Metrics implementation which is available in the current
> > > Storm
> > > > > version. *
> > > > >
> > > > > *Will this provide 'acks threshold', i.e, the number of acks per
> > > second (or
> > > > > minute)? If not, can you please suggest a way in which I can
> implement
> > > this
> > > > > myself. *
> > > > >
> > > > > *2. Is it possible to use the CLI and UI tools from a local
> cluster?*
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > *Contact Details: *
> > > > > Harsha Balasubramanian
> > > > > Graduate Student at University of Toronto
> > > > > Masters in Computer Science
> > > > >
> > > > > Mobile : 647-771-3596
> > > > > Email 1: harshab85@cs.toronto.edu
> > > > > Email 2: harshab85@gmail.com
> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > > >
> > > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> > > > > harshab85@gmail.com> wrote:
> > > > > >
> > > > > > Hi Bobby,
> > > > > >
> > > > > > Hope this email finds you well. I.m about to start designing the
> > > > > > auto-scaling system. Created a JIRA for it,
> > > > > > https://issues.apache.org/jira/browse/STORM-594
> > > > > >
> > > > > > Please take a look and let me know if there are any concerns.
> > > > > >
> > > > > > I have 2 questions for you;
> > > > > > 1. In one of your earlier emails, you mentioned that there is a
> > > *limited
> > > > > *Metrics
> > > > > > implementation which is available in the current Storm version.
> > > > > > Will this provide 'acks threshold'?
> > > > > >
> > > > > > 2. Is it possible to use the CLI tool from a local cluster?
> > > > > >
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > > *Contact Details: *
> > > > > > Harsha Balasubramanian
> > > > > > Graduate Student at University of Toronto
> > > > > > Masters in Computer Science
> > > > > >
> > > > > > Mobile : 647-771-3596
> > > > > > Email 1: harshab85@cs.toronto.edu
> > > > > > Email 2: harshab85@gmail.com
> > > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > > > >
> > > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> > > > > > harshab85@gmail.com> wrote:
> > > > > >>
> > > > > >> Thanks for the detailed explanation Bobby. I will keep this in
> mind
> > > when
> > > > > >> I create my design document.
> > > > > >>
> > > > > >> I will mostly not deal with adding/removing bolts to the
> topology
> > > and
> > > > > >> restrict myself to threads and tasks. This is because of the
> time I
> > > > > have to
> > > > > >> complete the project. Once I submit my report (early January),
> I can
> > > > > extend
> > > > > >> what I have implemented to more use cases.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Harsha
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Harsha
> > > > > >>
> > > > > >> *Contact Details: *
> > > > > >> Harsha Balasubramanian
> > > > > >> Graduate Student at University of Toronto
> > > > > >> Professional Masters in Computer Science
> > > > > >>
> > > > > >> Email 1: harshab85@cs.toronto.edu
> > > > > >> Email 2: harshab85@gmail.com
> > > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > > > > >>
> > > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > > > > <evans@yahoo-inc.com.invalid
> > > > > >> > wrote:
> > > > > >>
> > > > > >>> Sure,
> > > > > >>> The main thing that storm is missing right now is an metrics
> > > feedback
> > > > > >>> system to the scheduler (or possibly a separate piece of code
> that
> > > > > decides
> > > > > >>> when to grow/shrink a topology). We have some basic metrics,
> but
> > > they
> > > > > are
> > > > > >>> not exposed to the scheduler. The other question is how do we
> deal
> > > > > with
> > > > > >>> creating/destroying new bolts, especially around dealing with
> > > different
> > > > > >>> groupings. Some groupings make it difficult. There a number
> of
> > > ways
> > > > > to
> > > > > >>> get around this, but I think the simplest way is to not
> > > create/destroy
> > > > > >>> instances, but to treat it as a resources problem, and give
> them
> > > more
> > > > > or
> > > > > >>> less resources as needed. But that still needs to be
> discussed. -
> > > > > Bobby
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> > > Balasubramanian <
> > > > > >>> harshab85@gmail.com> wrote:
> > > > > >>>
> > > > > >>>
> > > > > >>> Hi Bobby,
> > > > > >>>
> > > > > >>> Thanks for getting back to me. It is encouraging to hear that
> this
> > > > > will
> > > > > >>> be a good addition to Storm if done well.
> > > > > >>>
> > > > > >>> I have not implemented the changes yet. My project just
> started.
> > > It
> > > > > >>> spans
> > > > > >>> over the next 2 months. In a few days, I will create a JIRA
> task
> > > and
> > > > > >>> submit
> > > > > >>> my proposal. It would be great to brainstorm with the Storm
> > > community
> > > > > and
> > > > > >>> iron out my design.
> > > > > >>>
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Harsha
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Harsha
> > > > > >>>
> > > > > >>> *Contact Details: *
> > > > > >>> Harsha Balasubramanian
> > > > > >>> Graduate Student at University of Toronto
> > > > > >>> Professional Masters in Computer Science
> > > > > >>>
> > > > > >>> Email 1: harshab85@cs.toronto.edu
> > > > > >>> Email 2: harshab85@gmail.com
> > > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > > > > >>>
> > > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > > > > <evans@yahoo-inc.com.invalid
> > > > > >>> >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> > Yes, this type of a feature would be great to have. I am
> rally
> > > > > curious
> > > > > >>> > how you have done the changes, as there are a lot of
> potential
> > > > > pitfalls
> > > > > >>> > here. At a minimum it would just be great to have the
> feedback
> > > > > >>> framework
> > > > > >>> > in place so we can iterate on these changes.
> > > > > >>> > - Bobby
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> > > > > storm@harsha.io>
> > > > > >>> > wrote:
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > Hi,
> > > > > >>> > It will definitely interesting to the storm community. It
> > > will be
> > > > > >>> > great if you can create a JIRA and post your code as PR
> for
> > > others
> > > > > >>> > to try out and review the code.
> > > > > >>> > Thanks,
> > > > > >>> > Harsha
> > > > > >>> >
> > > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian
> wrote:
> > > > > >>> > > Please let me know if my project (outlined below) will be
> > > useful to
> > > > > >>> > > Storm.
> > > > > >>> > >
> > > > > >>> > > Thanks,
> > > > > >>> > > Harsha
> > > > > >>> > >
> > > > > >>> > > *Contact Details: *
> > > > > >>> > > Harsha Balasubramanian
> > > > > >>> > > Graduate Student at University of Toronto
> > > > > >>> > > Professional Masters in Computer Science
> > > > > >>> > >
> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > > > >>> > > Email 2: harshab85@gmail.com
> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > > > >>> > >
> > > > > >>> > > ---------- Forwarded message ----------
> > > > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> > > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > > > > >>> > > Subject: Auto-Scaling
> > > > > >>> > > To: dev@storm.apache.org
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > > Hi,
> > > > > >>> > >
> > > > > >>> > > I am a Graduate student at the University of Toronto. As
> part
> > > of
> > > > > my
> > > > > >>> > > Advanced Database Systems course, I have proposed to
> implement
> > > an
> > > > > >>> Auto
> > > > > >>> > > Scaling mechanism for Storm topologies (using a Feedback
> > > System).
> > > > > >>> > >
> > > > > >>> > > I've gone through the pages on how to contribute to the
> Storm
> > > > > >>> project and
> > > > > >>> > > have some questions. Please let me know if auto-scaling is
> > > being
> > > > > >>> worked
> > > > > >>> > > on
> > > > > >>> > > currently. Also, should this be a project in StormCore or
> > > > > >>> StormProcessor
> > > > > >>> > > ?
> > > > > >>> > >
> > > > > >>> > > If this project will be a good addition to Storm, I will
> > > create a
> > > > > new
> > > > > >>> > > JIRA
> > > > > >>> > > task for it and add the details of my proposed
> implementation.
> > > > > >>> Please do
> > > > > >>> > > let me know.
> > > > > >>> > >
> > > > > >>> > > Thanks,
> > > > > >>> > > Harsha
> > > > > >>> > >
> > > > > >>> > > *Contact Details: *
> > > > > >>> > > Harsha Balasubramanian
> > > > > >>> > > Graduate Student at University of Toronto
> > > > > >>> > > Professional Masters in Computer Science
> > > > > >>> > >
> > > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > > > >>> > > Email 2: harshab85@gmail.com
> > > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > > > >>> >
> > > > > >>> >
> > > > > >>> >
> > > > > >>> >
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > > >
> > >
>
Re: Fwd: Auto-Scaling
Posted by Harsha <ma...@harsha.io>.
Yes he meant "master".
On your previous mail
"When I tried, c:\python>python c:\storm\bin\storm list, I got this
error"
you need to build storm and deploy . Running it from source doesn't
work.
and the correct script is storm\bin\storm like you are using above.
On Fri, Dec 26, 2014, at 09:23 AM, Harsha Balasubramanian wrote:
> Hi Bobby,
>
> When you said 'trunk', did you mean the master branch from
> https://github.com/apache/storm?
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
>
> > looks like there is a bug storm.cmd in make_command_arguments method.
> > Its converting -e count=10 to -e count 10 (removing =) which is causing
> > this issue.
> > If you using trunk you can look STORM-487 which makes python storm
> > command work in windows.
> >
> >
> > On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> > > Thanks Bobby.
> > >
> > > I setup a local server and CLI, UI and my topology run fine. However
> > > there
> > > is a feature that doesn't work and I need some help to figure out why.
> > >
> > > 1. I followed the instructions here to setup the storm system,
> > >
> > http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> > >
> > > 2. My feature requires the usage of the CLI tool to re-balance the
> > > topology
> > > when it needs more resources. For that I thought I could use the
> > > following
> > > CLI command,
> > >
> > > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
> > >
> > > I found this in the storm documentation at the bottom of this page:
> > >
> > http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> > >
> > > 3. When I run this command in my setup, I get the error pasted below. Is
> > > re-balancing supported at the spouts and bolts level? Am I using the
> > > wrong
> > > version of the storm jar.
> > >
> > > This works:
> > > *C:\storm>storm rebalance WordCount -n 15*
> > > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
> > > localhost:6627*
> > > *3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount
> > > is
> > > rebalancing*
> > >
> > > This does not work:
> > > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> > > *Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
> > > String index out of range: -1*
> > > * at java.lang.String.substring(Unknown Source)*
> > > * at
> > > backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> > > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> > > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> > > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> > > * at
> > > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> > > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> > > * at backtype.storm.command.rebalance.main(Unknown Source)*
> > >
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Masters in Computer Science
> > >
> > > Mobile : 647-771-3596
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> > > <ev...@yahoo-inc.com.invalid>
> > > wrote:
> > >
> > > > Harsha,
> > > > Sorry about that, I have been out at a conference.
> > > >
> > > > for 1 you can get the number of acks so far, but you are going to have
> > to
> > > > calculate the rate yourself, it keeps track of the number of acks over
> > > > several time windows, but reports them as whole numbers.
> > > > What do you mean by a local cluster? Do you mean a single node cluster
> > > > you launched yourself? then yes. If you mean a local-mode cluster,
> > then no.
> > > > - Bobby
> > > >
> > > >
> > > > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian <
> > > > harshab85@gmail.com> wrote:
> > > >
> > > >
> > > > Hi,
> > > >
> > > > Can someone pls answer my questions from the above thread.
> > > >
> > > >
> > > > *I have 2 questions for you;*
> > > > *1. In one of your earlier emails, you mentioned that there is
> > > > a limited Metrics implementation which is available in the current
> > Storm
> > > > version. *
> > > >
> > > > *Will this provide 'acks threshold', i.e, the number of acks per
> > second (or
> > > > minute)? If not, can you please suggest a way in which I can implement
> > this
> > > > myself. *
> > > >
> > > > *2. Is it possible to use the CLI and UI tools from a local cluster?*
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > *Contact Details: *
> > > > Harsha Balasubramanian
> > > > Graduate Student at University of Toronto
> > > > Masters in Computer Science
> > > >
> > > > Mobile : 647-771-3596
> > > > Email 1: harshab85@cs.toronto.edu
> > > > Email 2: harshab85@gmail.com
> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >
> > > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> > > > harshab85@gmail.com> wrote:
> > > > >
> > > > > Hi Bobby,
> > > > >
> > > > > Hope this email finds you well. I.m about to start designing the
> > > > > auto-scaling system. Created a JIRA for it,
> > > > > https://issues.apache.org/jira/browse/STORM-594
> > > > >
> > > > > Please take a look and let me know if there are any concerns.
> > > > >
> > > > > I have 2 questions for you;
> > > > > 1. In one of your earlier emails, you mentioned that there is a
> > *limited
> > > > *Metrics
> > > > > implementation which is available in the current Storm version.
> > > > > Will this provide 'acks threshold'?
> > > > >
> > > > > 2. Is it possible to use the CLI tool from a local cluster?
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > *Contact Details: *
> > > > > Harsha Balasubramanian
> > > > > Graduate Student at University of Toronto
> > > > > Masters in Computer Science
> > > > >
> > > > > Mobile : 647-771-3596
> > > > > Email 1: harshab85@cs.toronto.edu
> > > > > Email 2: harshab85@gmail.com
> > > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > > >
> > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> > > > > harshab85@gmail.com> wrote:
> > > > >>
> > > > >> Thanks for the detailed explanation Bobby. I will keep this in mind
> > when
> > > > >> I create my design document.
> > > > >>
> > > > >> I will mostly not deal with adding/removing bolts to the topology
> > and
> > > > >> restrict myself to threads and tasks. This is because of the time I
> > > > have to
> > > > >> complete the project. Once I submit my report (early January), I can
> > > > extend
> > > > >> what I have implemented to more use cases.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> *Contact Details: *
> > > > >> Harsha Balasubramanian
> > > > >> Graduate Student at University of Toronto
> > > > >> Professional Masters in Computer Science
> > > > >>
> > > > >> Email 1: harshab85@cs.toronto.edu
> > > > >> Email 2: harshab85@gmail.com
> > > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > > > >>
> > > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > > > <evans@yahoo-inc.com.invalid
> > > > >> > wrote:
> > > > >>
> > > > >>> Sure,
> > > > >>> The main thing that storm is missing right now is an metrics
> > feedback
> > > > >>> system to the scheduler (or possibly a separate piece of code that
> > > > decides
> > > > >>> when to grow/shrink a topology). We have some basic metrics, but
> > they
> > > > are
> > > > >>> not exposed to the scheduler. The other question is how do we deal
> > > > with
> > > > >>> creating/destroying new bolts, especially around dealing with
> > different
> > > > >>> groupings. Some groupings make it difficult. There a number of
> > ways
> > > > to
> > > > >>> get around this, but I think the simplest way is to not
> > create/destroy
> > > > >>> instances, but to treat it as a resources problem, and give them
> > more
> > > > or
> > > > >>> less resources as needed. But that still needs to be discussed. -
> > > > Bobby
> > > > >>>
> > > > >>>
> > > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> > Balasubramanian <
> > > > >>> harshab85@gmail.com> wrote:
> > > > >>>
> > > > >>>
> > > > >>> Hi Bobby,
> > > > >>>
> > > > >>> Thanks for getting back to me. It is encouraging to hear that this
> > > > will
> > > > >>> be a good addition to Storm if done well.
> > > > >>>
> > > > >>> I have not implemented the changes yet. My project just started.
> > It
> > > > >>> spans
> > > > >>> over the next 2 months. In a few days, I will create a JIRA task
> > and
> > > > >>> submit
> > > > >>> my proposal. It would be great to brainstorm with the Storm
> > community
> > > > and
> > > > >>> iron out my design.
> > > > >>>
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Harsha
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Harsha
> > > > >>>
> > > > >>> *Contact Details: *
> > > > >>> Harsha Balasubramanian
> > > > >>> Graduate Student at University of Toronto
> > > > >>> Professional Masters in Computer Science
> > > > >>>
> > > > >>> Email 1: harshab85@cs.toronto.edu
> > > > >>> Email 2: harshab85@gmail.com
> > > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > > > >>>
> > > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > > > <evans@yahoo-inc.com.invalid
> > > > >>> >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Yes, this type of a feature would be great to have. I am rally
> > > > curious
> > > > >>> > how you have done the changes, as there are a lot of potential
> > > > pitfalls
> > > > >>> > here. At a minimum it would just be great to have the feedback
> > > > >>> framework
> > > > >>> > in place so we can iterate on these changes.
> > > > >>> > - Bobby
> > > > >>> >
> > > > >>> >
> > > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> > > > storm@harsha.io>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> >
> > > > >>> > Hi,
> > > > >>> > It will definitely interesting to the storm community. It
> > will be
> > > > >>> > great if you can create a JIRA and post your code as PR for
> > others
> > > > >>> > to try out and review the code.
> > > > >>> > Thanks,
> > > > >>> > Harsha
> > > > >>> >
> > > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> > > > >>> > > Please let me know if my project (outlined below) will be
> > useful to
> > > > >>> > > Storm.
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > > Harsha
> > > > >>> > >
> > > > >>> > > *Contact Details: *
> > > > >>> > > Harsha Balasubramanian
> > > > >>> > > Graduate Student at University of Toronto
> > > > >>> > > Professional Masters in Computer Science
> > > > >>> > >
> > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > > >>> > > Email 2: harshab85@gmail.com
> > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > > >>> > >
> > > > >>> > > ---------- Forwarded message ----------
> > > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> > > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > > > >>> > > Subject: Auto-Scaling
> > > > >>> > > To: dev@storm.apache.org
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > Hi,
> > > > >>> > >
> > > > >>> > > I am a Graduate student at the University of Toronto. As part
> > of
> > > > my
> > > > >>> > > Advanced Database Systems course, I have proposed to implement
> > an
> > > > >>> Auto
> > > > >>> > > Scaling mechanism for Storm topologies (using a Feedback
> > System).
> > > > >>> > >
> > > > >>> > > I've gone through the pages on how to contribute to the Storm
> > > > >>> project and
> > > > >>> > > have some questions. Please let me know if auto-scaling is
> > being
> > > > >>> worked
> > > > >>> > > on
> > > > >>> > > currently. Also, should this be a project in StormCore or
> > > > >>> StormProcessor
> > > > >>> > > ?
> > > > >>> > >
> > > > >>> > > If this project will be a good addition to Storm, I will
> > create a
> > > > new
> > > > >>> > > JIRA
> > > > >>> > > task for it and add the details of my proposed implementation.
> > > > >>> Please do
> > > > >>> > > let me know.
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > > Harsha
> > > > >>> > >
> > > > >>> > > *Contact Details: *
> > > > >>> > > Harsha Balasubramanian
> > > > >>> > > Graduate Student at University of Toronto
> > > > >>> > > Professional Masters in Computer Science
> > > > >>> > >
> > > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > > >>> > > Email 2: harshab85@gmail.com
> > > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >>
> > > >
> > > >
> > > >
> > > >
> >
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Hi Bobby,
When you said 'trunk', did you mean the master branch from
https://github.com/apache/storm?
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote:
> looks like there is a bug storm.cmd in make_command_arguments method.
> Its converting -e count=10 to -e count 10 (removing =) which is causing
> this issue.
> If you using trunk you can look STORM-487 which makes python storm
> command work in windows.
>
>
> On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> > Thanks Bobby.
> >
> > I setup a local server and CLI, UI and my topology run fine. However
> > there
> > is a feature that doesn't work and I need some help to figure out why.
> >
> > 1. I followed the instructions here to setup the storm system,
> >
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
> >
> > 2. My feature requires the usage of the CLI tool to re-balance the
> > topology
> > when it needs more resources. For that I thought I could use the
> > following
> > CLI command,
> >
> > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
> >
> > I found this in the storm documentation at the bottom of this page:
> >
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
> >
> > 3. When I run this command in my setup, I get the error pasted below. Is
> > re-balancing supported at the spouts and bolts level? Am I using the
> > wrong
> > version of the storm jar.
> >
> > This works:
> > *C:\storm>storm rebalance WordCount -n 15*
> > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
> > localhost:6627*
> > *3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount
> > is
> > rebalancing*
> >
> > This does not work:
> > *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> > *Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
> > String index out of range: -1*
> > * at java.lang.String.substring(Unknown Source)*
> > * at
> > backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> > * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> > * at
> > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> > * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> > * at backtype.storm.command.rebalance.main(Unknown Source)*
> >
> >
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> > <ev...@yahoo-inc.com.invalid>
> > wrote:
> >
> > > Harsha,
> > > Sorry about that, I have been out at a conference.
> > >
> > > for 1 you can get the number of acks so far, but you are going to have
> to
> > > calculate the rate yourself, it keeps track of the number of acks over
> > > several time windows, but reports them as whole numbers.
> > > What do you mean by a local cluster? Do you mean a single node cluster
> > > you launched yourself? then yes. If you mean a local-mode cluster,
> then no.
> > > - Bobby
> > >
> > >
> > > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian <
> > > harshab85@gmail.com> wrote:
> > >
> > >
> > > Hi,
> > >
> > > Can someone pls answer my questions from the above thread.
> > >
> > >
> > > *I have 2 questions for you;*
> > > *1. In one of your earlier emails, you mentioned that there is
> > > a limited Metrics implementation which is available in the current
> Storm
> > > version. *
> > >
> > > *Will this provide 'acks threshold', i.e, the number of acks per
> second (or
> > > minute)? If not, can you please suggest a way in which I can implement
> this
> > > myself. *
> > >
> > > *2. Is it possible to use the CLI and UI tools from a local cluster?*
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Masters in Computer Science
> > >
> > > Mobile : 647-771-3596
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> > > harshab85@gmail.com> wrote:
> > > >
> > > > Hi Bobby,
> > > >
> > > > Hope this email finds you well. I.m about to start designing the
> > > > auto-scaling system. Created a JIRA for it,
> > > > https://issues.apache.org/jira/browse/STORM-594
> > > >
> > > > Please take a look and let me know if there are any concerns.
> > > >
> > > > I have 2 questions for you;
> > > > 1. In one of your earlier emails, you mentioned that there is a
> *limited
> > > *Metrics
> > > > implementation which is available in the current Storm version.
> > > > Will this provide 'acks threshold'?
> > > >
> > > > 2. Is it possible to use the CLI tool from a local cluster?
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > *Contact Details: *
> > > > Harsha Balasubramanian
> > > > Graduate Student at University of Toronto
> > > > Masters in Computer Science
> > > >
> > > > Mobile : 647-771-3596
> > > > Email 1: harshab85@cs.toronto.edu
> > > > Email 2: harshab85@gmail.com
> > > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >
> > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> > > > harshab85@gmail.com> wrote:
> > > >>
> > > >> Thanks for the detailed explanation Bobby. I will keep this in mind
> when
> > > >> I create my design document.
> > > >>
> > > >> I will mostly not deal with adding/removing bolts to the topology
> and
> > > >> restrict myself to threads and tasks. This is because of the time I
> > > have to
> > > >> complete the project. Once I submit my report (early January), I can
> > > extend
> > > >> what I have implemented to more use cases.
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> *Contact Details: *
> > > >> Harsha Balasubramanian
> > > >> Graduate Student at University of Toronto
> > > >> Professional Masters in Computer Science
> > > >>
> > > >> Email 1: harshab85@cs.toronto.edu
> > > >> Email 2: harshab85@gmail.com
> > > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>
> > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > > <evans@yahoo-inc.com.invalid
> > > >> > wrote:
> > > >>
> > > >>> Sure,
> > > >>> The main thing that storm is missing right now is an metrics
> feedback
> > > >>> system to the scheduler (or possibly a separate piece of code that
> > > decides
> > > >>> when to grow/shrink a topology). We have some basic metrics, but
> they
> > > are
> > > >>> not exposed to the scheduler. The other question is how do we deal
> > > with
> > > >>> creating/destroying new bolts, especially around dealing with
> different
> > > >>> groupings. Some groupings make it difficult. There a number of
> ways
> > > to
> > > >>> get around this, but I think the simplest way is to not
> create/destroy
> > > >>> instances, but to treat it as a resources problem, and give them
> more
> > > or
> > > >>> less resources as needed. But that still needs to be discussed. -
> > > Bobby
> > > >>>
> > > >>>
> > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha
> Balasubramanian <
> > > >>> harshab85@gmail.com> wrote:
> > > >>>
> > > >>>
> > > >>> Hi Bobby,
> > > >>>
> > > >>> Thanks for getting back to me. It is encouraging to hear that this
> > > will
> > > >>> be a good addition to Storm if done well.
> > > >>>
> > > >>> I have not implemented the changes yet. My project just started.
> It
> > > >>> spans
> > > >>> over the next 2 months. In a few days, I will create a JIRA task
> and
> > > >>> submit
> > > >>> my proposal. It would be great to brainstorm with the Storm
> community
> > > and
> > > >>> iron out my design.
> > > >>>
> > > >>>
> > > >>> Thanks,
> > > >>> Harsha
> > > >>>
> > > >>> Thanks,
> > > >>> Harsha
> > > >>>
> > > >>> *Contact Details: *
> > > >>> Harsha Balasubramanian
> > > >>> Graduate Student at University of Toronto
> > > >>> Professional Masters in Computer Science
> > > >>>
> > > >>> Email 1: harshab85@cs.toronto.edu
> > > >>> Email 2: harshab85@gmail.com
> > > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>>
> > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > > <evans@yahoo-inc.com.invalid
> > > >>> >
> > > >>> wrote:
> > > >>>
> > > >>> > Yes, this type of a feature would be great to have. I am rally
> > > curious
> > > >>> > how you have done the changes, as there are a lot of potential
> > > pitfalls
> > > >>> > here. At a minimum it would just be great to have the feedback
> > > >>> framework
> > > >>> > in place so we can iterate on these changes.
> > > >>> > - Bobby
> > > >>> >
> > > >>> >
> > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> > > storm@harsha.io>
> > > >>> > wrote:
> > > >>> >
> > > >>> >
> > > >>> > Hi,
> > > >>> > It will definitely interesting to the storm community. It
> will be
> > > >>> > great if you can create a JIRA and post your code as PR for
> others
> > > >>> > to try out and review the code.
> > > >>> > Thanks,
> > > >>> > Harsha
> > > >>> >
> > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> > > >>> > > Please let me know if my project (outlined below) will be
> useful to
> > > >>> > > Storm.
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Harsha
> > > >>> > >
> > > >>> > > *Contact Details: *
> > > >>> > > Harsha Balasubramanian
> > > >>> > > Graduate Student at University of Toronto
> > > >>> > > Professional Masters in Computer Science
> > > >>> > >
> > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > >>> > > Email 2: harshab85@gmail.com
> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>> > >
> > > >>> > > ---------- Forwarded message ----------
> > > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > > >>> > > Subject: Auto-Scaling
> > > >>> > > To: dev@storm.apache.org
> > > >>> > >
> > > >>> > >
> > > >>> > > Hi,
> > > >>> > >
> > > >>> > > I am a Graduate student at the University of Toronto. As part
> of
> > > my
> > > >>> > > Advanced Database Systems course, I have proposed to implement
> an
> > > >>> Auto
> > > >>> > > Scaling mechanism for Storm topologies (using a Feedback
> System).
> > > >>> > >
> > > >>> > > I've gone through the pages on how to contribute to the Storm
> > > >>> project and
> > > >>> > > have some questions. Please let me know if auto-scaling is
> being
> > > >>> worked
> > > >>> > > on
> > > >>> > > currently. Also, should this be a project in StormCore or
> > > >>> StormProcessor
> > > >>> > > ?
> > > >>> > >
> > > >>> > > If this project will be a good addition to Storm, I will
> create a
> > > new
> > > >>> > > JIRA
> > > >>> > > task for it and add the details of my proposed implementation.
> > > >>> Please do
> > > >>> > > let me know.
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Harsha
> > > >>> > >
> > > >>> > > *Contact Details: *
> > > >>> > > Harsha Balasubramanian
> > > >>> > > Graduate Student at University of Toronto
> > > >>> > > Professional Masters in Computer Science
> > > >>> > >
> > > >>> > > Email 1: harshab85@cs.toronto.edu
> > > >>> > > Email 2: harshab85@gmail.com
> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > >
> > >
> > >
> > >
>
Re: Fwd: Auto-Scaling
Posted by Harsha <st...@harsha.io>.
looks like there is a bug storm.cmd in make_command_arguments method.
Its converting -e count=10 to -e count 10 (removing =) which is causing
this issue.
If you using trunk you can look STORM-487 which makes python storm
command work in windows.
On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote:
> Thanks Bobby.
>
> I setup a local server and CLI, UI and my topology run fine. However
> there
> is a feature that doesn't work and I need some help to figure out why.
>
> 1. I followed the instructions here to setup the storm system,
> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
>
> 2. My feature requires the usage of the CLI tool to re-balance the
> topology
> when it needs more resources. For that I thought I could use the
> following
> CLI command,
>
> storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
>
> I found this in the storm documentation at the bottom of this page:
> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
>
> 3. When I run this command in my setup, I get the error pasted below. Is
> re-balancing supported at the spouts and bolts level? Am I using the
> wrong
> version of the storm jar.
>
> This works:
> *C:\storm>storm rebalance WordCount -n 15*
> *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
> localhost:6627*
> *3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount
> is
> rebalancing*
>
> This does not work:
> *C:\storm>storm rebalance WordCount -n 5 -e Count=10*
> *Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
> String index out of range: -1*
> * at java.lang.String.substring(Unknown Source)*
> * at
> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
> * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
> * at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
> * at clojure.lang.RestFn.invoke(RestFn.java:460)*
> * at
> backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
> * at clojure.lang.RestFn.applyTo(RestFn.java:137)*
> * at backtype.storm.command.rebalance.main(Unknown Source)*
>
>
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans
> <ev...@yahoo-inc.com.invalid>
> wrote:
>
> > Harsha,
> > Sorry about that, I have been out at a conference.
> >
> > for 1 you can get the number of acks so far, but you are going to have to
> > calculate the rate yourself, it keeps track of the number of acks over
> > several time windows, but reports them as whole numbers.
> > What do you mean by a local cluster? Do you mean a single node cluster
> > you launched yourself? then yes. If you mean a local-mode cluster, then no.
> > - Bobby
> >
> >
> > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian <
> > harshab85@gmail.com> wrote:
> >
> >
> > Hi,
> >
> > Can someone pls answer my questions from the above thread.
> >
> >
> > *I have 2 questions for you;*
> > *1. In one of your earlier emails, you mentioned that there is
> > a limited Metrics implementation which is available in the current Storm
> > version. *
> >
> > *Will this provide 'acks threshold', i.e, the number of acks per second (or
> > minute)? If not, can you please suggest a way in which I can implement this
> > myself. *
> >
> > *2. Is it possible to use the CLI and UI tools from a local cluster?*
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> > harshab85@gmail.com> wrote:
> > >
> > > Hi Bobby,
> > >
> > > Hope this email finds you well. I.m about to start designing the
> > > auto-scaling system. Created a JIRA for it,
> > > https://issues.apache.org/jira/browse/STORM-594
> > >
> > > Please take a look and let me know if there are any concerns.
> > >
> > > I have 2 questions for you;
> > > 1. In one of your earlier emails, you mentioned that there is a *limited
> > *Metrics
> > > implementation which is available in the current Storm version.
> > > Will this provide 'acks threshold'?
> > >
> > > 2. Is it possible to use the CLI tool from a local cluster?
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Masters in Computer Science
> > >
> > > Mobile : 647-771-3596
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> > > harshab85@gmail.com> wrote:
> > >>
> > >> Thanks for the detailed explanation Bobby. I will keep this in mind when
> > >> I create my design document.
> > >>
> > >> I will mostly not deal with adding/removing bolts to the topology and
> > >> restrict myself to threads and tasks. This is because of the time I
> > have to
> > >> complete the project. Once I submit my report (early January), I can
> > extend
> > >> what I have implemented to more use cases.
> > >>
> > >> Thanks,
> > >> Harsha
> > >>
> > >> Thanks,
> > >> Harsha
> > >>
> > >> *Contact Details: *
> > >> Harsha Balasubramanian
> > >> Graduate Student at University of Toronto
> > >> Professional Masters in Computer Science
> > >>
> > >> Email 1: harshab85@cs.toronto.edu
> > >> Email 2: harshab85@gmail.com
> > >> LinkedIn : ca.linkedin.com/in/harshabala/
> > >>
> > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> > <evans@yahoo-inc.com.invalid
> > >> > wrote:
> > >>
> > >>> Sure,
> > >>> The main thing that storm is missing right now is an metrics feedback
> > >>> system to the scheduler (or possibly a separate piece of code that
> > decides
> > >>> when to grow/shrink a topology). We have some basic metrics, but they
> > are
> > >>> not exposed to the scheduler. The other question is how do we deal
> > with
> > >>> creating/destroying new bolts, especially around dealing with different
> > >>> groupings. Some groupings make it difficult. There a number of ways
> > to
> > >>> get around this, but I think the simplest way is to not create/destroy
> > >>> instances, but to treat it as a resources problem, and give them more
> > or
> > >>> less resources as needed. But that still needs to be discussed. -
> > Bobby
> > >>>
> > >>>
> > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian <
> > >>> harshab85@gmail.com> wrote:
> > >>>
> > >>>
> > >>> Hi Bobby,
> > >>>
> > >>> Thanks for getting back to me. It is encouraging to hear that this
> > will
> > >>> be a good addition to Storm if done well.
> > >>>
> > >>> I have not implemented the changes yet. My project just started. It
> > >>> spans
> > >>> over the next 2 months. In a few days, I will create a JIRA task and
> > >>> submit
> > >>> my proposal. It would be great to brainstorm with the Storm community
> > and
> > >>> iron out my design.
> > >>>
> > >>>
> > >>> Thanks,
> > >>> Harsha
> > >>>
> > >>> Thanks,
> > >>> Harsha
> > >>>
> > >>> *Contact Details: *
> > >>> Harsha Balasubramanian
> > >>> Graduate Student at University of Toronto
> > >>> Professional Masters in Computer Science
> > >>>
> > >>> Email 1: harshab85@cs.toronto.edu
> > >>> Email 2: harshab85@gmail.com
> > >>> LinkedIn : ca.linkedin.com/in/harshabala/
> > >>>
> > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> > <evans@yahoo-inc.com.invalid
> > >>> >
> > >>> wrote:
> > >>>
> > >>> > Yes, this type of a feature would be great to have. I am rally
> > curious
> > >>> > how you have done the changes, as there are a lot of potential
> > pitfalls
> > >>> > here. At a minimum it would just be great to have the feedback
> > >>> framework
> > >>> > in place so we can iterate on these changes.
> > >>> > - Bobby
> > >>> >
> > >>> >
> > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> > storm@harsha.io>
> > >>> > wrote:
> > >>> >
> > >>> >
> > >>> > Hi,
> > >>> > It will definitely interesting to the storm community. It will be
> > >>> > great if you can create a JIRA and post your code as PR for others
> > >>> > to try out and review the code.
> > >>> > Thanks,
> > >>> > Harsha
> > >>> >
> > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> > >>> > > Please let me know if my project (outlined below) will be useful to
> > >>> > > Storm.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Harsha
> > >>> > >
> > >>> > > *Contact Details: *
> > >>> > > Harsha Balasubramanian
> > >>> > > Graduate Student at University of Toronto
> > >>> > > Professional Masters in Computer Science
> > >>> > >
> > >>> > > Email 1: harshab85@cs.toronto.edu
> > >>> > > Email 2: harshab85@gmail.com
> > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >>> > >
> > >>> > > ---------- Forwarded message ----------
> > >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > >>> > > Subject: Auto-Scaling
> > >>> > > To: dev@storm.apache.org
> > >>> > >
> > >>> > >
> > >>> > > Hi,
> > >>> > >
> > >>> > > I am a Graduate student at the University of Toronto. As part of
> > my
> > >>> > > Advanced Database Systems course, I have proposed to implement an
> > >>> Auto
> > >>> > > Scaling mechanism for Storm topologies (using a Feedback System).
> > >>> > >
> > >>> > > I've gone through the pages on how to contribute to the Storm
> > >>> project and
> > >>> > > have some questions. Please let me know if auto-scaling is being
> > >>> worked
> > >>> > > on
> > >>> > > currently. Also, should this be a project in StormCore or
> > >>> StormProcessor
> > >>> > > ?
> > >>> > >
> > >>> > > If this project will be a good addition to Storm, I will create a
> > new
> > >>> > > JIRA
> > >>> > > task for it and add the details of my proposed implementation.
> > >>> Please do
> > >>> > > let me know.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Harsha
> > >>> > >
> > >>> > > *Contact Details: *
> > >>> > > Harsha Balasubramanian
> > >>> > > Graduate Student at University of Toronto
> > >>> > > Professional Masters in Computer Science
> > >>> > >
> > >>> > > Email 1: harshab85@cs.toronto.edu
> > >>> > > Email 2: harshab85@gmail.com
> > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> >
> >
> >
> >
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Thanks Bobby.
I setup a local server and CLI, UI and my topology run fine. However there
is a feature that doesn't work and I need some help to figure out why.
1. I followed the instructions here to setup the storm system,
http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/
2. My feature requires the usage of the CLI tool to re-balance the topology
when it needs more resources. For that I thought I could use the following
CLI command,
storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10
I found this in the storm documentation at the bottom of this page:
http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
3. When I run this command in my setup, I get the error pasted below. Is
re-balancing supported at the spouts and bolts level? Am I using the wrong
version of the storm jar.
This works:
*C:\storm>storm rebalance WordCount -n 15*
*2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at
localhost:6627*
*3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount is
rebalancing*
This does not work:
*C:\storm>storm rebalance WordCount -n 5 -e Count=10*
*Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
String index out of range: -1*
* at java.lang.String.substring(Unknown Source)*
* at
backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)*
* at clojure.tools.cli$apply_specs.invoke(cli.clj:80)*
* at clojure.tools.cli$cli.doInvoke(cli.clj:130)*
* at clojure.lang.RestFn.invoke(RestFn.java:460)*
* at
backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)*
* at clojure.lang.RestFn.applyTo(RestFn.java:137)*
* at backtype.storm.command.rebalance.main(Unknown Source)*
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Harsha,
> Sorry about that, I have been out at a conference.
>
> for 1 you can get the number of acks so far, but you are going to have to
> calculate the rate yourself, it keeps track of the number of acks over
> several time windows, but reports them as whole numbers.
> What do you mean by a local cluster? Do you mean a single node cluster
> you launched yourself? then yes. If you mean a local-mode cluster, then no.
> - Bobby
>
>
> On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian <
> harshab85@gmail.com> wrote:
>
>
> Hi,
>
> Can someone pls answer my questions from the above thread.
>
>
> *I have 2 questions for you;*
> *1. In one of your earlier emails, you mentioned that there is
> a limited Metrics implementation which is available in the current Storm
> version. *
>
> *Will this provide 'acks threshold', i.e, the number of acks per second (or
> minute)? If not, can you please suggest a way in which I can implement this
> myself. *
>
> *2. Is it possible to use the CLI and UI tools from a local cluster?*
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
> harshab85@gmail.com> wrote:
> >
> > Hi Bobby,
> >
> > Hope this email finds you well. I.m about to start designing the
> > auto-scaling system. Created a JIRA for it,
> > https://issues.apache.org/jira/browse/STORM-594
> >
> > Please take a look and let me know if there are any concerns.
> >
> > I have 2 questions for you;
> > 1. In one of your earlier emails, you mentioned that there is a *limited
> *Metrics
> > implementation which is available in the current Storm version.
> > Will this provide 'acks threshold'?
> >
> > 2. Is it possible to use the CLI tool from a local cluster?
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Masters in Computer Science
> >
> > Mobile : 647-771-3596
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> > harshab85@gmail.com> wrote:
> >>
> >> Thanks for the detailed explanation Bobby. I will keep this in mind when
> >> I create my design document.
> >>
> >> I will mostly not deal with adding/removing bolts to the topology and
> >> restrict myself to threads and tasks. This is because of the time I
> have to
> >> complete the project. Once I submit my report (early January), I can
> extend
> >> what I have implemented to more use cases.
> >>
> >> Thanks,
> >> Harsha
> >>
> >> Thanks,
> >> Harsha
> >>
> >> *Contact Details: *
> >> Harsha Balasubramanian
> >> Graduate Student at University of Toronto
> >> Professional Masters in Computer Science
> >>
> >> Email 1: harshab85@cs.toronto.edu
> >> Email 2: harshab85@gmail.com
> >> LinkedIn : ca.linkedin.com/in/harshabala/
> >>
> >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans
> <evans@yahoo-inc.com.invalid
> >> > wrote:
> >>
> >>> Sure,
> >>> The main thing that storm is missing right now is an metrics feedback
> >>> system to the scheduler (or possibly a separate piece of code that
> decides
> >>> when to grow/shrink a topology). We have some basic metrics, but they
> are
> >>> not exposed to the scheduler. The other question is how do we deal
> with
> >>> creating/destroying new bolts, especially around dealing with different
> >>> groupings. Some groupings make it difficult. There a number of ways
> to
> >>> get around this, but I think the simplest way is to not create/destroy
> >>> instances, but to treat it as a resources problem, and give them more
> or
> >>> less resources as needed. But that still needs to be discussed. -
> Bobby
> >>>
> >>>
> >>> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian <
> >>> harshab85@gmail.com> wrote:
> >>>
> >>>
> >>> Hi Bobby,
> >>>
> >>> Thanks for getting back to me. It is encouraging to hear that this
> will
> >>> be a good addition to Storm if done well.
> >>>
> >>> I have not implemented the changes yet. My project just started. It
> >>> spans
> >>> over the next 2 months. In a few days, I will create a JIRA task and
> >>> submit
> >>> my proposal. It would be great to brainstorm with the Storm community
> and
> >>> iron out my design.
> >>>
> >>>
> >>> Thanks,
> >>> Harsha
> >>>
> >>> Thanks,
> >>> Harsha
> >>>
> >>> *Contact Details: *
> >>> Harsha Balasubramanian
> >>> Graduate Student at University of Toronto
> >>> Professional Masters in Computer Science
> >>>
> >>> Email 1: harshab85@cs.toronto.edu
> >>> Email 2: harshab85@gmail.com
> >>> LinkedIn : ca.linkedin.com/in/harshabala/
> >>>
> >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans
> <evans@yahoo-inc.com.invalid
> >>> >
> >>> wrote:
> >>>
> >>> > Yes, this type of a feature would be great to have. I am rally
> curious
> >>> > how you have done the changes, as there are a lot of potential
> pitfalls
> >>> > here. At a minimum it would just be great to have the feedback
> >>> framework
> >>> > in place so we can iterate on these changes.
> >>> > - Bobby
> >>> >
> >>> >
> >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <
> storm@harsha.io>
> >>> > wrote:
> >>> >
> >>> >
> >>> > Hi,
> >>> > It will definitely interesting to the storm community. It will be
> >>> > great if you can create a JIRA and post your code as PR for others
> >>> > to try out and review the code.
> >>> > Thanks,
> >>> > Harsha
> >>> >
> >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> >>> > > Please let me know if my project (outlined below) will be useful to
> >>> > > Storm.
> >>> > >
> >>> > > Thanks,
> >>> > > Harsha
> >>> > >
> >>> > > *Contact Details: *
> >>> > > Harsha Balasubramanian
> >>> > > Graduate Student at University of Toronto
> >>> > > Professional Masters in Computer Science
> >>> > >
> >>> > > Email 1: harshab85@cs.toronto.edu
> >>> > > Email 2: harshab85@gmail.com
> >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >>> > >
> >>> > > ---------- Forwarded message ----------
> >>> > > From: Harsha Balasubramanian <ha...@gmail.com>
> >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> >>> > > Subject: Auto-Scaling
> >>> > > To: dev@storm.apache.org
> >>> > >
> >>> > >
> >>> > > Hi,
> >>> > >
> >>> > > I am a Graduate student at the University of Toronto. As part of
> my
> >>> > > Advanced Database Systems course, I have proposed to implement an
> >>> Auto
> >>> > > Scaling mechanism for Storm topologies (using a Feedback System).
> >>> > >
> >>> > > I've gone through the pages on how to contribute to the Storm
> >>> project and
> >>> > > have some questions. Please let me know if auto-scaling is being
> >>> worked
> >>> > > on
> >>> > > currently. Also, should this be a project in StormCore or
> >>> StormProcessor
> >>> > > ?
> >>> > >
> >>> > > If this project will be a good addition to Storm, I will create a
> new
> >>> > > JIRA
> >>> > > task for it and add the details of my proposed implementation.
> >>> Please do
> >>> > > let me know.
> >>> > >
> >>> > > Thanks,
> >>> > > Harsha
> >>> > >
> >>> > > *Contact Details: *
> >>> > > Harsha Balasubramanian
> >>> > > Graduate Student at University of Toronto
> >>> > > Professional Masters in Computer Science
> >>> > >
> >>> > > Email 1: harshab85@cs.toronto.edu
> >>> > > Email 2: harshab85@gmail.com
> >>> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>>
> >>
> >>
>
>
>
>
Re: Fwd: Auto-Scaling
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Harsha,
Sorry about that, I have been out at a conference.
for 1 you can get the number of acks so far, but you are going to have to calculate the rate yourself, it keeps track of the number of acks over several time windows, but reports them as whole numbers.
What do you mean by a local cluster? Do you mean a single node cluster you launched yourself? then yes. If you mean a local-mode cluster, then no.
- Bobby
On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian <ha...@gmail.com> wrote:
Hi,
Can someone pls answer my questions from the above thread.
*I have 2 questions for you;*
*1. In one of your earlier emails, you mentioned that there is
a limited Metrics implementation which is available in the current Storm
version. *
*Will this provide 'acks threshold', i.e, the number of acks per second (or
minute)? If not, can you please suggest a way in which I can implement this
myself. *
*2. Is it possible to use the CLI and UI tools from a local cluster?*
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
harshab85@gmail.com> wrote:
>
> Hi Bobby,
>
> Hope this email finds you well. I.m about to start designing the
> auto-scaling system. Created a JIRA for it,
> https://issues.apache.org/jira/browse/STORM-594
>
> Please take a look and let me know if there are any concerns.
>
> I have 2 questions for you;
> 1. In one of your earlier emails, you mentioned that there is a *limited *Metrics
> implementation which is available in the current Storm version.
> Will this provide 'acks threshold'?
>
> 2. Is it possible to use the CLI tool from a local cluster?
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> harshab85@gmail.com> wrote:
>>
>> Thanks for the detailed explanation Bobby. I will keep this in mind when
>> I create my design document.
>>
>> I will mostly not deal with adding/removing bolts to the topology and
>> restrict myself to threads and tasks. This is because of the time I have to
>> complete the project. Once I submit my report (early January), I can extend
>> what I have implemented to more use cases.
>>
>> Thanks,
>> Harsha
>>
>> Thanks,
>> Harsha
>>
>> *Contact Details: *
>> Harsha Balasubramanian
>> Graduate Student at University of Toronto
>> Professional Masters in Computer Science
>>
>> Email 1: harshab85@cs.toronto.edu
>> Email 2: harshab85@gmail.com
>> LinkedIn : ca.linkedin.com/in/harshabala/
>>
>> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans <evans@yahoo-inc.com.invalid
>> > wrote:
>>
>>> Sure,
>>> The main thing that storm is missing right now is an metrics feedback
>>> system to the scheduler (or possibly a separate piece of code that decides
>>> when to grow/shrink a topology). We have some basic metrics, but they are
>>> not exposed to the scheduler. The other question is how do we deal with
>>> creating/destroying new bolts, especially around dealing with different
>>> groupings. Some groupings make it difficult. There a number of ways to
>>> get around this, but I think the simplest way is to not create/destroy
>>> instances, but to treat it as a resources problem, and give them more or
>>> less resources as needed. But that still needs to be discussed. - Bobby
>>>
>>>
>>> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian <
>>> harshab85@gmail.com> wrote:
>>>
>>>
>>> Hi Bobby,
>>>
>>> Thanks for getting back to me. It is encouraging to hear that this will
>>> be a good addition to Storm if done well.
>>>
>>> I have not implemented the changes yet. My project just started. It
>>> spans
>>> over the next 2 months. In a few days, I will create a JIRA task and
>>> submit
>>> my proposal. It would be great to brainstorm with the Storm community and
>>> iron out my design.
>>>
>>>
>>> Thanks,
>>> Harsha
>>>
>>> Thanks,
>>> Harsha
>>>
>>> *Contact Details: *
>>> Harsha Balasubramanian
>>> Graduate Student at University of Toronto
>>> Professional Masters in Computer Science
>>>
>>> Email 1: harshab85@cs.toronto.edu
>>> Email 2: harshab85@gmail.com
>>> LinkedIn : ca.linkedin.com/in/harshabala/
>>>
>>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans <evans@yahoo-inc.com.invalid
>>> >
>>> wrote:
>>>
>>> > Yes, this type of a feature would be great to have. I am rally curious
>>> > how you have done the changes, as there are a lot of potential pitfalls
>>> > here. At a minimum it would just be great to have the feedback
>>> framework
>>> > in place so we can iterate on these changes.
>>> > - Bobby
>>> >
>>> >
>>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io>
>>> > wrote:
>>> >
>>> >
>>> > Hi,
>>> > It will definitely interesting to the storm community. It will be
>>> > great if you can create a JIRA and post your code as PR for others
>>> > to try out and review the code.
>>> > Thanks,
>>> > Harsha
>>> >
>>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
>>> > > Please let me know if my project (outlined below) will be useful to
>>> > > Storm.
>>> > >
>>> > > Thanks,
>>> > > Harsha
>>> > >
>>> > > *Contact Details: *
>>> > > Harsha Balasubramanian
>>> > > Graduate Student at University of Toronto
>>> > > Professional Masters in Computer Science
>>> > >
>>> > > Email 1: harshab85@cs.toronto.edu
>>> > > Email 2: harshab85@gmail.com
>>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>>> > >
>>> > > ---------- Forwarded message ----------
>>> > > From: Harsha Balasubramanian <ha...@gmail.com>
>>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
>>> > > Subject: Auto-Scaling
>>> > > To: dev@storm.apache.org
>>> > >
>>> > >
>>> > > Hi,
>>> > >
>>> > > I am a Graduate student at the University of Toronto. As part of my
>>> > > Advanced Database Systems course, I have proposed to implement an
>>> Auto
>>> > > Scaling mechanism for Storm topologies (using a Feedback System).
>>> > >
>>> > > I've gone through the pages on how to contribute to the Storm
>>> project and
>>> > > have some questions. Please let me know if auto-scaling is being
>>> worked
>>> > > on
>>> > > currently. Also, should this be a project in StormCore or
>>> StormProcessor
>>> > > ?
>>> > >
>>> > > If this project will be a good addition to Storm, I will create a new
>>> > > JIRA
>>> > > task for it and add the details of my proposed implementation.
>>> Please do
>>> > > let me know.
>>> > >
>>> > > Thanks,
>>> > > Harsha
>>> > >
>>> > > *Contact Details: *
>>> > > Harsha Balasubramanian
>>> > > Graduate Student at University of Toronto
>>> > > Professional Masters in Computer Science
>>> > >
>>> > > Email 1: harshab85@cs.toronto.edu
>>> > > Email 2: harshab85@gmail.com
>>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>>
>>
>>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Hi,
Can someone pls answer my questions from the above thread.
*I have 2 questions for you;*
*1. In one of your earlier emails, you mentioned that there is
a limited Metrics implementation which is available in the current Storm
version. *
*Will this provide 'acks threshold', i.e, the number of acks per second (or
minute)? If not, can you please suggest a way in which I can implement this
myself. *
*2. Is it possible to use the CLI and UI tools from a local cluster?*
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian <
harshab85@gmail.com> wrote:
>
> Hi Bobby,
>
> Hope this email finds you well. I.m about to start designing the
> auto-scaling system. Created a JIRA for it,
> https://issues.apache.org/jira/browse/STORM-594
>
> Please take a look and let me know if there are any concerns.
>
> I have 2 questions for you;
> 1. In one of your earlier emails, you mentioned that there is a *limited *Metrics
> implementation which is available in the current Storm version.
> Will this provide 'acks threshold'?
>
> 2. Is it possible to use the CLI tool from a local cluster?
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Masters in Computer Science
>
> Mobile : 647-771-3596
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <
> harshab85@gmail.com> wrote:
>>
>> Thanks for the detailed explanation Bobby. I will keep this in mind when
>> I create my design document.
>>
>> I will mostly not deal with adding/removing bolts to the topology and
>> restrict myself to threads and tasks. This is because of the time I have to
>> complete the project. Once I submit my report (early January), I can extend
>> what I have implemented to more use cases.
>>
>> Thanks,
>> Harsha
>>
>> Thanks,
>> Harsha
>>
>> *Contact Details: *
>> Harsha Balasubramanian
>> Graduate Student at University of Toronto
>> Professional Masters in Computer Science
>>
>> Email 1: harshab85@cs.toronto.edu
>> Email 2: harshab85@gmail.com
>> LinkedIn : ca.linkedin.com/in/harshabala/
>>
>> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans <evans@yahoo-inc.com.invalid
>> > wrote:
>>
>>> Sure,
>>> The main thing that storm is missing right now is an metrics feedback
>>> system to the scheduler (or possibly a separate piece of code that decides
>>> when to grow/shrink a topology). We have some basic metrics, but they are
>>> not exposed to the scheduler. The other question is how do we deal with
>>> creating/destroying new bolts, especially around dealing with different
>>> groupings. Some groupings make it difficult. There a number of ways to
>>> get around this, but I think the simplest way is to not create/destroy
>>> instances, but to treat it as a resources problem, and give them more or
>>> less resources as needed. But that still needs to be discussed. - Bobby
>>>
>>>
>>> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian <
>>> harshab85@gmail.com> wrote:
>>>
>>>
>>> Hi Bobby,
>>>
>>> Thanks for getting back to me. It is encouraging to hear that this will
>>> be a good addition to Storm if done well.
>>>
>>> I have not implemented the changes yet. My project just started. It
>>> spans
>>> over the next 2 months. In a few days, I will create a JIRA task and
>>> submit
>>> my proposal. It would be great to brainstorm with the Storm community and
>>> iron out my design.
>>>
>>>
>>> Thanks,
>>> Harsha
>>>
>>> Thanks,
>>> Harsha
>>>
>>> *Contact Details: *
>>> Harsha Balasubramanian
>>> Graduate Student at University of Toronto
>>> Professional Masters in Computer Science
>>>
>>> Email 1: harshab85@cs.toronto.edu
>>> Email 2: harshab85@gmail.com
>>> LinkedIn : ca.linkedin.com/in/harshabala/
>>>
>>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans <evans@yahoo-inc.com.invalid
>>> >
>>> wrote:
>>>
>>> > Yes, this type of a feature would be great to have. I am rally curious
>>> > how you have done the changes, as there are a lot of potential pitfalls
>>> > here. At a minimum it would just be great to have the feedback
>>> framework
>>> > in place so we can iterate on these changes.
>>> > - Bobby
>>> >
>>> >
>>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io>
>>> > wrote:
>>> >
>>> >
>>> > Hi,
>>> > It will definitely interesting to the storm community. It will be
>>> > great if you can create a JIRA and post your code as PR for others
>>> > to try out and review the code.
>>> > Thanks,
>>> > Harsha
>>> >
>>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
>>> > > Please let me know if my project (outlined below) will be useful to
>>> > > Storm.
>>> > >
>>> > > Thanks,
>>> > > Harsha
>>> > >
>>> > > *Contact Details: *
>>> > > Harsha Balasubramanian
>>> > > Graduate Student at University of Toronto
>>> > > Professional Masters in Computer Science
>>> > >
>>> > > Email 1: harshab85@cs.toronto.edu
>>> > > Email 2: harshab85@gmail.com
>>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>>> > >
>>> > > ---------- Forwarded message ----------
>>> > > From: Harsha Balasubramanian <ha...@gmail.com>
>>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
>>> > > Subject: Auto-Scaling
>>> > > To: dev@storm.apache.org
>>> > >
>>> > >
>>> > > Hi,
>>> > >
>>> > > I am a Graduate student at the University of Toronto. As part of my
>>> > > Advanced Database Systems course, I have proposed to implement an
>>> Auto
>>> > > Scaling mechanism for Storm topologies (using a Feedback System).
>>> > >
>>> > > I've gone through the pages on how to contribute to the Storm
>>> project and
>>> > > have some questions. Please let me know if auto-scaling is being
>>> worked
>>> > > on
>>> > > currently. Also, should this be a project in StormCore or
>>> StormProcessor
>>> > > ?
>>> > >
>>> > > If this project will be a good addition to Storm, I will create a new
>>> > > JIRA
>>> > > task for it and add the details of my proposed implementation.
>>> Please do
>>> > > let me know.
>>> > >
>>> > > Thanks,
>>> > > Harsha
>>> > >
>>> > > *Contact Details: *
>>> > > Harsha Balasubramanian
>>> > > Graduate Student at University of Toronto
>>> > > Professional Masters in Computer Science
>>> > >
>>> > > Email 1: harshab85@cs.toronto.edu
>>> > > Email 2: harshab85@gmail.com
>>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>>
>>
>>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Hi Bobby,
Hope this email finds you well. I.m about to start designing the
auto-scaling system. Created a JIRA for it,
https://issues.apache.org/jira/browse/STORM-594
Please take a look and let me know if there are any concerns.
I have 2 questions for you;
1. In one of your earlier emails, you mentioned that there is a
*limited *Metrics
implementation which is available in the current Storm version.
Will this provide 'acks threshold'?
2. Is it possible to use the CLI tool from a local cluster?
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Masters in Computer Science
Mobile : 647-771-3596
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <harshab85@gmail.com
> wrote:
>
> Thanks for the detailed explanation Bobby. I will keep this in mind when I
> create my design document.
>
> I will mostly not deal with adding/removing bolts to the topology and
> restrict myself to threads and tasks. This is because of the time I have to
> complete the project. Once I submit my report (early January), I can extend
> what I have implemented to more use cases.
>
> Thanks,
> Harsha
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Professional Masters in Computer Science
>
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
> wrote:
>
>> Sure,
>> The main thing that storm is missing right now is an metrics feedback
>> system to the scheduler (or possibly a separate piece of code that decides
>> when to grow/shrink a topology). We have some basic metrics, but they are
>> not exposed to the scheduler. The other question is how do we deal with
>> creating/destroying new bolts, especially around dealing with different
>> groupings. Some groupings make it difficult. There a number of ways to
>> get around this, but I think the simplest way is to not create/destroy
>> instances, but to treat it as a resources problem, and give them more or
>> less resources as needed. But that still needs to be discussed. - Bobby
>>
>>
>> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian <
>> harshab85@gmail.com> wrote:
>>
>>
>> Hi Bobby,
>>
>> Thanks for getting back to me. It is encouraging to hear that this will
>> be a good addition to Storm if done well.
>>
>> I have not implemented the changes yet. My project just started. It
>> spans
>> over the next 2 months. In a few days, I will create a JIRA task and
>> submit
>> my proposal. It would be great to brainstorm with the Storm community and
>> iron out my design.
>>
>>
>> Thanks,
>> Harsha
>>
>> Thanks,
>> Harsha
>>
>> *Contact Details: *
>> Harsha Balasubramanian
>> Graduate Student at University of Toronto
>> Professional Masters in Computer Science
>>
>> Email 1: harshab85@cs.toronto.edu
>> Email 2: harshab85@gmail.com
>> LinkedIn : ca.linkedin.com/in/harshabala/
>>
>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans <evans@yahoo-inc.com.invalid
>> >
>> wrote:
>>
>> > Yes, this type of a feature would be great to have. I am rally curious
>> > how you have done the changes, as there are a lot of potential pitfalls
>> > here. At a minimum it would just be great to have the feedback
>> framework
>> > in place so we can iterate on these changes.
>> > - Bobby
>> >
>> >
>> > On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io>
>> > wrote:
>> >
>> >
>> > Hi,
>> > It will definitely interesting to the storm community. It will be
>> > great if you can create a JIRA and post your code as PR for others
>> > to try out and review the code.
>> > Thanks,
>> > Harsha
>> >
>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
>> > > Please let me know if my project (outlined below) will be useful to
>> > > Storm.
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > > *Contact Details: *
>> > > Harsha Balasubramanian
>> > > Graduate Student at University of Toronto
>> > > Professional Masters in Computer Science
>> > >
>> > > Email 1: harshab85@cs.toronto.edu
>> > > Email 2: harshab85@gmail.com
>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> > >
>> > > ---------- Forwarded message ----------
>> > > From: Harsha Balasubramanian <ha...@gmail.com>
>> > > Date: Tue, Nov 11, 2014 at 8:15 PM
>> > > Subject: Auto-Scaling
>> > > To: dev@storm.apache.org
>> > >
>> > >
>> > > Hi,
>> > >
>> > > I am a Graduate student at the University of Toronto. As part of my
>> > > Advanced Database Systems course, I have proposed to implement an Auto
>> > > Scaling mechanism for Storm topologies (using a Feedback System).
>> > >
>> > > I've gone through the pages on how to contribute to the Storm project
>> and
>> > > have some questions. Please let me know if auto-scaling is being
>> worked
>> > > on
>> > > currently. Also, should this be a project in StormCore or
>> StormProcessor
>> > > ?
>> > >
>> > > If this project will be a good addition to Storm, I will create a new
>> > > JIRA
>> > > task for it and add the details of my proposed implementation. Please
>> do
>> > > let me know.
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > > *Contact Details: *
>> > > Harsha Balasubramanian
>> > > Graduate Student at University of Toronto
>> > > Professional Masters in Computer Science
>> > >
>> > > Email 1: harshab85@cs.toronto.edu
>> > > Email 2: harshab85@gmail.com
>> > > LinkedIn : ca.linkedin.com/in/harshabala/
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Thanks for the detailed explanation Bobby. I will keep this in mind when I
create my design document.
I will mostly not deal with adding/removing bolts to the topology and
restrict myself to threads and tasks. This is because of the time I have to
complete the project. Once I submit my report (early January), I can extend
what I have implemented to more use cases.
Thanks,
Harsha
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Professional Masters in Computer Science
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Sure,
> The main thing that storm is missing right now is an metrics feedback
> system to the scheduler (or possibly a separate piece of code that decides
> when to grow/shrink a topology). We have some basic metrics, but they are
> not exposed to the scheduler. The other question is how do we deal with
> creating/destroying new bolts, especially around dealing with different
> groupings. Some groupings make it difficult. There a number of ways to
> get around this, but I think the simplest way is to not create/destroy
> instances, but to treat it as a resources problem, and give them more or
> less resources as needed. But that still needs to be discussed. - Bobby
>
>
> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian <
> harshab85@gmail.com> wrote:
>
>
> Hi Bobby,
>
> Thanks for getting back to me. It is encouraging to hear that this will
> be a good addition to Storm if done well.
>
> I have not implemented the changes yet. My project just started. It spans
> over the next 2 months. In a few days, I will create a JIRA task and submit
> my proposal. It would be great to brainstorm with the Storm community and
> iron out my design.
>
>
> Thanks,
> Harsha
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Professional Masters in Computer Science
>
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
> wrote:
>
> > Yes, this type of a feature would be great to have. I am rally curious
> > how you have done the changes, as there are a lot of potential pitfalls
> > here. At a minimum it would just be great to have the feedback framework
> > in place so we can iterate on these changes.
> > - Bobby
> >
> >
> > On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io>
> > wrote:
> >
> >
> > Hi,
> > It will definitely interesting to the storm community. It will be
> > great if you can create a JIRA and post your code as PR for others
> > to try out and review the code.
> > Thanks,
> > Harsha
> >
> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> > > Please let me know if my project (outlined below) will be useful to
> > > Storm.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Professional Masters in Computer Science
> > >
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> > >
> > > ---------- Forwarded message ----------
> > > From: Harsha Balasubramanian <ha...@gmail.com>
> > > Date: Tue, Nov 11, 2014 at 8:15 PM
> > > Subject: Auto-Scaling
> > > To: dev@storm.apache.org
> > >
> > >
> > > Hi,
> > >
> > > I am a Graduate student at the University of Toronto. As part of my
> > > Advanced Database Systems course, I have proposed to implement an Auto
> > > Scaling mechanism for Storm topologies (using a Feedback System).
> > >
> > > I've gone through the pages on how to contribute to the Storm project
> and
> > > have some questions. Please let me know if auto-scaling is being worked
> > > on
> > > currently. Also, should this be a project in StormCore or
> StormProcessor
> > > ?
> > >
> > > If this project will be a good addition to Storm, I will create a new
> > > JIRA
> > > task for it and add the details of my proposed implementation. Please
> do
> > > let me know.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > *Contact Details: *
> > > Harsha Balasubramanian
> > > Graduate Student at University of Toronto
> > > Professional Masters in Computer Science
> > >
> > > Email 1: harshab85@cs.toronto.edu
> > > Email 2: harshab85@gmail.com
> > > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> >
> >
> >
>
>
>
>
Re: Fwd: Auto-Scaling
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Sure,
The main thing that storm is missing right now is an metrics feedback system to the scheduler (or possibly a separate piece of code that decides when to grow/shrink a topology). We have some basic metrics, but they are not exposed to the scheduler. The other question is how do we deal with creating/destroying new bolts, especially around dealing with different groupings. Some groupings make it difficult. There a number of ways to get around this, but I think the simplest way is to not create/destroy instances, but to treat it as a resources problem, and give them more or less resources as needed. But that still needs to be discussed. - Bobby
On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian <ha...@gmail.com> wrote:
Hi Bobby,
Thanks for getting back to me. It is encouraging to hear that this will
be a good addition to Storm if done well.
I have not implemented the changes yet. My project just started. It spans
over the next 2 months. In a few days, I will create a JIRA task and submit
my proposal. It would be great to brainstorm with the Storm community and
iron out my design.
Thanks,
Harsha
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Professional Masters in Computer Science
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Yes, this type of a feature would be great to have. I am rally curious
> how you have done the changes, as there are a lot of potential pitfalls
> here. At a minimum it would just be great to have the feedback framework
> in place so we can iterate on these changes.
> - Bobby
>
>
> On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io>
> wrote:
>
>
> Hi,
> It will definitely interesting to the storm community. It will be
> great if you can create a JIRA and post your code as PR for others
> to try out and review the code.
> Thanks,
> Harsha
>
> On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> > Please let me know if my project (outlined below) will be useful to
> > Storm.
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Professional Masters in Computer Science
> >
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > ---------- Forwarded message ----------
> > From: Harsha Balasubramanian <ha...@gmail.com>
> > Date: Tue, Nov 11, 2014 at 8:15 PM
> > Subject: Auto-Scaling
> > To: dev@storm.apache.org
> >
> >
> > Hi,
> >
> > I am a Graduate student at the University of Toronto. As part of my
> > Advanced Database Systems course, I have proposed to implement an Auto
> > Scaling mechanism for Storm topologies (using a Feedback System).
> >
> > I've gone through the pages on how to contribute to the Storm project and
> > have some questions. Please let me know if auto-scaling is being worked
> > on
> > currently. Also, should this be a project in StormCore or StormProcessor
> > ?
> >
> > If this project will be a good addition to Storm, I will create a new
> > JIRA
> > task for it and add the details of my proposed implementation. Please do
> > let me know.
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Professional Masters in Computer Science
> >
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
>
>
>
>
Re: Fwd: Auto-Scaling
Posted by Harsha Balasubramanian <ha...@gmail.com>.
Hi Bobby,
Thanks for getting back to me. It is encouraging to hear that this will
be a good addition to Storm if done well.
I have not implemented the changes yet. My project just started. It spans
over the next 2 months. In a few days, I will create a JIRA task and submit
my proposal. It would be great to brainstorm with the Storm community and
iron out my design.
Thanks,
Harsha
Thanks,
Harsha
*Contact Details: *
Harsha Balasubramanian
Graduate Student at University of Toronto
Professional Masters in Computer Science
Email 1: harshab85@cs.toronto.edu
Email 2: harshab85@gmail.com
LinkedIn : ca.linkedin.com/in/harshabala/
On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Yes, this type of a feature would be great to have. I am rally curious
> how you have done the changes, as there are a lot of potential pitfalls
> here. At a minimum it would just be great to have the feedback framework
> in place so we can iterate on these changes.
> - Bobby
>
>
> On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io>
> wrote:
>
>
> Hi,
> It will definitely interesting to the storm community. It will be
> great if you can create a JIRA and post your code as PR for others
> to try out and review the code.
> Thanks,
> Harsha
>
> On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> > Please let me know if my project (outlined below) will be useful to
> > Storm.
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Professional Masters in Computer Science
> >
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
> >
> > ---------- Forwarded message ----------
> > From: Harsha Balasubramanian <ha...@gmail.com>
> > Date: Tue, Nov 11, 2014 at 8:15 PM
> > Subject: Auto-Scaling
> > To: dev@storm.apache.org
> >
> >
> > Hi,
> >
> > I am a Graduate student at the University of Toronto. As part of my
> > Advanced Database Systems course, I have proposed to implement an Auto
> > Scaling mechanism for Storm topologies (using a Feedback System).
> >
> > I've gone through the pages on how to contribute to the Storm project and
> > have some questions. Please let me know if auto-scaling is being worked
> > on
> > currently. Also, should this be a project in StormCore or StormProcessor
> > ?
> >
> > If this project will be a good addition to Storm, I will create a new
> > JIRA
> > task for it and add the details of my proposed implementation. Please do
> > let me know.
> >
> > Thanks,
> > Harsha
> >
> > *Contact Details: *
> > Harsha Balasubramanian
> > Graduate Student at University of Toronto
> > Professional Masters in Computer Science
> >
> > Email 1: harshab85@cs.toronto.edu
> > Email 2: harshab85@gmail.com
> > LinkedIn : ca.linkedin.com/in/harshabala/
>
>
>
>
Re: Fwd: Auto-Scaling
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Yes, this type of a feature would be great to have. I am rally curious how you have done the changes, as there are a lot of potential pitfalls here. At a minimum it would just be great to have the feedback framework in place so we can iterate on these changes.
- Bobby
On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io> wrote:
Hi,
It will definitely interesting to the storm community. It will be
great if you can create a JIRA and post your code as PR for others
to try out and review the code.
Thanks,
Harsha
On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> Please let me know if my project (outlined below) will be useful to
> Storm.
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Professional Masters in Computer Science
>
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> ---------- Forwarded message ----------
> From: Harsha Balasubramanian <ha...@gmail.com>
> Date: Tue, Nov 11, 2014 at 8:15 PM
> Subject: Auto-Scaling
> To: dev@storm.apache.org
>
>
> Hi,
>
> I am a Graduate student at the University of Toronto. As part of my
> Advanced Database Systems course, I have proposed to implement an Auto
> Scaling mechanism for Storm topologies (using a Feedback System).
>
> I've gone through the pages on how to contribute to the Storm project and
> have some questions. Please let me know if auto-scaling is being worked
> on
> currently. Also, should this be a project in StormCore or StormProcessor
> ?
>
> If this project will be a good addition to Storm, I will create a new
> JIRA
> task for it and add the details of my proposed implementation. Please do
> let me know.
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Professional Masters in Computer Science
>
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
Re: Fwd: Auto-Scaling
Posted by Harsha <st...@harsha.io>.
Hi,
It will definitely interesting to the storm community. It will be
great if you can create a JIRA and post your code as PR for others
to try out and review the code.
Thanks,
Harsha
On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote:
> Please let me know if my project (outlined below) will be useful to
> Storm.
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Professional Masters in Computer Science
>
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/
>
> ---------- Forwarded message ----------
> From: Harsha Balasubramanian <ha...@gmail.com>
> Date: Tue, Nov 11, 2014 at 8:15 PM
> Subject: Auto-Scaling
> To: dev@storm.apache.org
>
>
> Hi,
>
> I am a Graduate student at the University of Toronto. As part of my
> Advanced Database Systems course, I have proposed to implement an Auto
> Scaling mechanism for Storm topologies (using a Feedback System).
>
> I've gone through the pages on how to contribute to the Storm project and
> have some questions. Please let me know if auto-scaling is being worked
> on
> currently. Also, should this be a project in StormCore or StormProcessor
> ?
>
> If this project will be a good addition to Storm, I will create a new
> JIRA
> task for it and add the details of my proposed implementation. Please do
> let me know.
>
> Thanks,
> Harsha
>
> *Contact Details: *
> Harsha Balasubramanian
> Graduate Student at University of Toronto
> Professional Masters in Computer Science
>
> Email 1: harshab85@cs.toronto.edu
> Email 2: harshab85@gmail.com
> LinkedIn : ca.linkedin.com/in/harshabala/