You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Raphael Hsieh <ra...@gmail.com> on 2014/10/06 23:27:33 UTC
programmatically call rebalance
Is there an api or some way to programmatically rebalance a fleet if it is
detected that it has become unbalanced ?
Worst case scenario, I could create a cron job that queries the UI and
parses the tables to make sure everything is the same value, then if it's
not, spoof a button click for the rebalance button.
But this seems super hacky and gross.
Thanks
--
Raphael Hsieh
Re: Can I do cron-like job in spout?
Posted by Corey Nolet <cj...@gmail.com>.
Generally, in streams processing frameworks the general rule of thumb is to
try to keep the thread management to the framework but you can certainly
have your own TimerTask or a different thread in your spout that's
scheduling when to do things.
Of course, if you emit something on the timer without honoring the
nextTuple() method in the spout then you could blow the heap and cause
unexpected results (as you are not allowing Storm's flow control to kick
in). An excepted way of implementing tuples in a timer is to queue them up
yourself and have the nextTuple() method pull from your own queue.
On Tue, Oct 7, 2014 at 5:43 AM, 이승진 <sw...@navercorp.com> wrote:
> Dear all,
>
>
>
> I know there is a tick tuple component which makes it possible to do
> certain job periodically within bolts.
>
>
>
> To my knowledge however, it seems like spouts cannot receive a tick tuple
> since there's no argument in nextTuple method.
>
> (BaseRichSpout)
>
>
>
> Is there any other option to implement cron-like job within a spout? Any
> comment will help.
>
> Sincerely,
>
>
Re: Can I do cron-like job in spout?
Posted by Stephen Armstrong <st...@linqia.com>.
I believe Storm will call nextTuple() very often on a spout unless the
max_unacked_tuple limit has been reached. You should see calls to that
method pretty often, which means you could then just check the current time
and (if needed) emit the tuples that will start whatever given cron task
you wanted.
On Tue, Oct 7, 2014 at 2:43 AM, 이승진 <sw...@navercorp.com> wrote:
> Dear all,
>
>
>
> I know there is a tick tuple component which makes it possible to do
> certain job periodically within bolts.
>
>
>
> To my knowledge however, it seems like spouts cannot receive a tick tuple
> since there's no argument in nextTuple method.
>
> (BaseRichSpout)
>
>
>
> Is there any other option to implement cron-like job within a spout? Any
> comment will help.
>
> Sincerely,
>
>
Can I do cron-like job in spout?
Posted by 이승진 <sw...@navercorp.com>.
Dear all,
I know there is a tick tuple component which makes it possible to do certain job periodically within bolts.
To my knowledge however, it seems like spouts cannot receive a tick tuple since there's no argument in nextTuple method.
(BaseRichSpout)
Is there any other option to implement cron-like job within a spout? Any comment will help.
Sincerely,
Re: programmatically call rebalance
Posted by Gahl Rinat <ga...@alooma.io>.
An Amazon engineer asking questions about Storm.
-Gahl
---------- Forwarded message ----------
From: Raphael Hsieh <ra...@gmail.com>
Date: Tue, Oct 7, 2014 at 12:27 AM
Subject: programmatically call rebalance
To: user@storm.incubator.apache.org
Is there an api or some way to programmatically rebalance a fleet if it is
detected that it has become unbalanced ?
Worst case scenario, I could create a cron job that queries the UI and
parses the tables to make sure everything is the same value, then if it's
not, spoof a button click for the rebalance button.
But this seems super hacky and gross.
Thanks
--
Raphael Hsieh
--
Gahl Rinat
alooma | Product & Marketing
alooma.io | il.linkedin.com/in/gahlrinat
+972 (0) 50 951 9499 | +1 (650) 814-3550
Re: programmatically call rebalance
Posted by Richards Peter <hb...@gmail.com>.
If you have any such concerns, you can limit the use of nimbus client api
for your specific cases. I prefer to use this api because it is straight
forward and suits my use cases.
Regards,
Richards Peter
Re: programmatically call rebalance
Posted by Raphael Hsieh <ra...@gmail.com>.
Thanks for the heads up Peter,
By configuring the transport plugin do I lose the ability to do everything
else normally? Will I then need to figure out how to use the Nimbus client
for everything else ?
Thanks!
On Mon, Oct 6, 2014 at 11:46 PM, Richards Peter <hb...@gmail.com>
wrote:
> Hi Raphael,
>
> You have to configure an additional property, for storm 0.9.*, to connect
> to to nimbus.
>
> Please check my Aug 4th mail with subject: storm-0.9.2-incubating
> topology submission fails
> <http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/%3cCANMELZsAASc4DOJW8sQSa+N6SG4-_mVZx3gZShST-ptGtaY_1A@mail.gmail.com%3e>
>
> http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/thread
>
> Please check Taylor's reply:
> Re: storm-0.9.2-incubating topology submission fails
> <http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/%3cBC4C3D35-93B6-45FA-960D-47325CCAFD18@gmail.com%3e>
> He has asked to configure one more property in the map.
>
> Regards,
> Richards Peter.
>
--
Raphael Hsieh
Re: programmatically call rebalance
Posted by Richards Peter <hb...@gmail.com>.
Hi Raphael,
You have to configure an additional property, for storm 0.9.*, to connect
to to nimbus.
Please check my Aug 4th mail with subject: storm-0.9.2-incubating topology
submission fails
<http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/%3cCANMELZsAASc4DOJW8sQSa+N6SG4-_mVZx3gZShST-ptGtaY_1A@mail.gmail.com%3e>
http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/thread
Please check Taylor's reply:
Re: storm-0.9.2-incubating topology submission fails
<http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/%3cBC4C3D35-93B6-45FA-960D-47325CCAFD18@gmail.com%3e>
He has asked to configure one more property in the map.
Regards,
Richards Peter.
Re: programmatically call rebalance
Posted by Richards Peter <hb...@gmail.com>.
Hi Raphael,
I think you should use the Nimbus interface implementation:
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/generated/Nimbus.java
Please find the rebalance() in this interface.
I have not fully explored this option though. You can obtain an instance of
Nimbus interface implementation using NimbusClient.java
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/utils/NimbusClient.java
I have used NimbusClient and Nimbus class to submit topology without using
storm command. Please refer the following mail thread to find more details
about using NimbusClient:
https://groups.google.com/forum/#!searchin/storm-user/Can$20we$20submit$20the$20topology$20jar$20without$20using$20storm$20command/storm-user/5JjZQyviRqk/TGRCgciHKWwJ
I think the trickiest part of your task will be the creation of
RebalanceOptions instance:
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/generated/RebalanceOptions.java
Regards,
Richards Peter.
Re: programmatically call rebalance
Posted by Harsha <st...@harsha.io>.
Its part of 0.9.2 release.
-Harsha
On Mon, Oct 6, 2014, at 03:22 PM, Raphael Hsieh wrote:
is the rest API new within the past couple months?
for some reason when I send a GET request to the paths given in
the examples, I get a 404.
just to be clear, am I using the api correctly?
If I send a GET request
to [1]http://nimbus-ui-host:8080/api/v1/cluster/configuration
(via ajax or some other method) I should have returned to me a
json blob containing
{"dev.zookeeper.path":"/tmp/dev-storm-zookeeper",
"topology.tick.tuple.freq.secs":null,
"topology.builtin.metrics.bucket.size.secs":60,
... }
is that correct ?
On Mon, Oct 6, 2014 at 2:38 PM, Parth Brahmbhatt
<[2...@hortonworks.com> wrote:
You could use storm’s rest [3]API. That is what the ui uses.
Alternatively you can use storm’s nimbus [4]client or storm
command line.
Thanks
Parth
On Oct 6, 2014, at 2:27 PM, Raphael Hsieh
<[5...@gmail.com> wrote:
Worst case scenario, I could create a cron job that queries
the UI and parses the tables to make sure everything is the
same value, then if it's not, spoof a button click for the
rebalance button.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual
or entity to which it is addressed and may contain information
that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the
intended recipient, you are hereby notified that any printing,
copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately and delete it from your system. Thank You.
--
Raphael Hsieh
References
1. http://nimbus-ui-host:8080/api/v1/cluster/configuration
2. mailto:pbrahmbhatt@hortonworks.com
3. https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md
4. https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/generated/Nimbus.java#L295
5. mailto:raffihsieh@gmail.com
Re: programmatically call rebalance
Posted by Raphael Hsieh <ra...@gmail.com>.
is the rest API new within the past couple months?
for some reason when I send a GET request to the paths given in the
examples, I get a 404.
just to be clear, am I using the api correctly?
If I send a GET request to
http://nimbus-ui-host:8080/api/v1/cluster/configuration (via ajax or some
other method) I should have returned to me a json blob containing
{"dev.zookeeper.path": "/tmp/dev-storm-zookeeper",
"topology.tick.tuple.freq.secs": null,
"topology.builtin.metrics.bucket.size.secs": 60,
... }
is that correct ?
On Mon, Oct 6, 2014 at 2:38 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> You could use storm’s rest API
> <https://github.com/apache/storm/blob/master/STORM-UI-REST-API.md>. That
> is what the ui uses. Alternatively you can use storm’s nimbus client
> <https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/generated/Nimbus.java#L295> or
> storm command line.
>
> Thanks
> Parth
> On Oct 6, 2014, at 2:27 PM, Raphael Hsieh <ra...@gmail.com> wrote:
>
> Worst case scenario, I could create a cron job that queries the UI and
> parses the tables to make sure everything is the same value, then if it's
> not, spoof a button click for the rebalance button.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
Raphael Hsieh
Re: programmatically call rebalance
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
You could use storm’s rest API. That is what the ui uses. Alternatively you can use storm’s nimbus client or storm command line.
Thanks
Parth
On Oct 6, 2014, at 2:27 PM, Raphael Hsieh <ra...@gmail.com> wrote:
> Worst case scenario, I could create a cron job that queries the UI and parses the tables to make sure everything is the same value, then if it's not, spoof a button click for the rebalance button.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.