You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Eranga Heshan <er...@gmail.com> on 2016/12/01 04:27:51 UTC
Re: Schedule a Bolt/Spout to a Predefined Node
Yeah, by "input bolt", I meant "spout".
All I need is a strong and reliable method to calculate latency of an
event. In my benchmark, I stamp a time into every event when spout
generates it. When it arrives the output bolt (which means the process is
complete), it checks for the time difference between output bolt's current
time and the stamped time of the event. The problem is that spout might be
on a different node other than output bolt. Hence there might be a small
error when measuring the time difference (latency).
By implementing a custom scheduler, I am trying to bind both the spout and
output bolt to one node. Ultimately the time difference could be measured
perfectly.
I still could not understand what you meant by:
>and the number of worker processes for the topology should also be an even
divisor of the common factor for the spouts & bolts
When you say "even divisor", did you mean "divides without any remainder"
or "division result is even"? And when you say "common factor" did you mean
"greatest common factor"? I will brief what I understood from your
explanation.
Assume "x" and "y" and "z" as integers:
num_of_bolts = 2x //even multiple
num_of_spouts = 2y //even multiple
num_of_workers = z
f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
f / z is even //even divisor of the common factor
Did I understand you correctly or please correct me if I am wrong.
I appreciate your concern about my matter.
Thanks,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
wrote:
> Can you explain more about why you are trying this custom scheduler? (I'm
> afraid you're largely on your own if you *must* continue down that route).
> In your initial email you just said:
>
> > I need to run "input bolt" and "output bolt" on the same node in order
> to calculate the latency properly
>
> I do not understand why that requirement led you to conclude that you must
> use a custom scheduler. If you align everything appropriately in terms of
> executors and workers and hosts, and if you're using
> localOrShuffleGrouping, then you can ensure that all traffic will stay
> local to the host. By "appropriate alignment", I just mean that the number
> of spouts and bolts should be even multiples, and the number of worker
> processes for the topology should also be an even divisor of the common
> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
> 10 spouts and 20 bolts.
>
> Also when you say "input bolt", do you actually mean *spout*?
>
> - Erik
>
>
> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
> wrote:
>
>> Can we have some help to solve this issue?
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>>> Hi Ohad,
>>>
>>> Does your scheduler works with storm-0.9.x? I just made few changes and
>>> tested it. Still it won't work.
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>>> wrote:
>>>
>>>> I have opened this stack overflow question,
>>>> http://stackoverflow.com/questions/40294245/deploy-bolts-spo
>>>> ut-to-a-specific-supervisor
>>>> Follow it and add question there,
>>>> And lets share information if any progress is made
>>>>
>>>> From: Eranga Heshan <er...@gmail.com>
>>>> Date: Tuesday, 29 November 2016 at 13:49
>>>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>>>> user@storm.apache.org>
>>>>
>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>
>>>> So we both used http://www.exogeni.net/2015/04
>>>> /enabling-site-aware-scheduling-for-apache-storm-in-exogeni/ tutorial.
>>>> Then I guess we are facing the problem due to the version.
>>>>
>>>> I think, we should try this our own until help arrives and if any of us
>>>> figured it out, let us inform each other.
>>>>
>>>> Cheers,
>>>> Regards,
>>>>
>>>>
>>>> Eranga Heshan
>>>> *Undergraduate*
>>>> Computer Science & Engineering
>>>> University of Moratuwa
>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>> <https://www.facebook.com/erangaheshan>
>>>> <https://twitter.com/erangaheshan>
>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>
>>>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>>>> wrote:
>>>>
>>>>> I used this tutorial:
>>>>>
>>>>> http://www.exogeni.net/2015/04/enabling-site-aware-schedulin
>>>>> g-for-apache-storm-in-exogeni/
>>>>>
>>>>>
>>>>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>>>>> storm/DemoScheduler.java
>>>>>
>>>>> But its for storm 0.9.3
>>>>>
>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>> Date: Tuesday, 29 November 2016 at 13:13
>>>>> To: Ohad Edelstein <oh...@mintigo.com>
>>>>>
>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>
>>>>> Hi Ohad,
>>>>>
>>>>> Yeah, it works when I remove that meta data part from the storm.yaml.
>>>>> I thought about that also. Could you please set "supervisor.scheduler.meta"
>>>>> to "supervisor.meta" and see it works. However, it did not work for me. But
>>>>> in someplace I saw it like that.
>>>>>
>>>>> I am working with storm-1.0.2.
>>>>>
>>>>> Let us both win at once 😉
>>>>>
>>>>> Cheers,
>>>>> Regard,
>>>>>
>>>>>
>>>>> Eranga Heshan
>>>>> *Undergraduate*
>>>>> Computer Science & Engineering
>>>>> University of Moratuwa
>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>> <https://www.facebook.com/erangaheshan>
>>>>> <https://twitter.com/erangaheshan>
>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>
>>>>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>> wrote:
>>>>>
>>>>>> I guess you added the meta data part to the storm.yaml in the
>>>>>> supervisors…
>>>>>> I think that when ever the code has a problem with one of the params
>>>>>> you add there, it cause the workers not to start
>>>>>> When you remove the meta data the problem will go away.
>>>>>> My guess is that I am not writing the meta part correctly…
>>>>>> When I set a different parameter that was suppose to be array to a
>>>>>> simple string, it did the same thing…
>>>>>>
>>>>>> With what version of storm are you working with? And what
>>>>>> instructions are you following?
>>>>>> Thanks
>>>>>>
>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>> Date: Tuesday, 29 November 2016 at 12:55
>>>>>>
>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>
>>>>>> Hi Ohad,
>>>>>>
>>>>>> My supervisors are running. But inside them, the workers are not
>>>>>> working. That is what I think the issue is. Do you have the same issue?
>>>>>>
>>>>>>
>>>>>> Eranga Heshan
>>>>>> *Undergraduate*
>>>>>> Computer Science & Engineering
>>>>>> University of Moratuwa
>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>> <https://twitter.com/erangaheshan>
>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>
>>>>>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, I am having the same problem on storm 1.0.1
>>>>>>> I would love if you can share the information again,
>>>>>>> I also had the same problem that Eranga is having.
>>>>>>> The problem he has is that the supervisor can’t start because of
>>>>>>> configuration problem…
>>>>>>>
>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>> Date: Tuesday, 29 November 2016 at 11:00
>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks for the info. I implemented a custom scheduler according to
>>>>>>> the link you suggested. I changed "getAvailableWorkerSlots" in the original
>>>>>>> code to "getAvailableSlots". However when I submit my topology it shows on
>>>>>>> the Storm UI. But when I click on it and look for more information I see
>>>>>>> nothing. All the tables are emplty.
>>>>>>>
>>>>>>> I am here attaching the inside view of my topology. Could you please
>>>>>>> look into that and say what I might have done wrong?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Regards,
>>>>>>>
>>>>>>> ​
>>>>>>>
>>>>>>>
>>>>>>> Eranga Heshan
>>>>>>> *Undergraduate*
>>>>>>> Computer Science & Engineering
>>>>>>> University of Moratuwa
>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>
>>>>>>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <
>>>>>>> anshushukla0@gmail.com> wrote:
>>>>>>>
>>>>>>>> you have to use custom site aware scheduler for that . Or use
>>>>>>>> ntp to sync clocks for all VMs .
>>>>>>>>
>>>>>>>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <
>>>>>>>> eranga.h.n@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I have a cluster of four nodes. There is a benchmark I need to run
>>>>>>>>> on cluster. Hovever I need to run "input bolt" and "output bolt" on the
>>>>>>>>> same node in order to calculate the latency properly.
>>>>>>>>>
>>>>>>>>> Is there a way that I could run both the bols on the same node?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Eranga Heshan
>>>>>>>>> *Undergraduate*
>>>>>>>>> Computer Science & Engineering
>>>>>>>>> University of Moratuwa
>>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks & Regards,
>>>>>>>> Anshu Shukla
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by anshu shukla <an...@gmail.com>.
Actually aim is to calculate exact latency values for fixed Message ID. In
case of VMs running on cloud, even minor difference in clocks may give
negative values for latency (if spout and sink are placed on different VMs)
[--NTP]
Another way is if we use same VM for both spout and sink then how we can
horizontally scale to get more input rate . At some point of time resources
will be the issue . [-- vertical scaling is only possible ]
Please correct me if am wrong.
On Thu, Dec 1, 2016 at 10:17 AM, Erik Weathers <ew...@groupon.com>
wrote:
> What does NTP have to do with this? I mean, yes, if you're doing anything
> on a cluster you should be running NTP, but please explain why you are
> saying that is related to bottlenecking?
>
> On Wed, Nov 30, 2016 at 8:38 PM anshu shukla <an...@gmail.com>
> wrote:
>
>> I was also doing the same thing some days before . Although this is
>> possible by custom scheduler but for getting higher rate ~100000 mps ,
>> again problem will arise because machines with spout and sink will get
>> bottlenecked.
>>
>> *So the best thing is to use some way to synchronize clock on bothe VMs
>> (using NTP etc.). Currently I use a site aware custom scheduler with NTP
>> running on master node. *
>>
>> On Thu, Dec 1, 2016 at 10:02 AM, Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>> Whoa. Too complicated. How many bolts and spouts are you trying to have?
>>
>> On Wed, Nov 30, 2016 at 8:28 PM Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Yeah, by "input bolt", I meant "spout".
>>
>> All I need is a strong and reliable method to calculate latency of an
>> event. In my benchmark, I stamp a time into every event when spout
>> generates it. When it arrives the output bolt (which means the process is
>> complete), it checks for the time difference between output bolt's current
>> time and the stamped time of the event. The problem is that spout might be
>> on a different node other than output bolt. Hence there might be a small
>> error when measuring the time difference (latency).
>>
>> By implementing a custom scheduler, I am trying to bind both the spout
>> and output bolt to one node. Ultimately the time difference could be
>> measured perfectly.
>>
>> I still could not understand what you meant by:
>>
>>
>> >and the number of worker processes for the topology should also be an
>> even divisor of the common factor for the spouts & bolts
>>
>> When you say "even divisor", did you mean "divides without any remainder"
>> or "division result is even"? And when you say "common factor" did you mean
>> "greatest common factor"? I will brief what I understood from your
>> explanation.
>>
>> Assume "x" and "y" and "z" as integers:
>> num_of_bolts = 2x //even multiple
>> num_of_spouts = 2y //even multiple
>> num_of_workers = z
>> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
>> f / z is even //even divisor of the common factor
>>
>> Did I understand you correctly or please correct me if I am wrong.
>>
>> I appreciate your concern about my matter.
>>
>> Thanks,
>> Regards,
>>
>>
>>
>>
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>> Can you explain more about why you are trying this custom scheduler?
>> (I'm afraid you're largely on your own if you *must* continue down that
>> route).
>> In your initial email you just said:
>>
>> > I need to run "input bolt" and "output bolt" on the same node in order
>> to calculate the latency properly
>>
>> I do not understand why that requirement led you to conclude that you
>> must use a custom scheduler. If you align everything appropriately in
>> terms of executors and workers and hosts, and if you're using
>> localOrShuffleGrouping, then you can ensure that all traffic will stay
>> local to the host. By "appropriate alignment", I just mean that the number
>> of spouts and bolts should be even multiples, and the number of worker
>> processes for the topology should also be an even divisor of the common
>> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
>> 10 spouts and 20 bolts.
>>
>> Also when you say "input bolt", do you actually mean *spout*?
>>
>> - Erik
>>
>>
>> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Can we have some help to solve this issue?
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Hi Ohad,
>>
>> Does your scheduler works with storm-0.9.x? I just made few changes and
>> tested it. Still it won't work.
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I have opened this stack overflow question,
>> http://stackoverflow.com/questions/40294245/deploy-
>> bolts-spout-to-a-specific-supervisor
>> Follow it and add question there,
>> And lets share information if any progress is made
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Date: Tuesday, 29 November 2016 at 13:49
>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>> user@storm.apache.org>
>>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> So we both used http://www.exogeni.net/2015/04/enabling-site-aware-
>> scheduling-for-apache-storm-in-exogeni/ tutorial. Then I guess we are
>> facing the problem due to the version.
>>
>> I think, we should try this our own until help arrives and if any of us
>> figured it out, let us inform each other.
>>
>> Cheers,
>> Regards,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I used this tutorial:
>>
>> http://www.exogeni.net/2015/04/enabling-site-aware-
>> scheduling-for-apache-storm-in-exogeni/
>>
>>
>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>> storm/DemoScheduler.java
>>
>> But its for storm 0.9.3
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Date: Tuesday, 29 November 2016 at 13:13
>> To: Ohad Edelstein <oh...@mintigo.com>
>>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi Ohad,
>>
>> Yeah, it works when I remove that meta data part from the storm.yaml. I
>> thought about that also. Could you please set "supervisor.scheduler.meta"
>> to "supervisor.meta" and see it works. However, it did not work for me. But
>> in someplace I saw it like that.
>>
>> I am working with storm-1.0.2.
>>
>> Let us both win at once 😉
>>
>> Cheers,
>> Regard,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I guess you added the meta data part to the storm.yaml in the supervisors…
>> I think that when ever the code has a problem with one of the params you
>> add there, it cause the workers not to start
>> When you remove the meta data the problem will go away.
>> My guess is that I am not writing the meta part correctly…
>> When I set a different parameter that was suppose to be array to a simple
>> string, it did the same thing…
>>
>> With what version of storm are you working with? And what instructions
>> are you following?
>> Thanks
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, 29 November 2016 at 12:55
>>
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi Ohad,
>>
>> My supervisors are running. But inside them, the workers are not working.
>> That is what I think the issue is. Do you have the same issue?
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> Hi, I am having the same problem on storm 1.0.1
>> I would love if you can share the information again,
>> I also had the same problem that Eranga is having.
>> The problem he has is that the supervisor can’t start because of
>> configuration problem…
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, 29 November 2016 at 11:00
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi,
>>
>> Thanks for the info. I implemented a custom scheduler according to the
>> link you suggested. I changed "getAvailableWorkerSlots" in the original
>> code to "getAvailableSlots". However when I submit my topology it shows on
>> the Storm UI. But when I click on it and look for more information I see
>> nothing. All the tables are emplty.
>>
>> I am here attaching the inside view of my topology. Could you please look
>> into that and say what I might have done wrong?
>>
>> Thanks,
>> Regards,
>>
>> ​
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <an...@gmail.com>
>> wrote:
>>
>> you have to use custom site aware scheduler for that . Or use ntp to sync
>> clocks for all VMs .
>>
>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Hi all,
>>
>> I have a cluster of four nodes. There is a benchmark I need to run on
>> cluster. Hovever I need to run "input bolt" and "output bolt" on the same
>> node in order to calculate the latency properly.
>>
>> Is there a way that I could run both the bols on the same node?
>>
>> Thanks,
>> Regards,
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
--
Thanks & Regards,
Anshu Shukla
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by Erik Weathers <ew...@groupon.com>.
What does NTP have to do with this? I mean, yes, if you're doing anything
on a cluster you should be running NTP, but please explain why you are
saying that is related to bottlenecking?
On Wed, Nov 30, 2016 at 8:38 PM anshu shukla <an...@gmail.com> wrote:
> I was also doing the same thing some days before . Although this is
> possible by custom scheduler but for getting higher rate ~100000 mps ,
> again problem will arise because machines with spout and sink will get
> bottlenecked.
>
> *So the best thing is to use some way to synchronize clock on bothe VMs
> (using NTP etc.). Currently I use a site aware custom scheduler with NTP
> running on master node. *
>
> On Thu, Dec 1, 2016 at 10:02 AM, Erik Weathers <ew...@groupon.com>
> wrote:
>
> Whoa. Too complicated. How many bolts and spouts are you trying to have?
>
> On Wed, Nov 30, 2016 at 8:28 PM Eranga Heshan <er...@gmail.com>
> wrote:
>
> Yeah, by "input bolt", I meant "spout".
>
> All I need is a strong and reliable method to calculate latency of an
> event. In my benchmark, I stamp a time into every event when spout
> generates it. When it arrives the output bolt (which means the process is
> complete), it checks for the time difference between output bolt's current
> time and the stamped time of the event. The problem is that spout might be
> on a different node other than output bolt. Hence there might be a small
> error when measuring the time difference (latency).
>
> By implementing a custom scheduler, I am trying to bind both the spout and
> output bolt to one node. Ultimately the time difference could be measured
> perfectly.
>
> I still could not understand what you meant by:
>
>
> >and the number of worker processes for the topology should also be an
> even divisor of the common factor for the spouts & bolts
>
> When you say "even divisor", did you mean "divides without any remainder"
> or "division result is even"? And when you say "common factor" did you mean
> "greatest common factor"? I will brief what I understood from your
> explanation.
>
> Assume "x" and "y" and "z" as integers:
> num_of_bolts = 2x //even multiple
> num_of_spouts = 2y //even multiple
> num_of_workers = z
> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
> f / z is even //even divisor of the common factor
>
> Did I understand you correctly or please correct me if I am wrong.
>
> I appreciate your concern about my matter.
>
> Thanks,
> Regards,
>
>
>
>
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
> wrote:
>
> Can you explain more about why you are trying this custom scheduler? (I'm
> afraid you're largely on your own if you *must* continue down that route).
> In your initial email you just said:
>
> > I need to run "input bolt" and "output bolt" on the same node in order
> to calculate the latency properly
>
> I do not understand why that requirement led you to conclude that you must
> use a custom scheduler. If you align everything appropriately in terms of
> executors and workers and hosts, and if you're using
> localOrShuffleGrouping, then you can ensure that all traffic will stay
> local to the host. By "appropriate alignment", I just mean that the number
> of spouts and bolts should be even multiples, and the number of worker
> processes for the topology should also be an even divisor of the common
> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
> 10 spouts and 20 bolts.
>
> Also when you say "input bolt", do you actually mean *spout*?
>
> - Erik
>
>
> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
> wrote:
>
> Can we have some help to solve this issue?
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
> wrote:
>
> Hi Ohad,
>
> Does your scheduler works with storm-0.9.x? I just made few changes and
> tested it. Still it won't work.
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> I have opened this stack overflow question,
>
> http://stackoverflow.com/questions/40294245/deploy-bolts-spout-to-a-specific-supervisor
> Follow it and add question there,
> And lets share information if any progress is made
>
> From: Eranga Heshan <er...@gmail.com>
> Date: Tuesday, 29 November 2016 at 13:49
> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
> user@storm.apache.org>
>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> So we both used
> http://www.exogeni.net/2015/04/enabling-site-aware-scheduling-for-apache-storm-in-exogeni/
> tutorial. Then I guess we are facing the problem due to the version.
>
> I think, we should try this our own until help arrives and if any of us
> figured it out, let us inform each other.
>
> Cheers,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> I used this tutorial:
>
>
> http://www.exogeni.net/2015/04/enabling-site-aware-scheduling-for-apache-storm-in-exogeni
> /
>
>
>
> https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/DemoScheduler.java
>
> But its for storm 0.9.3
>
> From: Eranga Heshan <er...@gmail.com>
> Date: Tuesday, 29 November 2016 at 13:13
> To: Ohad Edelstein <oh...@mintigo.com>
>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> Hi Ohad,
>
> Yeah, it works when I remove that meta data part from the storm.yaml. I
> thought about that also. Could you please set "supervisor.scheduler.meta"
> to "supervisor.meta" and see it works. However, it did not work for me. But
> in someplace I saw it like that.
>
> I am working with storm-1.0.2.
>
> Let us both win at once 😉
>
> Cheers,
> Regard,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> I guess you added the meta data part to the storm.yaml in the supervisors…
> I think that when ever the code has a problem with one of the params you
> add there, it cause the workers not to start
> When you remove the meta data the problem will go away.
> My guess is that I am not writing the meta part correctly…
> When I set a different parameter that was suppose to be array to a simple
> string, it did the same thing…
>
> With what version of storm are you working with? And what instructions are
> you following?
> Thanks
>
> From: Eranga Heshan <er...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, 29 November 2016 at 12:55
>
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> Hi Ohad,
>
> My supervisors are running. But inside them, the workers are not working.
> That is what I think the issue is. Do you have the same issue?
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> Hi, I am having the same problem on storm 1.0.1
> I would love if you can share the information again,
> I also had the same problem that Eranga is having.
> The problem he has is that the supervisor can’t start because of
> configuration problem…
>
> From: Eranga Heshan <er...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, 29 November 2016 at 11:00
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> Hi,
>
> Thanks for the info. I implemented a custom scheduler according to the
> link you suggested. I changed "getAvailableWorkerSlots" in the original
> code to "getAvailableSlots". However when I submit my topology it shows on
> the Storm UI. But when I click on it and look for more information I see
> nothing. All the tables are emplty.
>
> I am here attaching the inside view of my topology. Could you please look
> into that and say what I might have done wrong?
>
> Thanks,
> Regards,
>
> ​
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <an...@gmail.com>
> wrote:
>
> you have to use custom site aware scheduler for that . Or use ntp to sync
> clocks for all VMs .
>
> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <er...@gmail.com>
> wrote:
>
> Hi all,
>
> I have a cluster of four nodes. There is a benchmark I need to run on
> cluster. Hovever I need to run "input bolt" and "output bolt" on the same
> node in order to calculate the latency properly.
>
> Is there a way that I could run both the bols on the same node?
>
> Thanks,
> Regards,
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by anshu shukla <an...@gmail.com>.
I was also doing the same thing some days before . Although this is
possible by custom scheduler but for getting higher rate ~100000 mps ,
again problem will arise because machines with spout and sink will get
bottlenecked.
*So the best thing is to use some way to synchronize clock on bothe VMs
(using NTP etc.). Currently I use a site aware custom scheduler with NTP
running on master node. *
On Thu, Dec 1, 2016 at 10:02 AM, Erik Weathers <ew...@groupon.com>
wrote:
> Whoa. Too complicated. How many bolts and spouts are you trying to have?
>
> On Wed, Nov 30, 2016 at 8:28 PM Eranga Heshan <er...@gmail.com>
> wrote:
>
>> Yeah, by "input bolt", I meant "spout".
>>
>> All I need is a strong and reliable method to calculate latency of an
>> event. In my benchmark, I stamp a time into every event when spout
>> generates it. When it arrives the output bolt (which means the process is
>> complete), it checks for the time difference between output bolt's current
>> time and the stamped time of the event. The problem is that spout might be
>> on a different node other than output bolt. Hence there might be a small
>> error when measuring the time difference (latency).
>>
>> By implementing a custom scheduler, I am trying to bind both the spout
>> and output bolt to one node. Ultimately the time difference could be
>> measured perfectly.
>>
>> I still could not understand what you meant by:
>>
>>
>> >and the number of worker processes for the topology should also be an
>> even divisor of the common factor for the spouts & bolts
>>
>> When you say "even divisor", did you mean "divides without any remainder"
>> or "division result is even"? And when you say "common factor" did you mean
>> "greatest common factor"? I will brief what I understood from your
>> explanation.
>>
>> Assume "x" and "y" and "z" as integers:
>> num_of_bolts = 2x //even multiple
>> num_of_spouts = 2y //even multiple
>> num_of_workers = z
>> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
>> f / z is even //even divisor of the common factor
>>
>> Did I understand you correctly or please correct me if I am wrong.
>>
>> I appreciate your concern about my matter.
>>
>> Thanks,
>> Regards,
>>
>>
>>
>>
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>> Can you explain more about why you are trying this custom scheduler?
>> (I'm afraid you're largely on your own if you *must* continue down that
>> route).
>> In your initial email you just said:
>>
>> > I need to run "input bolt" and "output bolt" on the same node in order
>> to calculate the latency properly
>>
>> I do not understand why that requirement led you to conclude that you
>> must use a custom scheduler. If you align everything appropriately in
>> terms of executors and workers and hosts, and if you're using
>> localOrShuffleGrouping, then you can ensure that all traffic will stay
>> local to the host. By "appropriate alignment", I just mean that the number
>> of spouts and bolts should be even multiples, and the number of worker
>> processes for the topology should also be an even divisor of the common
>> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
>> 10 spouts and 20 bolts.
>>
>> Also when you say "input bolt", do you actually mean *spout*?
>>
>> - Erik
>>
>>
>> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Can we have some help to solve this issue?
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Hi Ohad,
>>
>> Does your scheduler works with storm-0.9.x? I just made few changes and
>> tested it. Still it won't work.
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I have opened this stack overflow question,
>> http://stackoverflow.com/questions/40294245/deploy-
>> bolts-spout-to-a-specific-supervisor
>> Follow it and add question there,
>> And lets share information if any progress is made
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Date: Tuesday, 29 November 2016 at 13:49
>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>> user@storm.apache.org>
>>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> So we both used http://www.exogeni.net/2015/04/enabling-site-aware-
>> scheduling-for-apache-storm-in-exogeni/ tutorial. Then I guess we are
>> facing the problem due to the version.
>>
>> I think, we should try this our own until help arrives and if any of us
>> figured it out, let us inform each other.
>>
>> Cheers,
>> Regards,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I used this tutorial:
>>
>> http://www.exogeni.net/2015/04/enabling-site-aware-
>> scheduling-for-apache-storm-in-exogeni/
>>
>>
>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>> storm/DemoScheduler.java
>>
>> But its for storm 0.9.3
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Date: Tuesday, 29 November 2016 at 13:13
>> To: Ohad Edelstein <oh...@mintigo.com>
>>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi Ohad,
>>
>> Yeah, it works when I remove that meta data part from the storm.yaml. I
>> thought about that also. Could you please set "supervisor.scheduler.meta"
>> to "supervisor.meta" and see it works. However, it did not work for me. But
>> in someplace I saw it like that.
>>
>> I am working with storm-1.0.2.
>>
>> Let us both win at once 😉
>>
>> Cheers,
>> Regard,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I guess you added the meta data part to the storm.yaml in the supervisors…
>> I think that when ever the code has a problem with one of the params you
>> add there, it cause the workers not to start
>> When you remove the meta data the problem will go away.
>> My guess is that I am not writing the meta part correctly…
>> When I set a different parameter that was suppose to be array to a simple
>> string, it did the same thing…
>>
>> With what version of storm are you working with? And what instructions
>> are you following?
>> Thanks
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, 29 November 2016 at 12:55
>>
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi Ohad,
>>
>> My supervisors are running. But inside them, the workers are not working.
>> That is what I think the issue is. Do you have the same issue?
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> Hi, I am having the same problem on storm 1.0.1
>> I would love if you can share the information again,
>> I also had the same problem that Eranga is having.
>> The problem he has is that the supervisor can’t start because of
>> configuration problem…
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, 29 November 2016 at 11:00
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi,
>>
>> Thanks for the info. I implemented a custom scheduler according to the
>> link you suggested. I changed "getAvailableWorkerSlots" in the original
>> code to "getAvailableSlots". However when I submit my topology it shows on
>> the Storm UI. But when I click on it and look for more information I see
>> nothing. All the tables are emplty.
>>
>> I am here attaching the inside view of my topology. Could you please look
>> into that and say what I might have done wrong?
>>
>> Thanks,
>> Regards,
>>
>> ​
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <an...@gmail.com>
>> wrote:
>>
>> you have to use custom site aware scheduler for that . Or use ntp to sync
>> clocks for all VMs .
>>
>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Hi all,
>>
>> I have a cluster of four nodes. There is a benchmark I need to run on
>> cluster. Hovever I need to run "input bolt" and "output bolt" on the same
>> node in order to calculate the latency properly.
>>
>> Is there a way that I could run both the bols on the same node?
>>
>> Thanks,
>> Regards,
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
--
Thanks & Regards,
Anshu Shukla
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by Erik Weathers <ew...@groupon.com>.
Frankly, I didn't get confused, I just didn't want to spend my time
decoding some abstract thing instead of using concrete numbers so we can
have some idea of what you are even trying to do. ;-)
Ok, so let's work with your numbers:
4 workers / supervisor * 5 supervisors == 20 worker processes.
Then... you only have *1* spout instance? Why don't you have 20 spouts?
You have all these workers doing *what* exactly? The strategy I'm
proposing is to have an identical pipeline of spouts to bolts within each
worker process, so that the performance is more or less uniform and there
is no cross-worker communication.
So, I would probably do this:
1 worker / supervisor * 5 supervisors == 5 worker processes
5 spouts
5 bolts of type A
5 bolts of type B
5 bolts of type C
If you *must* have only 1 spout and 1 bolt, then I don't see how you can
get nicely consistent performance numbers, unless you just put everything
on 1 host. Is there any reasoning behind your choice of 20 workers on 5
hosts?
I think what Anshu is proposing is a way to *attempt* to handle this, by
allowing you to have tuples flow between worker hosts and have the
timestamps be sensical. But you'll still have more randomness with the
cross-host communication.
- Erik
On Wed, Nov 30, 2016 at 9:02 PM, Eranga Heshan <er...@gmail.com> wrote:
> Hi Erik,
>
> Sorry you got confused. 😉 I am kind of a math geek, so I might have
> understood wrong.
>
> I have 4 supervisors each containing 5 workers. I have 4 bolts in my
> topology and one spout. Spout and the final bolt ("Output") have
> PARALLELISM of 1 while other three bolts have PARALLELISM of 8 each.
>
> Thanks,
> Cheers,
>
>
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Thu, Dec 1, 2016 at 10:02 AM, Erik Weathers <ew...@groupon.com>
> wrote:
>
>> Whoa. Too complicated. How many bolts and spouts are you trying to have?
>>
>> On Wed, Nov 30, 2016 at 8:28 PM Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>>> Yeah, by "input bolt", I meant "spout".
>>>
>>> All I need is a strong and reliable method to calculate latency of an
>>> event. In my benchmark, I stamp a time into every event when spout
>>> generates it. When it arrives the output bolt (which means the process is
>>> complete), it checks for the time difference between output bolt's current
>>> time and the stamped time of the event. The problem is that spout might be
>>> on a different node other than output bolt. Hence there might be a small
>>> error when measuring the time difference (latency).
>>>
>>> By implementing a custom scheduler, I am trying to bind both the spout
>>> and output bolt to one node. Ultimately the time difference could be
>>> measured perfectly.
>>>
>>> I still could not understand what you meant by:
>>>
>>>
>>> >and the number of worker processes for the topology should also be an
>>> even divisor of the common factor for the spouts & bolts
>>>
>>> When you say "even divisor", did you mean "divides without any
>>> remainder" or "division result is even"? And when you say "common factor"
>>> did you mean "greatest common factor"? I will brief what I understood from
>>> your explanation.
>>>
>>> Assume "x" and "y" and "z" as integers:
>>> num_of_bolts = 2x //even multiple
>>> num_of_spouts = 2y //even multiple
>>> num_of_workers = z
>>> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
>>> f / z is even //even divisor of the common factor
>>>
>>> Did I understand you correctly or please correct me if I am wrong.
>>>
>>> I appreciate your concern about my matter.
>>>
>>> Thanks,
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
>>> wrote:
>>>
>>> Can you explain more about why you are trying this custom scheduler?
>>> (I'm afraid you're largely on your own if you *must* continue down that
>>> route).
>>> In your initial email you just said:
>>>
>>> > I need to run "input bolt" and "output bolt" on the same node in
>>> order to calculate the latency properly
>>>
>>> I do not understand why that requirement led you to conclude that you
>>> must use a custom scheduler. If you align everything appropriately in
>>> terms of executors and workers and hosts, and if you're using
>>> localOrShuffleGrouping, then you can ensure that all traffic will stay
>>> local to the host. By "appropriate alignment", I just mean that the number
>>> of spouts and bolts should be even multiples, and the number of worker
>>> processes for the topology should also be an even divisor of the common
>>> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
>>> 10 spouts and 20 bolts.
>>>
>>> Also when you say "input bolt", do you actually mean *spout*?
>>>
>>> - Erik
>>>
>>>
>>> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
>>> wrote:
>>>
>>> Can we have some help to solve this issue?
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>>> wrote:
>>>
>>> Hi Ohad,
>>>
>>> Does your scheduler works with storm-0.9.x? I just made few changes and
>>> tested it. Still it won't work.
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>>> wrote:
>>>
>>> I have opened this stack overflow question,
>>> http://stackoverflow.com/questions/40294245/deploy-bolts-spo
>>> ut-to-a-specific-supervisor
>>> Follow it and add question there,
>>> And lets share information if any progress is made
>>>
>>> From: Eranga Heshan <er...@gmail.com>
>>> Date: Tuesday, 29 November 2016 at 13:49
>>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>>> user@storm.apache.org>
>>>
>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>
>>> So we both used http://www.exogeni.net/2015/04
>>> /enabling-site-aware-scheduling-for-apache-storm-in-exogeni/ tutorial.
>>> Then I guess we are facing the problem due to the version.
>>>
>>> I think, we should try this our own until help arrives and if any of us
>>> figured it out, let us inform each other.
>>>
>>> Cheers,
>>> Regards,
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>>> wrote:
>>>
>>> I used this tutorial:
>>>
>>> http://www.exogeni.net/2015/04/enabling-site-aware-schedulin
>>> g-for-apache-storm-in-exogeni/
>>>
>>>
>>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>>> storm/DemoScheduler.java
>>>
>>> But its for storm 0.9.3
>>>
>>> From: Eranga Heshan <er...@gmail.com>
>>> Date: Tuesday, 29 November 2016 at 13:13
>>> To: Ohad Edelstein <oh...@mintigo.com>
>>>
>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>
>>> Hi Ohad,
>>>
>>> Yeah, it works when I remove that meta data part from the storm.yaml. I
>>> thought about that also. Could you please set "supervisor.scheduler.meta"
>>> to "supervisor.meta" and see it works. However, it did not work for me. But
>>> in someplace I saw it like that.
>>>
>>> I am working with storm-1.0.2.
>>>
>>> Let us both win at once 😉
>>>
>>> Cheers,
>>> Regard,
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>>> wrote:
>>>
>>> I guess you added the meta data part to the storm.yaml in the
>>> supervisors…
>>> I think that when ever the code has a problem with one of the params you
>>> add there, it cause the workers not to start
>>> When you remove the meta data the problem will go away.
>>> My guess is that I am not writing the meta part correctly…
>>> When I set a different parameter that was suppose to be array to a
>>> simple string, it did the same thing…
>>>
>>> With what version of storm are you working with? And what instructions
>>> are you following?
>>> Thanks
>>>
>>> From: Eranga Heshan <er...@gmail.com>
>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Date: Tuesday, 29 November 2016 at 12:55
>>>
>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>
>>> Hi Ohad,
>>>
>>> My supervisors are running. But inside them, the workers are not
>>> working. That is what I think the issue is. Do you have the same issue?
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com>
>>> wrote:
>>>
>>> Hi, I am having the same problem on storm 1.0.1
>>> I would love if you can share the information again,
>>> I also had the same problem that Eranga is having.
>>> The problem he has is that the supervisor can’t start because of
>>> configuration problem…
>>>
>>> From: Eranga Heshan <er...@gmail.com>
>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Date: Tuesday, 29 November 2016 at 11:00
>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>
>>> Hi,
>>>
>>> Thanks for the info. I implemented a custom scheduler according to the
>>> link you suggested. I changed "getAvailableWorkerSlots" in the original
>>> code to "getAvailableSlots". However when I submit my topology it shows on
>>> the Storm UI. But when I click on it and look for more information I see
>>> nothing. All the tables are emplty.
>>>
>>> I am here attaching the inside view of my topology. Could you please
>>> look into that and say what I might have done wrong?
>>>
>>> Thanks,
>>> Regards,
>>>
>>> ​
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <an...@gmail.com>
>>> wrote:
>>>
>>> you have to use custom site aware scheduler for that . Or use ntp to
>>> sync clocks for all VMs .
>>>
>>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <er...@gmail.com>
>>> wrote:
>>>
>>> Hi all,
>>>
>>> I have a cluster of four nodes. There is a benchmark I need to run on
>>> cluster. Hovever I need to run "input bolt" and "output bolt" on the same
>>> node in order to calculate the latency properly.
>>>
>>> Is there a way that I could run both the bols on the same node?
>>>
>>> Thanks,
>>> Regards,
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by Eranga Heshan <er...@gmail.com>.
Hi Erik,
Sorry you got confused. 😉 I am kind of a math geek, so I might have
understood wrong.
I have 4 supervisors each containing 5 workers. I have 4 bolts in my
topology and one spout. Spout and the final bolt ("Output") have
PARALLELISM of 1 while other three bolts have PARALLELISM of 8 each.
Thanks,
Cheers,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
On Thu, Dec 1, 2016 at 10:02 AM, Erik Weathers <ew...@groupon.com>
wrote:
> Whoa. Too complicated. How many bolts and spouts are you trying to have?
>
> On Wed, Nov 30, 2016 at 8:28 PM Eranga Heshan <er...@gmail.com>
> wrote:
>
>> Yeah, by "input bolt", I meant "spout".
>>
>> All I need is a strong and reliable method to calculate latency of an
>> event. In my benchmark, I stamp a time into every event when spout
>> generates it. When it arrives the output bolt (which means the process is
>> complete), it checks for the time difference between output bolt's current
>> time and the stamped time of the event. The problem is that spout might be
>> on a different node other than output bolt. Hence there might be a small
>> error when measuring the time difference (latency).
>>
>> By implementing a custom scheduler, I am trying to bind both the spout
>> and output bolt to one node. Ultimately the time difference could be
>> measured perfectly.
>>
>> I still could not understand what you meant by:
>>
>>
>> >and the number of worker processes for the topology should also be an
>> even divisor of the common factor for the spouts & bolts
>>
>> When you say "even divisor", did you mean "divides without any remainder"
>> or "division result is even"? And when you say "common factor" did you mean
>> "greatest common factor"? I will brief what I understood from your
>> explanation.
>>
>> Assume "x" and "y" and "z" as integers:
>> num_of_bolts = 2x //even multiple
>> num_of_spouts = 2y //even multiple
>> num_of_workers = z
>> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
>> f / z is even //even divisor of the common factor
>>
>> Did I understand you correctly or please correct me if I am wrong.
>>
>> I appreciate your concern about my matter.
>>
>> Thanks,
>> Regards,
>>
>>
>>
>>
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>> Can you explain more about why you are trying this custom scheduler?
>> (I'm afraid you're largely on your own if you *must* continue down that
>> route).
>> In your initial email you just said:
>>
>> > I need to run "input bolt" and "output bolt" on the same node in order
>> to calculate the latency properly
>>
>> I do not understand why that requirement led you to conclude that you
>> must use a custom scheduler. If you align everything appropriately in
>> terms of executors and workers and hosts, and if you're using
>> localOrShuffleGrouping, then you can ensure that all traffic will stay
>> local to the host. By "appropriate alignment", I just mean that the number
>> of spouts and bolts should be even multiples, and the number of worker
>> processes for the topology should also be an even divisor of the common
>> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
>> 10 spouts and 20 bolts.
>>
>> Also when you say "input bolt", do you actually mean *spout*?
>>
>> - Erik
>>
>>
>> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Can we have some help to solve this issue?
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Hi Ohad,
>>
>> Does your scheduler works with storm-0.9.x? I just made few changes and
>> tested it. Still it won't work.
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I have opened this stack overflow question,
>> http://stackoverflow.com/questions/40294245/deploy-
>> bolts-spout-to-a-specific-supervisor
>> Follow it and add question there,
>> And lets share information if any progress is made
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Date: Tuesday, 29 November 2016 at 13:49
>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>> user@storm.apache.org>
>>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> So we both used http://www.exogeni.net/2015/04/enabling-site-aware-
>> scheduling-for-apache-storm-in-exogeni/ tutorial. Then I guess we are
>> facing the problem due to the version.
>>
>> I think, we should try this our own until help arrives and if any of us
>> figured it out, let us inform each other.
>>
>> Cheers,
>> Regards,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I used this tutorial:
>>
>> http://www.exogeni.net/2015/04/enabling-site-aware-
>> scheduling-for-apache-storm-in-exogeni/
>>
>>
>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>> storm/DemoScheduler.java
>>
>> But its for storm 0.9.3
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Date: Tuesday, 29 November 2016 at 13:13
>> To: Ohad Edelstein <oh...@mintigo.com>
>>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi Ohad,
>>
>> Yeah, it works when I remove that meta data part from the storm.yaml. I
>> thought about that also. Could you please set "supervisor.scheduler.meta"
>> to "supervisor.meta" and see it works. However, it did not work for me. But
>> in someplace I saw it like that.
>>
>> I am working with storm-1.0.2.
>>
>> Let us both win at once 😉
>>
>> Cheers,
>> Regard,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> I guess you added the meta data part to the storm.yaml in the supervisors…
>> I think that when ever the code has a problem with one of the params you
>> add there, it cause the workers not to start
>> When you remove the meta data the problem will go away.
>> My guess is that I am not writing the meta part correctly…
>> When I set a different parameter that was suppose to be array to a simple
>> string, it did the same thing…
>>
>> With what version of storm are you working with? And what instructions
>> are you following?
>> Thanks
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, 29 November 2016 at 12:55
>>
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi Ohad,
>>
>> My supervisors are running. But inside them, the workers are not working.
>> That is what I think the issue is. Do you have the same issue?
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com>
>> wrote:
>>
>> Hi, I am having the same problem on storm 1.0.1
>> I would love if you can share the information again,
>> I also had the same problem that Eranga is having.
>> The problem he has is that the supervisor can’t start because of
>> configuration problem…
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, 29 November 2016 at 11:00
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi,
>>
>> Thanks for the info. I implemented a custom scheduler according to the
>> link you suggested. I changed "getAvailableWorkerSlots" in the original
>> code to "getAvailableSlots". However when I submit my topology it shows on
>> the Storm UI. But when I click on it and look for more information I see
>> nothing. All the tables are emplty.
>>
>> I am here attaching the inside view of my topology. Could you please look
>> into that and say what I might have done wrong?
>>
>> Thanks,
>> Regards,
>>
>> ​
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <an...@gmail.com>
>> wrote:
>>
>> you have to use custom site aware scheduler for that . Or use ntp to sync
>> clocks for all VMs .
>>
>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>> Hi all,
>>
>> I have a cluster of four nodes. There is a benchmark I need to run on
>> cluster. Hovever I need to run "input bolt" and "output bolt" on the same
>> node in order to calculate the latency properly.
>>
>> Is there a way that I could run both the bols on the same node?
>>
>> Thanks,
>> Regards,
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by Erik Weathers <ew...@groupon.com>.
Whoa. Too complicated. How many bolts and spouts are you trying to have?
On Wed, Nov 30, 2016 at 8:28 PM Eranga Heshan <er...@gmail.com> wrote:
> Yeah, by "input bolt", I meant "spout".
>
> All I need is a strong and reliable method to calculate latency of an
> event. In my benchmark, I stamp a time into every event when spout
> generates it. When it arrives the output bolt (which means the process is
> complete), it checks for the time difference between output bolt's current
> time and the stamped time of the event. The problem is that spout might be
> on a different node other than output bolt. Hence there might be a small
> error when measuring the time difference (latency).
>
> By implementing a custom scheduler, I am trying to bind both the spout and
> output bolt to one node. Ultimately the time difference could be measured
> perfectly.
>
> I still could not understand what you meant by:
>
>
> >and the number of worker processes for the topology should also be an
> even divisor of the common factor for the spouts & bolts
>
> When you say "even divisor", did you mean "divides without any remainder"
> or "division result is even"? And when you say "common factor" did you mean
> "greatest common factor"? I will brief what I understood from your
> explanation.
>
> Assume "x" and "y" and "z" as integers:
> num_of_bolts = 2x //even multiple
> num_of_spouts = 2y //even multiple
> num_of_workers = z
> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
> f / z is even //even divisor of the common factor
>
> Did I understand you correctly or please correct me if I am wrong.
>
> I appreciate your concern about my matter.
>
> Thanks,
> Regards,
>
>
>
>
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
> wrote:
>
> Can you explain more about why you are trying this custom scheduler? (I'm
> afraid you're largely on your own if you *must* continue down that route).
> In your initial email you just said:
>
> > I need to run "input bolt" and "output bolt" on the same node in order
> to calculate the latency properly
>
> I do not understand why that requirement led you to conclude that you must
> use a custom scheduler. If you align everything appropriately in terms of
> executors and workers and hosts, and if you're using
> localOrShuffleGrouping, then you can ensure that all traffic will stay
> local to the host. By "appropriate alignment", I just mean that the number
> of spouts and bolts should be even multiples, and the number of worker
> processes for the topology should also be an even divisor of the common
> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
> 10 spouts and 20 bolts.
>
> Also when you say "input bolt", do you actually mean *spout*?
>
> - Erik
>
>
> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
> wrote:
>
> Can we have some help to solve this issue?
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
> wrote:
>
> Hi Ohad,
>
> Does your scheduler works with storm-0.9.x? I just made few changes and
> tested it. Still it won't work.
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> I have opened this stack overflow question,
>
> http://stackoverflow.com/questions/40294245/deploy-bolts-spout-to-a-specific-supervisor
> Follow it and add question there,
> And lets share information if any progress is made
>
> From: Eranga Heshan <er...@gmail.com>
> Date: Tuesday, 29 November 2016 at 13:49
> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
> user@storm.apache.org>
>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> So we both used
> http://www.exogeni.net/2015/04/enabling-site-aware-scheduling-for-apache-storm-in-exogeni/
> tutorial. Then I guess we are facing the problem due to the version.
>
> I think, we should try this our own until help arrives and if any of us
> figured it out, let us inform each other.
>
> Cheers,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> I used this tutorial:
>
>
> http://www.exogeni.net/2015/04/enabling-site-aware-scheduling-for-apache-storm-in-exogeni
> /
>
>
>
> https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/DemoScheduler.java
>
> But its for storm 0.9.3
>
> From: Eranga Heshan <er...@gmail.com>
> Date: Tuesday, 29 November 2016 at 13:13
> To: Ohad Edelstein <oh...@mintigo.com>
>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> Hi Ohad,
>
> Yeah, it works when I remove that meta data part from the storm.yaml. I
> thought about that also. Could you please set "supervisor.scheduler.meta"
> to "supervisor.meta" and see it works. However, it did not work for me. But
> in someplace I saw it like that.
>
> I am working with storm-1.0.2.
>
> Let us both win at once 😉
>
> Cheers,
> Regard,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> I guess you added the meta data part to the storm.yaml in the supervisors…
> I think that when ever the code has a problem with one of the params you
> add there, it cause the workers not to start
> When you remove the meta data the problem will go away.
> My guess is that I am not writing the meta part correctly…
> When I set a different parameter that was suppose to be array to a simple
> string, it did the same thing…
>
> With what version of storm are you working with? And what instructions are
> you following?
> Thanks
>
> From: Eranga Heshan <er...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, 29 November 2016 at 12:55
>
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> Hi Ohad,
>
> My supervisors are running. But inside them, the workers are not working.
> That is what I think the issue is. Do you have the same issue?
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
> Hi, I am having the same problem on storm 1.0.1
> I would love if you can share the information again,
> I also had the same problem that Eranga is having.
> The problem he has is that the supervisor can’t start because of
> configuration problem…
>
> From: Eranga Heshan <er...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, 29 November 2016 at 11:00
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> Hi,
>
> Thanks for the info. I implemented a custom scheduler according to the
> link you suggested. I changed "getAvailableWorkerSlots" in the original
> code to "getAvailableSlots". However when I submit my topology it shows on
> the Storm UI. But when I click on it and look for more information I see
> nothing. All the tables are emplty.
>
> I am here attaching the inside view of my topology. Could you please look
> into that and say what I might have done wrong?
>
> Thanks,
> Regards,
>
> ​
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <an...@gmail.com>
> wrote:
>
> you have to use custom site aware scheduler for that . Or use ntp to sync
> clocks for all VMs .
>
> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <er...@gmail.com>
> wrote:
>
> Hi all,
>
> I have a cluster of four nodes. There is a benchmark I need to run on
> cluster. Hovever I need to run "input bolt" and "output bolt" on the same
> node in order to calculate the latency properly.
>
> Is there a way that I could run both the bols on the same node?
>
> Thanks,
> Regards,
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
>
>
>
>
>
>
>
>
>
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by Eranga Heshan <er...@gmail.com>.
Haha :-)
I have 5 workers / supervisor * 4 supervisors == 20 worker processes. ;-)
Basically the same result you said.
I agree with your way of approach and I will test the performance of that
approach as well. The cross communication among workers and also among
hosts would be an issue.
For now, why I had only one spout is because I thought it sends events in
much higher rate than bolts can actually process them. For having 5 workers
for each host is not because of any particular reason.
Thanks,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
On Thu, Dec 1, 2016 at 10:58 AM, Eranga Heshan <er...@gmail.com> wrote:
> Yeah Ohad 😃. Everything works fine once I commented that line. That is
> all. Enjoy!
>
> Cheers,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Thu, Dec 1, 2016 at 10:56 AM, Ohad Edelstein <oh...@mintigo.com> wrote:
>
>> Really?!? Nice!
>> The meta data part was not the problem?
>> The workers are now… working for you? :)
>>
>>
>> From: Eranga Heshan <er...@gmail.com>
>> Date: Thursday, 1 December 2016 at 6:37
>> To: "user@storm.apache.org" <us...@storm.apache.org>, Ohad Edelstein <
>> ohade@mintigo.com>
>>
>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>
>> Hi Ohad,
>>
>> I was able to run the scheduler properly. The problem is in the following
>> conditional statement:
>>
>> "if(cluster.needsScheduling(t)) continue;"
>>
>> Remove it and hopefully it would work.
>>
>> Cheers,
>> Regards,
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Thu, Dec 1, 2016 at 9:57 AM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>>> Yeah, by "input bolt", I meant "spout".
>>>
>>> All I need is a strong and reliable method to calculate latency of an
>>> event. In my benchmark, I stamp a time into every event when spout
>>> generates it. When it arrives the output bolt (which means the process is
>>> complete), it checks for the time difference between output bolt's current
>>> time and the stamped time of the event. The problem is that spout might be
>>> on a different node other than output bolt. Hence there might be a small
>>> error when measuring the time difference (latency).
>>>
>>> By implementing a custom scheduler, I am trying to bind both the spout
>>> and output bolt to one node. Ultimately the time difference could be
>>> measured perfectly.
>>>
>>> I still could not understand what you meant by:
>>>
>>> >and the number of worker processes for the topology should also be an
>>> even divisor of the common factor for the spouts & bolts
>>>
>>> When you say "even divisor", did you mean "divides without any
>>> remainder" or "division result is even"? And when you say "common factor"
>>> did you mean "greatest common factor"? I will brief what I understood from
>>> your explanation.
>>>
>>> Assume "x" and "y" and "z" as integers:
>>> num_of_bolts = 2x //even multiple
>>> num_of_spouts = 2y //even multiple
>>> num_of_workers = z
>>> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
>>> f / z is even //even divisor of the common factor
>>>
>>> Did I understand you correctly or please correct me if I am wrong.
>>>
>>> I appreciate your concern about my matter.
>>>
>>> Thanks,
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
>>> wrote:
>>>
>>>> Can you explain more about why you are trying this custom scheduler?
>>>> (I'm afraid you're largely on your own if you *must* continue down that
>>>> route).
>>>> In your initial email you just said:
>>>>
>>>> > I need to run "input bolt" and "output bolt" on the same node in
>>>> order to calculate the latency properly
>>>>
>>>> I do not understand why that requirement led you to conclude that you
>>>> must use a custom scheduler. If you align everything appropriately in
>>>> terms of executors and workers and hosts, and if you're using
>>>> localOrShuffleGrouping, then you can ensure that all traffic will stay
>>>> local to the host. By "appropriate alignment", I just mean that the number
>>>> of spouts and bolts should be even multiples, and the number of worker
>>>> processes for the topology should also be an even divisor of the common
>>>> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
>>>> 10 spouts and 20 bolts.
>>>>
>>>> Also when you say "input bolt", do you actually mean *spout*?
>>>>
>>>> - Erik
>>>>
>>>>
>>>> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
>>>> wrote:
>>>>
>>>>> Can we have some help to solve this issue?
>>>>>
>>>>>
>>>>> Eranga Heshan
>>>>> *Undergraduate*
>>>>> Computer Science & Engineering
>>>>> University of Moratuwa
>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>> <https://www.facebook.com/erangaheshan>
>>>>> <https://twitter.com/erangaheshan>
>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>
>>>>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Ohad,
>>>>>>
>>>>>> Does your scheduler works with storm-0.9.x? I just made few changes
>>>>>> and tested it. Still it won't work.
>>>>>>
>>>>>>
>>>>>> Eranga Heshan
>>>>>> *Undergraduate*
>>>>>> Computer Science & Engineering
>>>>>> University of Moratuwa
>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>> <https://twitter.com/erangaheshan>
>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>
>>>>>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have opened this stack overflow question,
>>>>>>> http://stackoverflow.com/questions/40294245/deploy-bolts-spo
>>>>>>> ut-to-a-specific-supervisor
>>>>>>> Follow it and add question there,
>>>>>>> And lets share information if any progress is made
>>>>>>>
>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>> Date: Tuesday, 29 November 2016 at 13:49
>>>>>>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>>>>>>> user@storm.apache.org>
>>>>>>>
>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>
>>>>>>> So we both used http://www.exogeni.net/2015/04
>>>>>>> /enabling-site-aware-scheduling-for-apache-storm-in-exogeni/
>>>>>>> tutorial. Then I guess we are facing the problem due to the version.
>>>>>>>
>>>>>>> I think, we should try this our own until help arrives and if any of
>>>>>>> us figured it out, let us inform each other.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>> Eranga Heshan
>>>>>>> *Undergraduate*
>>>>>>> Computer Science & Engineering
>>>>>>> University of Moratuwa
>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>
>>>>>>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I used this tutorial:
>>>>>>>>
>>>>>>>> http://www.exogeni.net/2015/04/enabling-site-aware-schedulin
>>>>>>>> g-for-apache-storm-in-exogeni/
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>>>>>>>> storm/DemoScheduler.java
>>>>>>>>
>>>>>>>> But its for storm 0.9.3
>>>>>>>>
>>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>>> Date: Tuesday, 29 November 2016 at 13:13
>>>>>>>> To: Ohad Edelstein <oh...@mintigo.com>
>>>>>>>>
>>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>>
>>>>>>>> Hi Ohad,
>>>>>>>>
>>>>>>>> Yeah, it works when I remove that meta data part from the
>>>>>>>> storm.yaml. I thought about that also. Could you please set
>>>>>>>> "supervisor.scheduler.meta" to "supervisor.meta" and see it works. However,
>>>>>>>> it did not work for me. But in someplace I saw it like that.
>>>>>>>>
>>>>>>>> I am working with storm-1.0.2.
>>>>>>>>
>>>>>>>> Let us both win at once 😉
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Regard,
>>>>>>>>
>>>>>>>>
>>>>>>>> Eranga Heshan
>>>>>>>> *Undergraduate*
>>>>>>>> Computer Science & Engineering
>>>>>>>> University of Moratuwa
>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>
>>>>>>>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I guess you added the meta data part to the storm.yaml in the
>>>>>>>>> supervisors…
>>>>>>>>> I think that when ever the code has a problem with one of the
>>>>>>>>> params you add there, it cause the workers not to start
>>>>>>>>> When you remove the meta data the problem will go away.
>>>>>>>>> My guess is that I am not writing the meta part correctly…
>>>>>>>>> When I set a different parameter that was suppose to be array to a
>>>>>>>>> simple string, it did the same thing…
>>>>>>>>>
>>>>>>>>> With what version of storm are you working with? And what
>>>>>>>>> instructions are you following?
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>>> Date: Tuesday, 29 November 2016 at 12:55
>>>>>>>>>
>>>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>>>
>>>>>>>>> Hi Ohad,
>>>>>>>>>
>>>>>>>>> My supervisors are running. But inside them, the workers are not
>>>>>>>>> working. That is what I think the issue is. Do you have the same issue?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Eranga Heshan
>>>>>>>>> *Undergraduate*
>>>>>>>>> Computer Science & Engineering
>>>>>>>>> University of Moratuwa
>>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>>
>>>>>>>>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <ohade@mintigo.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hi, I am having the same problem on storm 1.0.1
>>>>>>>>>> I would love if you can share the information again,
>>>>>>>>>> I also had the same problem that Eranga is having.
>>>>>>>>>> The problem he has is that the supervisor can’t start because of
>>>>>>>>>> configuration problem…
>>>>>>>>>>
>>>>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>>>> Date: Tuesday, 29 November 2016 at 11:00
>>>>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Thanks for the info. I implemented a custom scheduler according
>>>>>>>>>> to the link you suggested. I changed "getAvailableWorkerSlots" in the
>>>>>>>>>> original code to "getAvailableSlots". However when I submit my topology it
>>>>>>>>>> shows on the Storm UI. But when I click on it and look for more information
>>>>>>>>>> I see nothing. All the tables are emplty.
>>>>>>>>>>
>>>>>>>>>> I am here attaching the inside view of my topology. Could you
>>>>>>>>>> please look into that and say what I might have done wrong?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> ​
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Eranga Heshan
>>>>>>>>>> *Undergraduate*
>>>>>>>>>> Computer Science & Engineering
>>>>>>>>>> University of Moratuwa
>>>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <
>>>>>>>>>> anshushukla0@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> you have to use custom site aware scheduler for that . Or use
>>>>>>>>>>> ntp to sync clocks for all VMs .
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <
>>>>>>>>>>> eranga.h.n@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> I have a cluster of four nodes. There is a benchmark I need to
>>>>>>>>>>>> run on cluster. Hovever I need to run "input bolt" and "output bolt" on the
>>>>>>>>>>>> same node in order to calculate the latency properly.
>>>>>>>>>>>>
>>>>>>>>>>>> Is there a way that I could run both the bols on the same node?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Eranga Heshan
>>>>>>>>>>>> *Undergraduate*
>>>>>>>>>>>> Computer Science & Engineering
>>>>>>>>>>>> University of Moratuwa
>>>>>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>> Anshu Shukla
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by Eranga Heshan <er...@gmail.com>.
Yeah Ohad 😃. Everything works fine once I commented that line. That is
all. Enjoy!
Cheers,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
On Thu, Dec 1, 2016 at 10:56 AM, Ohad Edelstein <oh...@mintigo.com> wrote:
> Really?!? Nice!
> The meta data part was not the problem?
> The workers are now… working for you? :)
>
>
> From: Eranga Heshan <er...@gmail.com>
> Date: Thursday, 1 December 2016 at 6:37
> To: "user@storm.apache.org" <us...@storm.apache.org>, Ohad Edelstein <
> ohade@mintigo.com>
>
> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>
> Hi Ohad,
>
> I was able to run the scheduler properly. The problem is in the following
> conditional statement:
>
> "if(cluster.needsScheduling(t)) continue;"
>
> Remove it and hopefully it would work.
>
> Cheers,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Thu, Dec 1, 2016 at 9:57 AM, Eranga Heshan <er...@gmail.com>
> wrote:
>
>> Yeah, by "input bolt", I meant "spout".
>>
>> All I need is a strong and reliable method to calculate latency of an
>> event. In my benchmark, I stamp a time into every event when spout
>> generates it. When it arrives the output bolt (which means the process is
>> complete), it checks for the time difference between output bolt's current
>> time and the stamped time of the event. The problem is that spout might be
>> on a different node other than output bolt. Hence there might be a small
>> error when measuring the time difference (latency).
>>
>> By implementing a custom scheduler, I am trying to bind both the spout
>> and output bolt to one node. Ultimately the time difference could be
>> measured perfectly.
>>
>> I still could not understand what you meant by:
>>
>> >and the number of worker processes for the topology should also be an
>> even divisor of the common factor for the spouts & bolts
>>
>> When you say "even divisor", did you mean "divides without any remainder"
>> or "division result is even"? And when you say "common factor" did you mean
>> "greatest common factor"? I will brief what I understood from your
>> explanation.
>>
>> Assume "x" and "y" and "z" as integers:
>> num_of_bolts = 2x //even multiple
>> num_of_spouts = 2y //even multiple
>> num_of_workers = z
>> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
>> f / z is even //even divisor of the common factor
>>
>> Did I understand you correctly or please correct me if I am wrong.
>>
>> I appreciate your concern about my matter.
>>
>> Thanks,
>> Regards,
>>
>>
>>
>>
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga@wso2.com <os...@gmail.com>
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>>> Can you explain more about why you are trying this custom scheduler?
>>> (I'm afraid you're largely on your own if you *must* continue down that
>>> route).
>>> In your initial email you just said:
>>>
>>> > I need to run "input bolt" and "output bolt" on the same node in
>>> order to calculate the latency properly
>>>
>>> I do not understand why that requirement led you to conclude that you
>>> must use a custom scheduler. If you align everything appropriately in
>>> terms of executors and workers and hosts, and if you're using
>>> localOrShuffleGrouping, then you can ensure that all traffic will stay
>>> local to the host. By "appropriate alignment", I just mean that the number
>>> of spouts and bolts should be even multiples, and the number of worker
>>> processes for the topology should also be an even divisor of the common
>>> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
>>> 10 spouts and 20 bolts.
>>>
>>> Also when you say "input bolt", do you actually mean *spout*?
>>>
>>> - Erik
>>>
>>>
>>> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
>>> wrote:
>>>
>>>> Can we have some help to solve this issue?
>>>>
>>>>
>>>> Eranga Heshan
>>>> *Undergraduate*
>>>> Computer Science & Engineering
>>>> University of Moratuwa
>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>> <https://www.facebook.com/erangaheshan>
>>>> <https://twitter.com/erangaheshan>
>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>
>>>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Ohad,
>>>>>
>>>>> Does your scheduler works with storm-0.9.x? I just made few changes
>>>>> and tested it. Still it won't work.
>>>>>
>>>>>
>>>>> Eranga Heshan
>>>>> *Undergraduate*
>>>>> Computer Science & Engineering
>>>>> University of Moratuwa
>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>> <https://www.facebook.com/erangaheshan>
>>>>> <https://twitter.com/erangaheshan>
>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>
>>>>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>> wrote:
>>>>>
>>>>>> I have opened this stack overflow question,
>>>>>> http://stackoverflow.com/questions/40294245/deploy-bolts-spo
>>>>>> ut-to-a-specific-supervisor
>>>>>> Follow it and add question there,
>>>>>> And lets share information if any progress is made
>>>>>>
>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>> Date: Tuesday, 29 November 2016 at 13:49
>>>>>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>>>>>> user@storm.apache.org>
>>>>>>
>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>
>>>>>> So we both used http://www.exogeni.net/2015/04
>>>>>> /enabling-site-aware-scheduling-for-apache-storm-in-exogeni/
>>>>>> tutorial. Then I guess we are facing the problem due to the version.
>>>>>>
>>>>>> I think, we should try this our own until help arrives and if any of
>>>>>> us figured it out, let us inform each other.
>>>>>>
>>>>>> Cheers,
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> Eranga Heshan
>>>>>> *Undergraduate*
>>>>>> Computer Science & Engineering
>>>>>> University of Moratuwa
>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>> <https://twitter.com/erangaheshan>
>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>
>>>>>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I used this tutorial:
>>>>>>>
>>>>>>> http://www.exogeni.net/2015/04/enabling-site-aware-schedulin
>>>>>>> g-for-apache-storm-in-exogeni/
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>>>>>>> storm/DemoScheduler.java
>>>>>>>
>>>>>>> But its for storm 0.9.3
>>>>>>>
>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>> Date: Tuesday, 29 November 2016 at 13:13
>>>>>>> To: Ohad Edelstein <oh...@mintigo.com>
>>>>>>>
>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>
>>>>>>> Hi Ohad,
>>>>>>>
>>>>>>> Yeah, it works when I remove that meta data part from the
>>>>>>> storm.yaml. I thought about that also. Could you please set
>>>>>>> "supervisor.scheduler.meta" to "supervisor.meta" and see it works. However,
>>>>>>> it did not work for me. But in someplace I saw it like that.
>>>>>>>
>>>>>>> I am working with storm-1.0.2.
>>>>>>>
>>>>>>> Let us both win at once 😉
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Regard,
>>>>>>>
>>>>>>>
>>>>>>> Eranga Heshan
>>>>>>> *Undergraduate*
>>>>>>> Computer Science & Engineering
>>>>>>> University of Moratuwa
>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>
>>>>>>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I guess you added the meta data part to the storm.yaml in the
>>>>>>>> supervisors…
>>>>>>>> I think that when ever the code has a problem with one of the
>>>>>>>> params you add there, it cause the workers not to start
>>>>>>>> When you remove the meta data the problem will go away.
>>>>>>>> My guess is that I am not writing the meta part correctly…
>>>>>>>> When I set a different parameter that was suppose to be array to a
>>>>>>>> simple string, it did the same thing…
>>>>>>>>
>>>>>>>> With what version of storm are you working with? And what
>>>>>>>> instructions are you following?
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>> Date: Tuesday, 29 November 2016 at 12:55
>>>>>>>>
>>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>>
>>>>>>>> Hi Ohad,
>>>>>>>>
>>>>>>>> My supervisors are running. But inside them, the workers are not
>>>>>>>> working. That is what I think the issue is. Do you have the same issue?
>>>>>>>>
>>>>>>>>
>>>>>>>> Eranga Heshan
>>>>>>>> *Undergraduate*
>>>>>>>> Computer Science & Engineering
>>>>>>>> University of Moratuwa
>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>
>>>>>>>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi, I am having the same problem on storm 1.0.1
>>>>>>>>> I would love if you can share the information again,
>>>>>>>>> I also had the same problem that Eranga is having.
>>>>>>>>> The problem he has is that the supervisor can’t start because of
>>>>>>>>> configuration problem…
>>>>>>>>>
>>>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>>> Date: Tuesday, 29 November 2016 at 11:00
>>>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Thanks for the info. I implemented a custom scheduler according to
>>>>>>>>> the link you suggested. I changed "getAvailableWorkerSlots" in the original
>>>>>>>>> code to "getAvailableSlots". However when I submit my topology it shows on
>>>>>>>>> the Storm UI. But when I click on it and look for more information I see
>>>>>>>>> nothing. All the tables are emplty.
>>>>>>>>>
>>>>>>>>> I am here attaching the inside view of my topology. Could you
>>>>>>>>> please look into that and say what I might have done wrong?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> ​
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Eranga Heshan
>>>>>>>>> *Undergraduate*
>>>>>>>>> Computer Science & Engineering
>>>>>>>>> University of Moratuwa
>>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>>
>>>>>>>>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <
>>>>>>>>> anshushukla0@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> you have to use custom site aware scheduler for that . Or use
>>>>>>>>>> ntp to sync clocks for all VMs .
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <
>>>>>>>>>> eranga.h.n@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I have a cluster of four nodes. There is a benchmark I need to
>>>>>>>>>>> run on cluster. Hovever I need to run "input bolt" and "output bolt" on the
>>>>>>>>>>> same node in order to calculate the latency properly.
>>>>>>>>>>>
>>>>>>>>>>> Is there a way that I could run both the bols on the same node?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Eranga Heshan
>>>>>>>>>>> *Undergraduate*
>>>>>>>>>>> Computer Science & Engineering
>>>>>>>>>>> University of Moratuwa
>>>>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Anshu Shukla
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Schedule a Bolt/Spout to a Predefined Node
Posted by Eranga Heshan <er...@gmail.com>.
Hi Ohad,
I was able to run the scheduler properly. The problem is in the following
conditional statement:
"if(cluster.needsScheduling(t)) continue;"
Remove it and hopefully it would work.
Cheers,
Regards,
Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga@wso2.com <os...@gmail.com>
<https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan>
<https://www.linkedin.com/in/erangaheshan>
On Thu, Dec 1, 2016 at 9:57 AM, Eranga Heshan <er...@gmail.com> wrote:
> Yeah, by "input bolt", I meant "spout".
>
> All I need is a strong and reliable method to calculate latency of an
> event. In my benchmark, I stamp a time into every event when spout
> generates it. When it arrives the output bolt (which means the process is
> complete), it checks for the time difference between output bolt's current
> time and the stamped time of the event. The problem is that spout might be
> on a different node other than output bolt. Hence there might be a small
> error when measuring the time difference (latency).
>
> By implementing a custom scheduler, I am trying to bind both the spout and
> output bolt to one node. Ultimately the time difference could be measured
> perfectly.
>
> I still could not understand what you meant by:
>
> >and the number of worker processes for the topology should also be an
> even divisor of the common factor for the spouts & bolts
>
> When you say "even divisor", did you mean "divides without any remainder"
> or "division result is even"? And when you say "common factor" did you mean
> "greatest common factor"? I will brief what I understood from your
> explanation.
>
> Assume "x" and "y" and "z" as integers:
> num_of_bolts = 2x //even multiple
> num_of_spouts = 2y //even multiple
> num_of_workers = z
> f = gereatest_common_factor (num_of_bolts, num_of_spouts) = gcf(2x, 2y)
> f / z is even //even divisor of the common factor
>
> Did I understand you correctly or please correct me if I am wrong.
>
> I appreciate your concern about my matter.
>
> Thanks,
> Regards,
>
>
>
>
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga@wso2.com <os...@gmail.com>
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Wed, Nov 30, 2016 at 12:44 PM, Erik Weathers <ew...@groupon.com>
> wrote:
>
>> Can you explain more about why you are trying this custom scheduler?
>> (I'm afraid you're largely on your own if you *must* continue down that
>> route).
>> In your initial email you just said:
>>
>> > I need to run "input bolt" and "output bolt" on the same node in order
>> to calculate the latency properly
>>
>> I do not understand why that requirement led you to conclude that you
>> must use a custom scheduler. If you align everything appropriately in
>> terms of executors and workers and hosts, and if you're using
>> localOrShuffleGrouping, then you can ensure that all traffic will stay
>> local to the host. By "appropriate alignment", I just mean that the number
>> of spouts and bolts should be even multiples, and the number of worker
>> processes for the topology should also be an even divisor of the common
>> factor for the spouts & bolts. e.g., 5 worker processes on 5 hosts with
>> 10 spouts and 20 bolts.
>>
>> Also when you say "input bolt", do you actually mean *spout*?
>>
>> - Erik
>>
>>
>> On Tue, Nov 29, 2016 at 10:54 PM, Eranga Heshan <er...@gmail.com>
>> wrote:
>>
>>> Can we have some help to solve this issue?
>>>
>>>
>>> Eranga Heshan
>>> *Undergraduate*
>>> Computer Science & Engineering
>>> University of Moratuwa
>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>> Email: eranga@wso2.com <os...@gmail.com>
>>> <https://www.facebook.com/erangaheshan>
>>> <https://twitter.com/erangaheshan>
>>> <https://www.linkedin.com/in/erangaheshan>
>>>
>>> On Wed, Nov 30, 2016 at 9:30 AM, Eranga Heshan <er...@gmail.com>
>>> wrote:
>>>
>>>> Hi Ohad,
>>>>
>>>> Does your scheduler works with storm-0.9.x? I just made few changes and
>>>> tested it. Still it won't work.
>>>>
>>>>
>>>> Eranga Heshan
>>>> *Undergraduate*
>>>> Computer Science & Engineering
>>>> University of Moratuwa
>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>> <https://www.facebook.com/erangaheshan>
>>>> <https://twitter.com/erangaheshan>
>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>
>>>> On Tue, Nov 29, 2016 at 5:46 PM, Ohad Edelstein <oh...@mintigo.com>
>>>> wrote:
>>>>
>>>>> I have opened this stack overflow question,
>>>>> http://stackoverflow.com/questions/40294245/deploy-bolts-spo
>>>>> ut-to-a-specific-supervisor
>>>>> Follow it and add question there,
>>>>> And lets share information if any progress is made
>>>>>
>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>> Date: Tuesday, 29 November 2016 at 13:49
>>>>> To: Ohad Edelstein <oh...@mintigo.com>, "user@storm.apache.org" <
>>>>> user@storm.apache.org>
>>>>>
>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>
>>>>> So we both used http://www.exogeni.net/2015/04
>>>>> /enabling-site-aware-scheduling-for-apache-storm-in-exogeni/
>>>>> tutorial. Then I guess we are facing the problem due to the version.
>>>>>
>>>>> I think, we should try this our own until help arrives and if any of
>>>>> us figured it out, let us inform each other.
>>>>>
>>>>> Cheers,
>>>>> Regards,
>>>>>
>>>>>
>>>>> Eranga Heshan
>>>>> *Undergraduate*
>>>>> Computer Science & Engineering
>>>>> University of Moratuwa
>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>> <https://www.facebook.com/erangaheshan>
>>>>> <https://twitter.com/erangaheshan>
>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>
>>>>> On Tue, Nov 29, 2016 at 5:07 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>> wrote:
>>>>>
>>>>>> I used this tutorial:
>>>>>>
>>>>>> http://www.exogeni.net/2015/04/enabling-site-aware-schedulin
>>>>>> g-for-apache-storm-in-exogeni/
>>>>>>
>>>>>>
>>>>>> https://github.com/xumingming/storm-lib/blob/master/src/jvm/
>>>>>> storm/DemoScheduler.java
>>>>>>
>>>>>> But its for storm 0.9.3
>>>>>>
>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>> Date: Tuesday, 29 November 2016 at 13:13
>>>>>> To: Ohad Edelstein <oh...@mintigo.com>
>>>>>>
>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>
>>>>>> Hi Ohad,
>>>>>>
>>>>>> Yeah, it works when I remove that meta data part from the storm.yaml.
>>>>>> I thought about that also. Could you please set "supervisor.scheduler.meta"
>>>>>> to "supervisor.meta" and see it works. However, it did not work for me. But
>>>>>> in someplace I saw it like that.
>>>>>>
>>>>>> I am working with storm-1.0.2.
>>>>>>
>>>>>> Let us both win at once 😉
>>>>>>
>>>>>> Cheers,
>>>>>> Regard,
>>>>>>
>>>>>>
>>>>>> Eranga Heshan
>>>>>> *Undergraduate*
>>>>>> Computer Science & Engineering
>>>>>> University of Moratuwa
>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>> <https://twitter.com/erangaheshan>
>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>
>>>>>> On Tue, Nov 29, 2016 at 4:37 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I guess you added the meta data part to the storm.yaml in the
>>>>>>> supervisors…
>>>>>>> I think that when ever the code has a problem with one of the params
>>>>>>> you add there, it cause the workers not to start
>>>>>>> When you remove the meta data the problem will go away.
>>>>>>> My guess is that I am not writing the meta part correctly…
>>>>>>> When I set a different parameter that was suppose to be array to a
>>>>>>> simple string, it did the same thing…
>>>>>>>
>>>>>>> With what version of storm are you working with? And what
>>>>>>> instructions are you following?
>>>>>>> Thanks
>>>>>>>
>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>> Date: Tuesday, 29 November 2016 at 12:55
>>>>>>>
>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>
>>>>>>> Hi Ohad,
>>>>>>>
>>>>>>> My supervisors are running. But inside them, the workers are not
>>>>>>> working. That is what I think the issue is. Do you have the same issue?
>>>>>>>
>>>>>>>
>>>>>>> Eranga Heshan
>>>>>>> *Undergraduate*
>>>>>>> Computer Science & Engineering
>>>>>>> University of Moratuwa
>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>
>>>>>>> On Tue, Nov 29, 2016 at 3:23 PM, Ohad Edelstein <oh...@mintigo.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, I am having the same problem on storm 1.0.1
>>>>>>>> I would love if you can share the information again,
>>>>>>>> I also had the same problem that Eranga is having.
>>>>>>>> The problem he has is that the supervisor can’t start because of
>>>>>>>> configuration problem…
>>>>>>>>
>>>>>>>> From: Eranga Heshan <er...@gmail.com>
>>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>> Date: Tuesday, 29 November 2016 at 11:00
>>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>>> Subject: Re: Schedule a Bolt/Spout to a Predefined Node
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Thanks for the info. I implemented a custom scheduler according to
>>>>>>>> the link you suggested. I changed "getAvailableWorkerSlots" in the original
>>>>>>>> code to "getAvailableSlots". However when I submit my topology it shows on
>>>>>>>> the Storm UI. But when I click on it and look for more information I see
>>>>>>>> nothing. All the tables are emplty.
>>>>>>>>
>>>>>>>> I am here attaching the inside view of my topology. Could you
>>>>>>>> please look into that and say what I might have done wrong?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> ​
>>>>>>>>
>>>>>>>>
>>>>>>>> Eranga Heshan
>>>>>>>> *Undergraduate*
>>>>>>>> Computer Science & Engineering
>>>>>>>> University of Moratuwa
>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>
>>>>>>>> On Mon, Nov 28, 2016 at 9:54 AM, anshu shukla <
>>>>>>>> anshushukla0@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> you have to use custom site aware scheduler for that . Or use
>>>>>>>>> ntp to sync clocks for all VMs .
>>>>>>>>>
>>>>>>>>> On Mon, Nov 28, 2016 at 9:49 AM, Eranga Heshan <
>>>>>>>>> eranga.h.n@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I have a cluster of four nodes. There is a benchmark I need to
>>>>>>>>>> run on cluster. Hovever I need to run "input bolt" and "output bolt" on the
>>>>>>>>>> same node in order to calculate the latency properly.
>>>>>>>>>>
>>>>>>>>>> Is there a way that I could run both the bols on the same node?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Eranga Heshan
>>>>>>>>>> *Undergraduate*
>>>>>>>>>> Computer Science & Engineering
>>>>>>>>>> University of Moratuwa
>>>>>>>>>> Mobile: +94 71 138 2686 <%2B94%2071%20552%202087>
>>>>>>>>>> Email: eranga@wso2.com <os...@gmail.com>
>>>>>>>>>> <https://www.facebook.com/erangaheshan>
>>>>>>>>>> <https://twitter.com/erangaheshan>
>>>>>>>>>> <https://www.linkedin.com/in/erangaheshan>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks & Regards,
>>>>>>>>> Anshu Shukla
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>