Spout activate/deactivate

[Javier Gonzalez <ja...@gmail.com>]

Hi,

I have a use case where I would need to stop a spout from emitting for a
period of time. I'm looking at the activate /deactivate methods, but
there's not much information apart from the API and the java base classes
have empty implementations. Can anybody shed any insight on how those work?

Thanks,
Javier
(storm 0.9.4 btw)

Re: Spout activate/deactivate

[Javier Gonzalez <ja...@gmail.com>]

I have a single spout instance so that's not a problem. I planned to do
this control from another spout listening on a "control" channel, so to
speak. If I block on nextTuple I will not get acks from the previous
messages to that spout.

Now that you mention it, if I don't have the two spouts in the same worker
it won't work. Using an external store like zk sounds like a workable
solution. Thanks!

Regards,
Jg
On Aug 14, 2015 3:14 PM, "Kishore Senji" <ks...@gmail.com> wrote:

> That only works if you want to control a bolt. You cannot control a Spout
> from another spout, that spout being part of the topology. What you can do
> is that, you have to embed the Kafka consumer to look up the messages from
> the control topic and block nextTuple. You also need to store offsets and
> make sure it works for every instance of your main spout. This is the
> reason I think Zk solution is better.
> On Fri, Aug 14, 2015 at 12:05 PM Javier Gonzalez <ja...@gmail.com>
> wrote:
>
>> I was thinking of using another spout as "control channel", and from that
>> spout manipulate the original spout to cause the nextTuple method to not
>> call the next message from the incoming queue (but not block so that acks
>> can be processed). A second signal on that spout can manipulate it back to
>> BAU.
>>
>> Thanks,
>> Javier
>> On Aug 14, 2015 2:59 PM, "Kishore Senji" <ks...@gmail.com> wrote:
>>
>>> This is exactly what is done on deactivate of topology. Storm has rest
>>> api to deactivate and you can use that to integrate with your external
>>> system. Otherwise you can extend your spout and use Zk to signal it. On
>>> receiving signal it can block and unblock appropriately the nextTuple() call
>>> On Fri, Aug 14, 2015 at 5:34 AM Javier Gonzalez <ja...@gmail.com>
>>> wrote:
>>>
>>>> I'm trying to ensure everything is processed for coordination with an
>>>> external system. Therefore, on a given signal, I have to stop the spout,
>>>> wait for the acks of all pending/inflight messages, then resume the spout
>>>> intake on a second signal.
>>>> On Aug 13, 2015 10:01 PM, "Kishore Senji" <ks...@gmail.com> wrote:
>>>>
>>>>> The deactivate() on the spout will be called when the topology is
>>>>> deactivated. Below is the output of storm deactivate help. This will
>>>>> deactivate the whole topology and this can be from the Storm UI as well.
>>>>> Does you topology have more than one Spout and are you looking to stop only
>>>>> a subset of those Spouts?
>>>>>
>>>>> $ storm help deactivate
>>>>> Syntax: [storm deactivate topology-name]
>>>>>
>>>>>     Deactivates the specified topology's spouts.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <ja...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On a more broader term, can you share the strategies you've used to
>>>>>> pause (not emit anything else into the topology and not read anything else
>>>>>> from the data source) a topology's spouts?
>>>>>>
>>>>>> Thanks,
>>>>>> Javier
>>>>>> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <ja...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a use case where I would need to stop a spout from emitting
>>>>>>> for a period of time. I'm looking at the activate /deactivate methods, but
>>>>>>> there's not much information apart from the API and the java base classes
>>>>>>> have empty implementations. Can anybody shed any insight on how those work?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Javier
>>>>>>> (storm 0.9.4 btw)
>>>>>>>
>>>>>>

Re: Spout activate/deactivate

[Kishore Senji <ks...@gmail.com>]

That only works if you want to control a bolt. You cannot control a Spout
from another spout, that spout being part of the topology. What you can do
is that, you have to embed the Kafka consumer to look up the messages from
the control topic and block nextTuple. You also need to store offsets and
make sure it works for every instance of your main spout. This is the
reason I think Zk solution is better.
On Fri, Aug 14, 2015 at 12:05 PM Javier Gonzalez <ja...@gmail.com> wrote:

> I was thinking of using another spout as "control channel", and from that
> spout manipulate the original spout to cause the nextTuple method to not
> call the next message from the incoming queue (but not block so that acks
> can be processed). A second signal on that spout can manipulate it back to
> BAU.
>
> Thanks,
> Javier
> On Aug 14, 2015 2:59 PM, "Kishore Senji" <ks...@gmail.com> wrote:
>
>> This is exactly what is done on deactivate of topology. Storm has rest
>> api to deactivate and you can use that to integrate with your external
>> system. Otherwise you can extend your spout and use Zk to signal it. On
>> receiving signal it can block and unblock appropriately the nextTuple() call
>> On Fri, Aug 14, 2015 at 5:34 AM Javier Gonzalez <ja...@gmail.com>
>> wrote:
>>
>>> I'm trying to ensure everything is processed for coordination with an
>>> external system. Therefore, on a given signal, I have to stop the spout,
>>> wait for the acks of all pending/inflight messages, then resume the spout
>>> intake on a second signal.
>>> On Aug 13, 2015 10:01 PM, "Kishore Senji" <ks...@gmail.com> wrote:
>>>
>>>> The deactivate() on the spout will be called when the topology is
>>>> deactivated. Below is the output of storm deactivate help. This will
>>>> deactivate the whole topology and this can be from the Storm UI as well.
>>>> Does you topology have more than one Spout and are you looking to stop only
>>>> a subset of those Spouts?
>>>>
>>>> $ storm help deactivate
>>>> Syntax: [storm deactivate topology-name]
>>>>
>>>>     Deactivates the specified topology's spouts.
>>>>
>>>>
>>>>
>>>> On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <ja...@gmail.com>
>>>> wrote:
>>>>
>>>>> On a more broader term, can you share the strategies you've used to
>>>>> pause (not emit anything else into the topology and not read anything else
>>>>> from the data source) a topology's spouts?
>>>>>
>>>>> Thanks,
>>>>> Javier
>>>>> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <ja...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a use case where I would need to stop a spout from emitting
>>>>>> for a period of time. I'm looking at the activate /deactivate methods, but
>>>>>> there's not much information apart from the API and the java base classes
>>>>>> have empty implementations. Can anybody shed any insight on how those work?
>>>>>>
>>>>>> Thanks,
>>>>>> Javier
>>>>>> (storm 0.9.4 btw)
>>>>>>
>>>>>

Re: Spout activate/deactivate

[Javier Gonzalez <ja...@gmail.com>]

I was thinking of using another spout as "control channel", and from that
spout manipulate the original spout to cause the nextTuple method to not
call the next message from the incoming queue (but not block so that acks
can be processed). A second signal on that spout can manipulate it back to
BAU.

Thanks,
Javier
On Aug 14, 2015 2:59 PM, "Kishore Senji" <ks...@gmail.com> wrote:

> This is exactly what is done on deactivate of topology. Storm has rest api
> to deactivate and you can use that to integrate with your external system.
> Otherwise you can extend your spout and use Zk to signal it. On receiving
> signal it can block and unblock appropriately the nextTuple() call
> On Fri, Aug 14, 2015 at 5:34 AM Javier Gonzalez <ja...@gmail.com>
> wrote:
>
>> I'm trying to ensure everything is processed for coordination with an
>> external system. Therefore, on a given signal, I have to stop the spout,
>> wait for the acks of all pending/inflight messages, then resume the spout
>> intake on a second signal.
>> On Aug 13, 2015 10:01 PM, "Kishore Senji" <ks...@gmail.com> wrote:
>>
>>> The deactivate() on the spout will be called when the topology is
>>> deactivated. Below is the output of storm deactivate help. This will
>>> deactivate the whole topology and this can be from the Storm UI as well.
>>> Does you topology have more than one Spout and are you looking to stop only
>>> a subset of those Spouts?
>>>
>>> $ storm help deactivate
>>> Syntax: [storm deactivate topology-name]
>>>
>>>     Deactivates the specified topology's spouts.
>>>
>>>
>>>
>>> On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <ja...@gmail.com>
>>> wrote:
>>>
>>>> On a more broader term, can you share the strategies you've used to
>>>> pause (not emit anything else into the topology and not read anything else
>>>> from the data source) a topology's spouts?
>>>>
>>>> Thanks,
>>>> Javier
>>>> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <ja...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a use case where I would need to stop a spout from emitting for
>>>>> a period of time. I'm looking at the activate /deactivate methods, but
>>>>> there's not much information apart from the API and the java base classes
>>>>> have empty implementations. Can anybody shed any insight on how those work?
>>>>>
>>>>> Thanks,
>>>>> Javier
>>>>> (storm 0.9.4 btw)
>>>>>
>>>>

Re: Spout activate/deactivate

[Kishore Senji <ks...@gmail.com>]

This is exactly what is done on deactivate of topology. Storm has rest api
to deactivate and you can use that to integrate with your external system.
Otherwise you can extend your spout and use Zk to signal it. On receiving
signal it can block and unblock appropriately the nextTuple() call
On Fri, Aug 14, 2015 at 5:34 AM Javier Gonzalez <ja...@gmail.com> wrote:

> I'm trying to ensure everything is processed for coordination with an
> external system. Therefore, on a given signal, I have to stop the spout,
> wait for the acks of all pending/inflight messages, then resume the spout
> intake on a second signal.
> On Aug 13, 2015 10:01 PM, "Kishore Senji" <ks...@gmail.com> wrote:
>
>> The deactivate() on the spout will be called when the topology is
>> deactivated. Below is the output of storm deactivate help. This will
>> deactivate the whole topology and this can be from the Storm UI as well.
>> Does you topology have more than one Spout and are you looking to stop only
>> a subset of those Spouts?
>>
>> $ storm help deactivate
>> Syntax: [storm deactivate topology-name]
>>
>>     Deactivates the specified topology's spouts.
>>
>>
>>
>> On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <ja...@gmail.com>
>> wrote:
>>
>>> On a more broader term, can you share the strategies you've used to
>>> pause (not emit anything else into the topology and not read anything else
>>> from the data source) a topology's spouts?
>>>
>>> Thanks,
>>> Javier
>>> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <ja...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a use case where I would need to stop a spout from emitting for
>>>> a period of time. I'm looking at the activate /deactivate methods, but
>>>> there's not much information apart from the API and the java base classes
>>>> have empty implementations. Can anybody shed any insight on how those work?
>>>>
>>>> Thanks,
>>>> Javier
>>>> (storm 0.9.4 btw)
>>>>
>>>

Re: Spout activate/deactivate

[Javier Gonzalez <ja...@gmail.com>]

I'm trying to ensure everything is processed for coordination with an
external system. Therefore, on a given signal, I have to stop the spout,
wait for the acks of all pending/inflight messages, then resume the spout
intake on a second signal.
On Aug 13, 2015 10:01 PM, "Kishore Senji" <ks...@gmail.com> wrote:

> The deactivate() on the spout will be called when the topology is
> deactivated. Below is the output of storm deactivate help. This will
> deactivate the whole topology and this can be from the Storm UI as well.
> Does you topology have more than one Spout and are you looking to stop only
> a subset of those Spouts?
>
> $ storm help deactivate
> Syntax: [storm deactivate topology-name]
>
>     Deactivates the specified topology's spouts.
>
>
>
> On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <ja...@gmail.com>
> wrote:
>
>> On a more broader term, can you share the strategies you've used to pause
>> (not emit anything else into the topology and not read anything else from
>> the data source) a topology's spouts?
>>
>> Thanks,
>> Javier
>> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <ja...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a use case where I would need to stop a spout from emitting for a
>>> period of time. I'm looking at the activate /deactivate methods, but
>>> there's not much information apart from the API and the java base classes
>>> have empty implementations. Can anybody shed any insight on how those work?
>>>
>>> Thanks,
>>> Javier
>>> (storm 0.9.4 btw)
>>>
>>

Re: Spout activate/deactivate

[Kishore Senji <ks...@gmail.com>]

The deactivate() on the spout will be called when the topology is
deactivated. Below is the output of storm deactivate help. This will
deactivate the whole topology and this can be from the Storm UI as well.
Does you topology have more than one Spout and are you looking to stop only
a subset of those Spouts?

$ storm help deactivate
Syntax: [storm deactivate topology-name]

    Deactivates the specified topology's spouts.



On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <ja...@gmail.com> wrote:

> On a more broader term, can you share the strategies you've used to pause
> (not emit anything else into the topology and not read anything else from
> the data source) a topology's spouts?
>
> Thanks,
> Javier
> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <ja...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a use case where I would need to stop a spout from emitting for a
>> period of time. I'm looking at the activate /deactivate methods, but
>> there's not much information apart from the API and the java base classes
>> have empty implementations. Can anybody shed any insight on how those work?
>>
>> Thanks,
>> Javier
>> (storm 0.9.4 btw)
>>
>

Re: Spout activate/deactivate

[Javier Gonzalez <ja...@gmail.com>]

On a more broader term, can you share the strategies you've used to pause
(not emit anything else into the topology and not read anything else from
the data source) a topology's spouts?

Thanks,
Javier
On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <ja...@gmail.com> wrote:

> Hi,
>
> I have a use case where I would need to stop a spout from emitting for a
> period of time. I'm looking at the activate /deactivate methods, but
> there's not much information apart from the API and the java base classes
> have empty implementations. Can anybody shed any insight on how those work?
>
> Thanks,
> Javier
> (storm 0.9.4 btw)
>