STORM-513

[William Oberman <ob...@civicscience.com>]

I was reading the release notes for 0.9.3, and STORM-513 concerns me.  I'm
using php, which being single threaded will have challenges to respond in a
timely manner to a sync command.  I just checked,
and supervisor.worker.timeout.secs defaults to 30 seconds, which I'll have
to sadly increase (globally, unless I use an isolation scheduler?  ugh...)
as I have an oddball legacy php-based bolt that can take minutes to process
a Tuple.

I just looked over the pull request.  While this doesn't solve my problem,
is there any reason to not call setHeartbeat() for all responses (ack,
fail, emit, etc...) from the managed subprocess in ShellBolt?  I mean, if
the managed subprocess sends *anything* back, that seems as good as a sync
to me.  In fact, why bother adding a new protocol command (sync) that broke
all existing multilang implementations when you could have piggybacked on
all of the other messages already coming from the managed subprocess?  You
basically did that for ShellSpout...

I hate to "throw stones", and I'm sure I'm just missing something not being
as deep into the code on the Storm-side.

Thanks!

will

Re: STORM-513

[William Oberman <ob...@civicscience.com>]

That makes total sense.  Even though I have some low/lower volume
topologies, I never considered a completely empty (if only for a period of
time) stream!

I'll try to figure out how to cope with the new system.

On Wed, Nov 26, 2014 at 5:04 PM, 임정택 <ka...@gmail.com> wrote:

> Hello!
>
> It's due to differences between spout and bolt.
> In spout, nextTuple is automatically called from Storm, so depending on
> your configuration but nextTuple in subprocess would be called periodically.
> That's great for heartbeat.
>
> But bolt is based on event, bolt never calls subprocess when there're no
> tuples to process. It means idle bolt would kill itself.
> It's terrible, so I have to introduce multilang protocol's changed.
>
> There's other discussions, why not letting subprocess send heartbeat
> periodically by itself.
> Surely it's better than current, but it would be not easy because of
> "multilang", and AFAIK PHP doesn't have real thread so PHP is an example.
>
> Hope this helps.
>
> Regards.
> Jungtaek Lim (HeartSaVioR)
>
> On 2014년 11월 27일 (목) at 오전 12:35 William Oberman <ob...@civicscience.com>
> wrote:
>
>> I was reading the release notes for 0.9.3, and STORM-513 concerns me.
>> I'm using php, which being single threaded will have challenges to respond
>> in a timely manner to a sync command.  I just checked,
>> and supervisor.worker.timeout.secs defaults to 30 seconds, which I'll have
>> to sadly increase (globally, unless I use an isolation scheduler?  ugh...)
>> as I have an oddball legacy php-based bolt that can take minutes to process
>> a Tuple.
>>
>> I just looked over the pull request.  While this doesn't solve my
>> problem, is there any reason to not call setHeartbeat() for all responses
>> (ack, fail, emit, etc...) from the managed subprocess in ShellBolt?  I
>> mean, if the managed subprocess sends *anything* back, that seems as good
>> as a sync to me.  In fact, why bother adding a new protocol command (sync)
>> that broke all existing multilang implementations when you could have
>> piggybacked on all of the other messages already coming from the managed
>> subprocess?  You basically did that for ShellSpout...
>>
>> I hate to "throw stones", and I'm sure I'm just missing something not
>> being as deep into the code on the Storm-side.
>>
>> Thanks!
>>
>> will
>>
>

Re: STORM-513

[임정택 <ka...@gmail.com>]

Hello!

It's due to differences between spout and bolt.
In spout, nextTuple is automatically called from Storm, so depending on
your configuration but nextTuple in subprocess would be called periodically.
That's great for heartbeat.

But bolt is based on event, bolt never calls subprocess when there're no
tuples to process. It means idle bolt would kill itself.
It's terrible, so I have to introduce multilang protocol's changed.

There's other discussions, why not letting subprocess send heartbeat
periodically by itself.
Surely it's better than current, but it would be not easy because of
"multilang", and AFAIK PHP doesn't have real thread so PHP is an example.

Hope this helps.

Regards.
Jungtaek Lim (HeartSaVioR)
On 2014년 11월 27일 (목) at 오전 12:35 William Oberman <ob...@civicscience.com>
wrote:

> I was reading the release notes for 0.9.3, and STORM-513 concerns me.  I'm
> using php, which being single threaded will have challenges to respond in a
> timely manner to a sync command.  I just checked,
> and supervisor.worker.timeout.secs defaults to 30 seconds, which I'll have
> to sadly increase (globally, unless I use an isolation scheduler?  ugh...)
> as I have an oddball legacy php-based bolt that can take minutes to process
> a Tuple.
>
> I just looked over the pull request.  While this doesn't solve my problem,
> is there any reason to not call setHeartbeat() for all responses (ack,
> fail, emit, etc...) from the managed subprocess in ShellBolt?  I mean, if
> the managed subprocess sends *anything* back, that seems as good as a sync
> to me.  In fact, why bother adding a new protocol command (sync) that broke
> all existing multilang implementations when you could have piggybacked on
> all of the other messages already coming from the managed subprocess?  You
> basically did that for ShellSpout...
>
> I hate to "throw stones", and I'm sure I'm just missing something not
> being as deep into the code on the Storm-side.
>
> Thanks!
>
> will
>