You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Adam Mitchell <ad...@salesforce.com> on 2015/05/12 19:12:39 UTC
Can Spout.nextTuple() emit more than one tuple?
If I've got a bunch of data in memory, ready to emit, can nextTuple() go
ahead and call SpoutOutputCollector.emit() many times?
Or is it best to emit() once per call to nextTuple()?
Re: Can Spout.nextTuple() emit more than one tuple?
Posted by Adam Mitchell <ad...@salesforce.com>.
Makes sense. Thanks for the feedback.
On Tue, May 12, 2015 at 4:18 PM, Mark Tomko <mt...@broadinstitute.org>
wrote:
> Agreed. If you're using MAX_SPOUT_PENDING, it's better to emit 0 or 1
> tuples and let Storm call for the next tuple when it is ready. Probably
> best to to 'flatten' your streams upstream of nextTuple so that it can
> simple pull one unit of work off the queue and emit it.
>
> On Tue, May 12, 2015 at 3:46 PM, Jeffery Maass <ma...@gmail.com> wrote:
>
>> I don't know how MAX_SPOUT_PENDING works, but, I bet it works by
>> throttling calls to nextTuple(). If that is the case, calling
>> SpoutOutputCollector.emit() multiple times within one call to nextTuple()
>> would tend to over ride that throttle.
>>
>> Thank you for your time!
>>
>> +++++++++++++++++++++
>> Jeff Maass <ma...@gmail.com>
>> linkedin.com/in/jeffmaass
>> stackoverflow.com/users/373418/maassql
>> +++++++++++++++++++++
>>
>>
>> On Tue, May 12, 2015 at 12:30 PM, Nathan Leung <nc...@gmail.com> wrote:
>>
>>> Yes. It's best not to loop for a very long time, but you can call it
>>> multiple times.
>>> On May 12, 2015 1:12 PM, "Adam Mitchell" <ad...@salesforce.com>
>>> wrote:
>>>
>>>> If I've got a bunch of data in memory, ready to emit, can nextTuple()
>>>> go ahead and call SpoutOutputCollector.emit() many times?
>>>>
>>>> Or is it best to emit() once per call to nextTuple()?
>>>>
>>>>
>>>>
>>
>
Re: Can Spout.nextTuple() emit more than one tuple?
Posted by Mark Tomko <mt...@broadinstitute.org>.
Agreed. If you're using MAX_SPOUT_PENDING, it's better to emit 0 or 1
tuples and let Storm call for the next tuple when it is ready. Probably
best to to 'flatten' your streams upstream of nextTuple so that it can
simple pull one unit of work off the queue and emit it.
On Tue, May 12, 2015 at 3:46 PM, Jeffery Maass <ma...@gmail.com> wrote:
> I don't know how MAX_SPOUT_PENDING works, but, I bet it works by
> throttling calls to nextTuple(). If that is the case, calling
> SpoutOutputCollector.emit() multiple times within one call to nextTuple()
> would tend to over ride that throttle.
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
> On Tue, May 12, 2015 at 12:30 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> Yes. It's best not to loop for a very long time, but you can call it
>> multiple times.
>> On May 12, 2015 1:12 PM, "Adam Mitchell" <ad...@salesforce.com>
>> wrote:
>>
>>> If I've got a bunch of data in memory, ready to emit, can nextTuple() go
>>> ahead and call SpoutOutputCollector.emit() many times?
>>>
>>> Or is it best to emit() once per call to nextTuple()?
>>>
>>>
>>>
>
Re: Can Spout.nextTuple() emit more than one tuple?
Posted by Jeffery Maass <ma...@gmail.com>.
I don't know how MAX_SPOUT_PENDING works, but, I bet it works by throttling
calls to nextTuple(). If that is the case, calling
SpoutOutputCollector.emit() multiple times within one call to nextTuple()
would tend to over ride that throttle.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Tue, May 12, 2015 at 12:30 PM, Nathan Leung <nc...@gmail.com> wrote:
> Yes. It's best not to loop for a very long time, but you can call it
> multiple times.
> On May 12, 2015 1:12 PM, "Adam Mitchell" <ad...@salesforce.com>
> wrote:
>
>> If I've got a bunch of data in memory, ready to emit, can nextTuple() go
>> ahead and call SpoutOutputCollector.emit() many times?
>>
>> Or is it best to emit() once per call to nextTuple()?
>>
>>
>>
Re: Can Spout.nextTuple() emit more than one tuple?
Posted by Nathan Leung <nc...@gmail.com>.
Yes. It's best not to loop for a very long time, but you can call it
multiple times.
On May 12, 2015 1:12 PM, "Adam Mitchell" <ad...@salesforce.com>
wrote:
> If I've got a bunch of data in memory, ready to emit, can nextTuple() go
> ahead and call SpoutOutputCollector.emit() many times?
>
> Or is it best to emit() once per call to nextTuple()?
>
>
>