You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Robert Turner <ro...@bigfoot.com> on 2014/02/24 23:09:02 UTC
Trident localOrShuffle
This appears to be missing from Stream, is there any way to specify this or
is there a reason why this is not possible with Trident.
--
Cheers
Rob.
Re: Trident localOrShuffle
Posted by Robert Turner <ro...@bigfoot.com>.
I have created JIRA STORM-245 - Implement localOrShuffle() on Stream for
performance <https://issues.apache.org/jira/browse/STORM-245>.
Thanks you all for your help.
On 24 February 2014 23:15, P. Taylor Goetz <pt...@gmail.com> wrote:
> Hi Rob,
>
> You are correct about needing a repartitioning operation in order for the
> spout to have a different parallelism.
>
> Can you file a JIRA to implement localOrShuffle()?
>
> Taylor
>
> On Feb 24, 2014, at 5:31 PM, Robert Turner <ro...@bigfoot.com> wrote:
>
> Thanks Tom for the quick reply.
>
> I need a single spout, based on IRichSpout, and parallelism 4 for my
> Trident functions per worker. If I do not do a shuffle the spout
> parallelism is set to 4 as well. It seems that I need the shuffle to have
> separate parallelism for the spout and functions. I would like to avoid
> serialization so localOrShuffle would be ideal, but not apparently an
> option in Trident.
>
>
> On 24 February 2014 22:17, Tom Brown <to...@gmail.com> wrote:
>
>> On trident, all operations happen locally unless you explicitly
>> repartition (shuffle, partition by key, etc)
>>
>>
>> On Monday, February 24, 2014, Robert Turner <ro...@bigfoot.com> wrote:
>>
>>> This appears to be missing from Stream, is there any way to specify this
>>> or is there a reason why this is not possible with Trident.
>>>
>>> --
>>> Cheers
>>> Rob.
>>>
>>
>
>
> --
> Cheers
> Rob.
>
>
--
Cheers
Rob.
Re: Trident localOrShuffle
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Rob,
You are correct about needing a repartitioning operation in order for the spout to have a different parallelism.
Can you file a JIRA to implement localOrShuffle()?
Taylor
> On Feb 24, 2014, at 5:31 PM, Robert Turner <ro...@bigfoot.com> wrote:
>
> Thanks Tom for the quick reply.
>
> I need a single spout, based on IRichSpout, and parallelism 4 for my Trident functions per worker. If I do not do a shuffle the spout parallelism is set to 4 as well. It seems that I need the shuffle to have separate parallelism for the spout and functions. I would like to avoid serialization so localOrShuffle would be ideal, but not apparently an option in Trident.
>
>
>> On 24 February 2014 22:17, Tom Brown <to...@gmail.com> wrote:
>> On trident, all operations happen locally unless you explicitly repartition (shuffle, partition by key, etc)
>>
>>
>> On Monday, February 24, 2014, Robert Turner <ro...@bigfoot.com> wrote:
>>> This appears to be missing from Stream, is there any way to specify this or is there a reason why this is not possible with Trident.
>>>
>>> --
>>> Cheers
>>> Rob.
>
>
>
> --
> Cheers
> Rob.
Re: Trident localOrShuffle
Posted by Robert Turner <ro...@bigfoot.com>.
Thanks Tom for the quick reply.
I need a single spout, based on IRichSpout, and parallelism 4 for my
Trident functions per worker. If I do not do a shuffle the spout
parallelism is set to 4 as well. It seems that I need the shuffle to have
separate parallelism for the spout and functions. I would like to avoid
serialization so localOrShuffle would be ideal, but not apparently an
option in Trident.
On 24 February 2014 22:17, Tom Brown <to...@gmail.com> wrote:
> On trident, all operations happen locally unless you explicitly
> repartition (shuffle, partition by key, etc)
>
>
> On Monday, February 24, 2014, Robert Turner <ro...@bigfoot.com> wrote:
>
>> This appears to be missing from Stream, is there any way to specify this
>> or is there a reason why this is not possible with Trident.
>>
>> --
>> Cheers
>> Rob.
>>
>
--
Cheers
Rob.
Re: Trident localOrShuffle
Posted by Tom Brown <to...@gmail.com>.
On trident, all operations happen locally unless you explicitly repartition
(shuffle, partition by key, etc)
On Monday, February 24, 2014, Robert Turner <ro...@bigfoot.com> wrote:
> This appears to be missing from Stream, is there any way to specify this
> or is there a reason why this is not possible with Trident.
>
> --
> Cheers
> Rob.
>