You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Lydia Ickler <ic...@googlemail.com> on 2014/05/06 12:57:11 UTC
question
Hi there,
my name is Lydia and i am writing a program for my course „Parallel Programming and Distributed Computing“.
for example if i have a spout like this:
builder.setSpout("word-reader",new ApiStreamingSpout(),2);
Within the spout: Is there a way to address each worker separately?
Like to tell one to read in all files that are even numbered and the other one to take the odd numbered…?
Thanks in advance!
Best regards,
Lydia
Re: question
Posted by Bobby Evans <ev...@yahoo-inc.com>.
The spouts also have a number you can access through
TopologyContext.getThisTaskIndex().
The DRPCSpout code does this and can be a good example.
https://git.corp.yahoo.com/storm/storm/blob/master-security/storm-core/src/
jvm/backtype/storm/drpc/DRPCSpout.java#L118-119
Bobby
On 5/7/14, 2:28 AM, "padma priya chitturi" <pa...@gmail.com> wrote:
>Hi,
>
> This could be the way of specifying spouts:
>
>builder.setSpout("word-reader1",new ApiStreamingSpout1(),1);
>builder.setSpout("word-reader2",new ApiStreamingSpout2(),1);
>
>So, define word-reader-1 to read tuples from even numbered files.
>And for word-reader-2 to read from odd numbered files.
>
>
>
>
>On Wed, May 7, 2014 at 12:41 PM, padma priya chitturi <
>padmapriya30@gmail.com> wrote:
>
>> Hi,
>>
>> Specifying the spout as above won't let you write separate piece of code
>> as the executors/spout tasks would be executing same nextTuple method.
>> Instead, to have intended behavior i.e to let one spout task read from
>>even
>> numbered files and other read from odd numbered files, try specifying
>>the
>> spouts as below:
>>
>> builder.setSpout("word-reader1",new ApiStreamingSpout(),1);
>> builder.setSpout("word-reader2",new ApiStreamingSpout(),1);
>>
>> Now, for word-reader-1, define nextTuple in such a way that it reads
>>from
>> even numbered files.
>> And for word-reader-2, define nextTuple to read from odd numbered files.
>>
>> I guess the above way of defining the spouts would work.
>>
>>
>> On Tue, May 6, 2014 at 4:27 PM, Lydia Ickler
>><ic...@googlemail.com>wrote:
>>
>>> Hi there,
>>> my name is Lydia and i am writing a program for my course „Parallel
>>> Programming and Distributed Computing“.
>>>
>>> for example if i have a spout like this:
>>> builder.setSpout("word-reader",new ApiStreamingSpout(),2);
>>>
>>> Within the spout: Is there a way to address each worker separately?
>>> Like to tell one to read in all files that are even numbered and the
>>> other one to take the odd numbered…?
>>> Thanks in advance!
>>>
>>> Best regards,
>>> Lydia
>>>
>>>
>>>
>>>
>>
Re: question
Posted by padma priya chitturi <pa...@gmail.com>.
Hi,
This could be the way of specifying spouts:
builder.setSpout("word-reader1",new ApiStreamingSpout1(),1);
builder.setSpout("word-reader2",new ApiStreamingSpout2(),1);
So, define word-reader-1 to read tuples from even numbered files.
And for word-reader-2 to read from odd numbered files.
On Wed, May 7, 2014 at 12:41 PM, padma priya chitturi <
padmapriya30@gmail.com> wrote:
> Hi,
>
> Specifying the spout as above won't let you write separate piece of code
> as the executors/spout tasks would be executing same nextTuple method.
> Instead, to have intended behavior i.e to let one spout task read from even
> numbered files and other read from odd numbered files, try specifying the
> spouts as below:
>
> builder.setSpout("word-reader1",new ApiStreamingSpout(),1);
> builder.setSpout("word-reader2",new ApiStreamingSpout(),1);
>
> Now, for word-reader-1, define nextTuple in such a way that it reads from
> even numbered files.
> And for word-reader-2, define nextTuple to read from odd numbered files.
>
> I guess the above way of defining the spouts would work.
>
>
> On Tue, May 6, 2014 at 4:27 PM, Lydia Ickler <ic...@googlemail.com>wrote:
>
>> Hi there,
>> my name is Lydia and i am writing a program for my course „Parallel
>> Programming and Distributed Computing“.
>>
>> for example if i have a spout like this:
>> builder.setSpout("word-reader",new ApiStreamingSpout(),2);
>>
>> Within the spout: Is there a way to address each worker separately?
>> Like to tell one to read in all files that are even numbered and the
>> other one to take the odd numbered…?
>> Thanks in advance!
>>
>> Best regards,
>> Lydia
>>
>>
>>
>>
>
Re: question
Posted by padma priya chitturi <pa...@gmail.com>.
Hi,
Sorry, this could be the way of specifying the spouts:
On Wed, May 7, 2014 at 12:41 PM, padma priya chitturi <
padmapriya30@gmail.com> wrote:
> Hi,
>
> Specifying the spout as above won't let you write separate piece of code
> as the executors/spout tasks would be executing same nextTuple method.
> Instead, to have intended behavior i.e to let one spout task read from even
> numbered files and other read from odd numbered files, try specifying the
> spouts as below:
>
> builder.setSpout("word-reader1",new ApiStreamingSpout(),1);
> builder.setSpout("word-reader2",new ApiStreamingSpout(),1);
>
> Now, for word-reader-1, define nextTuple in such a way that it reads from
> even numbered files.
> And for word-reader-2, define nextTuple to read from odd numbered files.
>
> I guess the above way of defining the spouts would work.
>
>
> On Tue, May 6, 2014 at 4:27 PM, Lydia Ickler <ic...@googlemail.com>wrote:
>
>> Hi there,
>> my name is Lydia and i am writing a program for my course „Parallel
>> Programming and Distributed Computing“.
>>
>> for example if i have a spout like this:
>> builder.setSpout("word-reader",new ApiStreamingSpout(),2);
>>
>> Within the spout: Is there a way to address each worker separately?
>> Like to tell one to read in all files that are even numbered and the
>> other one to take the odd numbered…?
>> Thanks in advance!
>>
>> Best regards,
>> Lydia
>>
>>
>>
>>
>
Re: question
Posted by padma priya chitturi <pa...@gmail.com>.
Hi,
Specifying the spout as above won't let you write separate piece of code as
the executors/spout tasks would be executing same nextTuple method.
Instead, to have intended behavior i.e to let one spout task read from even
numbered files and other read from odd numbered files, try specifying the
spouts as below:
builder.setSpout("word-reader1",new ApiStreamingSpout(),1);
builder.setSpout("word-reader2",new ApiStreamingSpout(),1);
Now, for word-reader-1, define nextTuple in such a way that it reads from
even numbered files.
And for word-reader-2, define nextTuple to read from odd numbered files.
I guess the above way of defining the spouts would work.
On Tue, May 6, 2014 at 4:27 PM, Lydia Ickler <ic...@googlemail.com>wrote:
> Hi there,
> my name is Lydia and i am writing a program for my course „Parallel
> Programming and Distributed Computing“.
>
> for example if i have a spout like this:
> builder.setSpout("word-reader",new ApiStreamingSpout(),2);
>
> Within the spout: Is there a way to address each worker separately?
> Like to tell one to read in all files that are even numbered and the other
> one to take the odd numbered…?
> Thanks in advance!
>
> Best regards,
> Lydia
>
>
>
>
Re: question
Posted by John Gilmore <jo...@gmail.com>.
Hi Lydia,
I think your question is more suited to the Storm user mailing list: user@storm.incubator.apache.org
Regards
John
On 06 May 2014, at 12:57 PM, Lydia Ickler <ic...@googlemail.com> wrote:
> Hi there,
> my name is Lydia and i am writing a program for my course „Parallel Programming and Distributed Computing“.
>
> for example if i have a spout like this:
> builder.setSpout("word-reader",new ApiStreamingSpout(),2);
>
> Within the spout: Is there a way to address each worker separately?
> Like to tell one to read in all files that are even numbered and the other one to take the odd numbered…?
> Thanks in advance!
>
> Best regards,
> Lydia
>
>
>