You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Nick Katsipoulakis <ni...@gmail.com> on 2014/11/17 22:56:08 UTC
Incrementally add direct groupings to a Bolt
Hello,
I am currently creating an XML topology parser. Each bolt has its input
spout stored in a List, so I want to do something like the following:
...
TopologyParser parser = new TopologyParser();
parser.parseTopology("some-file.xml");
components = parser.getComponents();
Iterator<Component> itr = components.iterator();
builder = new TopologyBuilder();
while(itr.hasNext()) {
Component comp = itr.next();
if(comp.getType().equals("Bolt")) {
BoltDeclarer declarer = builder.setBolt(comp.getName(), new
SampleBolt(comp.getName(), comp.getExecutors());
if(comp.getUpstreamTasks() != null && comp.getUpstreamTasks().size() >
0) {
for(String upstream_task : comp.getUpstreamTasks()) {
declarer.directGrouping(upstream_task);
}
}
}
}
Config conf = new Config();
conf.setDebug(true);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("test", conf, builder.createTopology());
...
So, in the above code the TopologyParser parser parses the topology from a
file and the components variable is a list of all the Component objects
(the TopologyParser and the Component classes
are custom built by me). If a component is for type "Bolt" and has upstream
tasks, they have their output registered as directly grouped to the bolt.
Unfortunately, the above approach does not seem to work.
Any suggestions on how to make something the above work?
Thank you,
Nikos
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Incrementally add direct groupings to a Bolt
Posted by Nick Katsipoulakis <ni...@gmail.com>.
Hi,
I use direct grouping because in my use case scenario, tuples should be
sent to specifc downstream bolts. Yes, I do use emitDirect.
2014-11-18 8:01 GMT-05:00 Kobi Salant <ko...@liveperson.com>:
> Hi Nick,
>
> I see you use "directGrouping", any special reason?
> do you use "emitDirect" as well?
>
> Kobi
>
> On Tue, Nov 18, 2014 at 2:54 PM, Nick Katsipoulakis <nick.katsip@gmail.com
> > wrote:
>
>> Hello Kobi,
>>
>> I have spouts but I have not included them in my code. The problem is
>> that when I submit my topology, no tuples are sent around.
>>
>> Thank you,
>> Nikos
>>
>> 2014-11-18 1:45 GMT-05:00 Kobi Salant <ko...@liveperson.com>:
>>
>>> Hi Nikos,
>>>
>>> What about the spouts? you must have a spout to have data in the
>>> topology.
>>>
>>> What exactly is not working?
>>>
>>> Kobi
>>>
>>> On Mon, Nov 17, 2014 at 11:56 PM, Nick Katsipoulakis <
>>> nick.katsip@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am currently creating an XML topology parser. Each bolt has its input
>>>> spout stored in a List, so I want to do something like the following:
>>>>
>>>> ...
>>>>
>>>> TopologyParser parser = new TopologyParser();
>>>> parser.parseTopology("some-file.xml");
>>>> components = parser.getComponents();
>>>> Iterator<Component> itr = components.iterator();
>>>> builder = new TopologyBuilder();
>>>> while(itr.hasNext()) {
>>>> Component comp = itr.next();
>>>> if(comp.getType().equals("Bolt")) {
>>>> BoltDeclarer declarer = builder.setBolt(comp.getName(), new
>>>> SampleBolt(comp.getName(), comp.getExecutors());
>>>> if(comp.getUpstreamTasks() != null &&
>>>> comp.getUpstreamTasks().size() > 0) {
>>>> for(String upstream_task : comp.getUpstreamTasks()) {
>>>> declarer.directGrouping(upstream_task);
>>>> }
>>>> }
>>>> }
>>>> }
>>>>
>>>> Config conf = new Config();
>>>> conf.setDebug(true);
>>>> LocalCluster cluster = new LocalCluster();
>>>> cluster.submitTopology("test", conf, builder.createTopology());
>>>>
>>>> ...
>>>>
>>>> So, in the above code the TopologyParser parser parses the topology
>>>> from a file and the components variable is a list of all the Component
>>>> objects (the TopologyParser and the Component classes
>>>> are custom built by me). If a component is for type "Bolt" and has
>>>> upstream tasks, they have their output registered as directly grouped to
>>>> the bolt. Unfortunately, the above approach does not seem to work.
>>>> Any suggestions on how to make something the above work?
>>>>
>>>> Thank you,
>>>> Nikos
>>>>
>>>> --
>>>> Nikolaos Romanos Katsipoulakis,
>>>> University of Pittsburgh, PhD candidate
>>>>
>>>
>>>
>>> This message may contain confidential and/or privileged information.
>>> If you are not the addressee or authorized to receive this on behalf of
>>> the addressee you must not use, copy, disclose or take action based on this
>>> message or any information herein.
>>> If you have received this message in error, please advise the sender
>>> immediately by reply email and delete this message. Thank you.
>>>
>>
>>
>>
>> --
>> Nikolaos Romanos Katsipoulakis,
>> University of Pittsburgh, PhD candidate
>>
>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Incrementally add direct groupings to a Bolt
Posted by Kobi Salant <ko...@liveperson.com>.
Hi Nick,
I see you use "directGrouping", any special reason?
do you use "emitDirect" as well?
Kobi
On Tue, Nov 18, 2014 at 2:54 PM, Nick Katsipoulakis <ni...@gmail.com>
wrote:
> Hello Kobi,
>
> I have spouts but I have not included them in my code. The problem is that
> when I submit my topology, no tuples are sent around.
>
> Thank you,
> Nikos
>
> 2014-11-18 1:45 GMT-05:00 Kobi Salant <ko...@liveperson.com>:
>
>> Hi Nikos,
>>
>> What about the spouts? you must have a spout to have data in the topology.
>>
>> What exactly is not working?
>>
>> Kobi
>>
>> On Mon, Nov 17, 2014 at 11:56 PM, Nick Katsipoulakis <
>> nick.katsip@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am currently creating an XML topology parser. Each bolt has its input
>>> spout stored in a List, so I want to do something like the following:
>>>
>>> ...
>>>
>>> TopologyParser parser = new TopologyParser();
>>> parser.parseTopology("some-file.xml");
>>> components = parser.getComponents();
>>> Iterator<Component> itr = components.iterator();
>>> builder = new TopologyBuilder();
>>> while(itr.hasNext()) {
>>> Component comp = itr.next();
>>> if(comp.getType().equals("Bolt")) {
>>> BoltDeclarer declarer = builder.setBolt(comp.getName(), new
>>> SampleBolt(comp.getName(), comp.getExecutors());
>>> if(comp.getUpstreamTasks() != null &&
>>> comp.getUpstreamTasks().size() > 0) {
>>> for(String upstream_task : comp.getUpstreamTasks()) {
>>> declarer.directGrouping(upstream_task);
>>> }
>>> }
>>> }
>>> }
>>>
>>> Config conf = new Config();
>>> conf.setDebug(true);
>>> LocalCluster cluster = new LocalCluster();
>>> cluster.submitTopology("test", conf, builder.createTopology());
>>>
>>> ...
>>>
>>> So, in the above code the TopologyParser parser parses the topology from
>>> a file and the components variable is a list of all the Component objects
>>> (the TopologyParser and the Component classes
>>> are custom built by me). If a component is for type "Bolt" and has
>>> upstream tasks, they have their output registered as directly grouped to
>>> the bolt. Unfortunately, the above approach does not seem to work.
>>> Any suggestions on how to make something the above work?
>>>
>>> Thank you,
>>> Nikos
>>>
>>> --
>>> Nikolaos Romanos Katsipoulakis,
>>> University of Pittsburgh, PhD candidate
>>>
>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
--
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the
addressee you must not use, copy, disclose or take action based on this
message or any information herein.
If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Thank you.
Re: Incrementally add direct groupings to a Bolt
Posted by Nick Katsipoulakis <ni...@gmail.com>.
Hello Kobi,
I have spouts but I have not included them in my code. The problem is that
when I submit my topology, no tuples are sent around.
Thank you,
Nikos
2014-11-18 1:45 GMT-05:00 Kobi Salant <ko...@liveperson.com>:
> Hi Nikos,
>
> What about the spouts? you must have a spout to have data in the topology.
>
> What exactly is not working?
>
> Kobi
>
> On Mon, Nov 17, 2014 at 11:56 PM, Nick Katsipoulakis <
> nick.katsip@gmail.com> wrote:
>
>> Hello,
>>
>> I am currently creating an XML topology parser. Each bolt has its input
>> spout stored in a List, so I want to do something like the following:
>>
>> ...
>>
>> TopologyParser parser = new TopologyParser();
>> parser.parseTopology("some-file.xml");
>> components = parser.getComponents();
>> Iterator<Component> itr = components.iterator();
>> builder = new TopologyBuilder();
>> while(itr.hasNext()) {
>> Component comp = itr.next();
>> if(comp.getType().equals("Bolt")) {
>> BoltDeclarer declarer = builder.setBolt(comp.getName(), new
>> SampleBolt(comp.getName(), comp.getExecutors());
>> if(comp.getUpstreamTasks() != null && comp.getUpstreamTasks().size()
>> > 0) {
>> for(String upstream_task : comp.getUpstreamTasks()) {
>> declarer.directGrouping(upstream_task);
>> }
>> }
>> }
>> }
>>
>> Config conf = new Config();
>> conf.setDebug(true);
>> LocalCluster cluster = new LocalCluster();
>> cluster.submitTopology("test", conf, builder.createTopology());
>>
>> ...
>>
>> So, in the above code the TopologyParser parser parses the topology from
>> a file and the components variable is a list of all the Component objects
>> (the TopologyParser and the Component classes
>> are custom built by me). If a component is for type "Bolt" and has
>> upstream tasks, they have their output registered as directly grouped to
>> the bolt. Unfortunately, the above approach does not seem to work.
>> Any suggestions on how to make something the above work?
>>
>> Thank you,
>> Nikos
>>
>> --
>> Nikolaos Romanos Katsipoulakis,
>> University of Pittsburgh, PhD candidate
>>
>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Incrementally add direct groupings to a Bolt
Posted by Kobi Salant <ko...@liveperson.com>.
Hi Nikos,
What about the spouts? you must have a spout to have data in the topology.
What exactly is not working?
Kobi
On Mon, Nov 17, 2014 at 11:56 PM, Nick Katsipoulakis <ni...@gmail.com>
wrote:
> Hello,
>
> I am currently creating an XML topology parser. Each bolt has its input
> spout stored in a List, so I want to do something like the following:
>
> ...
>
> TopologyParser parser = new TopologyParser();
> parser.parseTopology("some-file.xml");
> components = parser.getComponents();
> Iterator<Component> itr = components.iterator();
> builder = new TopologyBuilder();
> while(itr.hasNext()) {
> Component comp = itr.next();
> if(comp.getType().equals("Bolt")) {
> BoltDeclarer declarer = builder.setBolt(comp.getName(), new
> SampleBolt(comp.getName(), comp.getExecutors());
> if(comp.getUpstreamTasks() != null && comp.getUpstreamTasks().size()
> > 0) {
> for(String upstream_task : comp.getUpstreamTasks()) {
> declarer.directGrouping(upstream_task);
> }
> }
> }
> }
>
> Config conf = new Config();
> conf.setDebug(true);
> LocalCluster cluster = new LocalCluster();
> cluster.submitTopology("test", conf, builder.createTopology());
>
> ...
>
> So, in the above code the TopologyParser parser parses the topology from a
> file and the components variable is a list of all the Component objects
> (the TopologyParser and the Component classes
> are custom built by me). If a component is for type "Bolt" and has
> upstream tasks, they have their output registered as directly grouped to
> the bolt. Unfortunately, the above approach does not seem to work.
> Any suggestions on how to make something the above work?
>
> Thank you,
> Nikos
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
--
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the
addressee you must not use, copy, disclose or take action based on this
message or any information herein.
If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Thank you.