You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@heron.apache.org by Josh Fischer <jo...@joshfischer.io> on 2018/01/06 02:30:51 UTC
Heron & Flux Style Topologies
Hi Dave,
I am currently working on a Flux like way to deploy topologies for Heron.
We've decided to call it ECO. It's based off of the already working code
that Flux is built on. Within the next week I plan to open a pull request
to the core committers.
So the idea is (hopefully) that you can take any Storm Flux topology with
spouts and bolts defined in Java and run it directly in Heron without any
code changes. So far I have this working with several topologies.
However, I would like the community to test it more as I am sure there are
edge cases I haven't accounted for yet.
As of now, this will only work with Storm components. If you have any
questions, please let me know!
-Josh
Re: Heron & Flux Style Topologies
Posted by Dave Fisher <da...@comcast.net>.
Hi -
Thank you both.
Regards,
Dave
Sent from my iPhone
> On Jan 6, 2018, at 6:24 AM, Josh Fischer <jo...@joshfischer.io> wrote:
>
> Karthik,
>
> I apologize, I explained that wrong. Thanks for the follow up on the correct way to move over.
>
>
>> On Sat, Jan 6, 2018 at 1:22 AM Karthik Ramasamy <ka...@streaml.io> wrote:
>> Dave -
>>
>> In order to launch the Storm topology, you need to recompile your Storm topology using Heron which is pretty straightforward.
>>
>> cheers
>> /karthik
>>
>> > On Jan 5, 2018, at 7:43 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>> >
>> > Dave, sorry I missed your last inquiry:
>> >
>> > You said "I was wondering about the YAML format to define a topology."
>> >
>> > *At this time the yaml definition should be the exact same as Storm Flux
>> > Topology. This way you shouldn't have to make any changes to migrate to
>> > Heron with a Storm Flux topology*
>> >
>> > On Fri, Jan 5, 2018 at 9:39 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>> >
>> >> My responses are inlined in *blue*
>> >>
>> >> On Fri, Jan 5, 2018 at 9:08 PM, Dave Fisher <da...@comcast.net> wrote:
>> >>
>> >>> Hi Josh,
>> >>>
>> >>> Sent from my iPhone
>> >>>
>> >>>> On Jan 5, 2018, at 6:30 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>> >>>>
>> >>>> Hi Dave,
>> >>>>
>> >>>> I am currently working on a Flux like way to deploy topologies for
>> >>> Heron. We've decided to call it ECO. It's based off of the already
>> >>> working code that Flux is built on. Within the next week I plan to open a
>> >>> pull request to the core committers.
>> >>>
>> >>> Please let the email list know. I’d like to take a look.
>> >>> *will do*
>> >>>>
>> >>>> So the idea is (hopefully) that you can take any Storm Flux topology
>> >>> with spouts and bolts defined in Java and run it directly in Heron without
>> >>> any code changes. So far I have this working with several topologies.
>> >>> However, I would like the community to test it more as I am sure there are
>> >>> edge cases I haven't accounted for yet.
>> >>>
>> >>> Does this handle Python based bolts?
>> >>>
>> >> *I have ran a topology that uses the FluxShellBolt and FluxShellSpout,
>> >> each of them worked as expected. If you run the Flux Topologies from the
>> >> Flux examples they should all work. I haven't extensively tested the
>> >> multi-lang feature yet.*
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>>
>> >>> Does this mean I take a topology jar built with Maven for Storm and it
>> >>> runs fine in Heron? That would be cool.
>> >>>
>> >>
>> >>
>> >>> *Yes, you should be able to take the same jar that was build with maven
>> >>> and run it without the need to recompile or change anything. *
>> >>
>> >>
>> >> * An example of how submit an ECO topology is below*
>> >> heron submit local ~/path/to/jar-with-dependencies.jar \
>> >> com.twitter.heron.eco.Eco \
>> >> --eco-config-file /path/to/yaml/simple_wordcount.yaml
>> >>
>> >>>
>> >>>>
>> >>>> As of now, this will only work with Storm components. If you have any
>> >>> questions, please let me know!
>> >>>
>> >>> I was wondering about the YAML format to define a topology.
>> >>>
>> >>
>> >>
>> >>>
>> >>> Regards,
>> >>> Dave
>> >>>
>> >>>
>> >>>>
>> >>>> -Josh
>> >>>
>> >>>
>> >>
>>
>
> --
> Sent from A Mobile Device
Re: Heron & Flux Style Topologies
Posted by Karthik Ramasamy <ka...@streaml.io>.
No worries. We are API compatible.
On Sat, Jan 6, 2018 at 6:24 AM, Josh Fischer <jo...@joshfischer.io> wrote:
> Karthik,
>
> I apologize, I explained that wrong. Thanks for the follow up on the
> correct way to move over.
>
>
> On Sat, Jan 6, 2018 at 1:22 AM Karthik Ramasamy <ka...@streaml.io>
> wrote:
>
> > Dave -
> >
> > In order to launch the Storm topology, you need to recompile your Storm
> > topology using Heron which is pretty straightforward.
> >
> > cheers
> > /karthik
> >
> > > On Jan 5, 2018, at 7:43 PM, Josh Fischer <jo...@joshfischer.io> wrote:
> > >
> > > Dave, sorry I missed your last inquiry:
> > >
> > > You said "I was wondering about the YAML format to define a topology."
> > >
> > > *At this time the yaml definition should be the exact same as Storm
> Flux
> > > Topology. This way you shouldn't have to make any changes to migrate
> to
> > > Heron with a Storm Flux topology*
> > >
> > > On Fri, Jan 5, 2018 at 9:39 PM, Josh Fischer <jo...@joshfischer.io>
> > wrote:
> > >
> > >> My responses are inlined in *blue*
> > >>
> > >> On Fri, Jan 5, 2018 at 9:08 PM, Dave Fisher <da...@comcast.net>
> > wrote:
> > >>
> > >>> Hi Josh,
> > >>>
> > >>> Sent from my iPhone
> > >>>
> > >>>> On Jan 5, 2018, at 6:30 PM, Josh Fischer <jo...@joshfischer.io>
> wrote:
> > >>>>
> > >>>> Hi Dave,
> > >>>>
> > >>>> I am currently working on a Flux like way to deploy topologies for
> > >>> Heron. We've decided to call it ECO. It's based off of the already
> > >>> working code that Flux is built on. Within the next week I plan to
> > open a
> > >>> pull request to the core committers.
> > >>>
> > >>> Please let the email list know. I’d like to take a look.
> > >>> *will do*
> > >>>>
> > >>>> So the idea is (hopefully) that you can take any Storm Flux topology
> > >>> with spouts and bolts defined in Java and run it directly in Heron
> > without
> > >>> any code changes. So far I have this working with several
> topologies.
> > >>> However, I would like the community to test it more as I am sure
> there
> > are
> > >>> edge cases I haven't accounted for yet.
> > >>>
> > >>> Does this handle Python based bolts?
> > >>>
> > >> *I have ran a topology that uses the FluxShellBolt and FluxShellSpout,
> > >> each of them worked as expected. If you run the Flux Topologies from
> > the
> > >> Flux examples they should all work. I haven't extensively tested the
> > >> multi-lang feature yet.*
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>>
> > >>> Does this mean I take a topology jar built with Maven for Storm and
> it
> > >>> runs fine in Heron? That would be cool.
> > >>>
> > >>
> > >>
> > >>> *Yes, you should be able to take the same jar that was build with
> maven
> > >>> and run it without the need to recompile or change anything. *
> > >>
> > >>
> > >> * An example of how submit an ECO topology is below*
> > >> heron submit local ~/path/to/jar-with-dependencies.jar \
> > >> com.twitter.heron.eco.Eco \
> > >> --eco-config-file /path/to/yaml/simple_wordcount.yaml
> > >>
> > >>>
> > >>>>
> > >>>> As of now, this will only work with Storm components. If you have
> any
> > >>> questions, please let me know!
> > >>>
> > >>> I was wondering about the YAML format to define a topology.
> > >>>
> > >>
> > >>
> > >>>
> > >>> Regards,
> > >>> Dave
> > >>>
> > >>>
> > >>>>
> > >>>> -Josh
> > >>>
> > >>>
> > >>
> >
> > --
> Sent from A Mobile Device
>
Re: Heron & Flux Style Topologies
Posted by Josh Fischer <jo...@joshfischer.io>.
Karthik,
I apologize, I explained that wrong. Thanks for the follow up on the
correct way to move over.
On Sat, Jan 6, 2018 at 1:22 AM Karthik Ramasamy <ka...@streaml.io> wrote:
> Dave -
>
> In order to launch the Storm topology, you need to recompile your Storm
> topology using Heron which is pretty straightforward.
>
> cheers
> /karthik
>
> > On Jan 5, 2018, at 7:43 PM, Josh Fischer <jo...@joshfischer.io> wrote:
> >
> > Dave, sorry I missed your last inquiry:
> >
> > You said "I was wondering about the YAML format to define a topology."
> >
> > *At this time the yaml definition should be the exact same as Storm Flux
> > Topology. This way you shouldn't have to make any changes to migrate to
> > Heron with a Storm Flux topology*
> >
> > On Fri, Jan 5, 2018 at 9:39 PM, Josh Fischer <jo...@joshfischer.io>
> wrote:
> >
> >> My responses are inlined in *blue*
> >>
> >> On Fri, Jan 5, 2018 at 9:08 PM, Dave Fisher <da...@comcast.net>
> wrote:
> >>
> >>> Hi Josh,
> >>>
> >>> Sent from my iPhone
> >>>
> >>>> On Jan 5, 2018, at 6:30 PM, Josh Fischer <jo...@joshfischer.io> wrote:
> >>>>
> >>>> Hi Dave,
> >>>>
> >>>> I am currently working on a Flux like way to deploy topologies for
> >>> Heron. We've decided to call it ECO. It's based off of the already
> >>> working code that Flux is built on. Within the next week I plan to
> open a
> >>> pull request to the core committers.
> >>>
> >>> Please let the email list know. I’d like to take a look.
> >>> *will do*
> >>>>
> >>>> So the idea is (hopefully) that you can take any Storm Flux topology
> >>> with spouts and bolts defined in Java and run it directly in Heron
> without
> >>> any code changes. So far I have this working with several topologies.
> >>> However, I would like the community to test it more as I am sure there
> are
> >>> edge cases I haven't accounted for yet.
> >>>
> >>> Does this handle Python based bolts?
> >>>
> >> *I have ran a topology that uses the FluxShellBolt and FluxShellSpout,
> >> each of them worked as expected. If you run the Flux Topologies from
> the
> >> Flux examples they should all work. I haven't extensively tested the
> >> multi-lang feature yet.*
> >>
> >>
> >>
> >>
> >>
> >>>
> >>> Does this mean I take a topology jar built with Maven for Storm and it
> >>> runs fine in Heron? That would be cool.
> >>>
> >>
> >>
> >>> *Yes, you should be able to take the same jar that was build with maven
> >>> and run it without the need to recompile or change anything. *
> >>
> >>
> >> * An example of how submit an ECO topology is below*
> >> heron submit local ~/path/to/jar-with-dependencies.jar \
> >> com.twitter.heron.eco.Eco \
> >> --eco-config-file /path/to/yaml/simple_wordcount.yaml
> >>
> >>>
> >>>>
> >>>> As of now, this will only work with Storm components. If you have any
> >>> questions, please let me know!
> >>>
> >>> I was wondering about the YAML format to define a topology.
> >>>
> >>
> >>
> >>>
> >>> Regards,
> >>> Dave
> >>>
> >>>
> >>>>
> >>>> -Josh
> >>>
> >>>
> >>
>
> --
Sent from A Mobile Device
Re: Heron & Flux Style Topologies
Posted by Karthik Ramasamy <ka...@streaml.io>.
Dave -
In order to launch the Storm topology, you need to recompile your Storm topology using Heron which is pretty straightforward.
cheers
/karthik
> On Jan 5, 2018, at 7:43 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>
> Dave, sorry I missed your last inquiry:
>
> You said "I was wondering about the YAML format to define a topology."
>
> *At this time the yaml definition should be the exact same as Storm Flux
> Topology. This way you shouldn't have to make any changes to migrate to
> Heron with a Storm Flux topology*
>
> On Fri, Jan 5, 2018 at 9:39 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>
>> My responses are inlined in *blue*
>>
>> On Fri, Jan 5, 2018 at 9:08 PM, Dave Fisher <da...@comcast.net> wrote:
>>
>>> Hi Josh,
>>>
>>> Sent from my iPhone
>>>
>>>> On Jan 5, 2018, at 6:30 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>>>>
>>>> Hi Dave,
>>>>
>>>> I am currently working on a Flux like way to deploy topologies for
>>> Heron. We've decided to call it ECO. It's based off of the already
>>> working code that Flux is built on. Within the next week I plan to open a
>>> pull request to the core committers.
>>>
>>> Please let the email list know. I’d like to take a look.
>>> *will do*
>>>>
>>>> So the idea is (hopefully) that you can take any Storm Flux topology
>>> with spouts and bolts defined in Java and run it directly in Heron without
>>> any code changes. So far I have this working with several topologies.
>>> However, I would like the community to test it more as I am sure there are
>>> edge cases I haven't accounted for yet.
>>>
>>> Does this handle Python based bolts?
>>>
>> *I have ran a topology that uses the FluxShellBolt and FluxShellSpout,
>> each of them worked as expected. If you run the Flux Topologies from the
>> Flux examples they should all work. I haven't extensively tested the
>> multi-lang feature yet.*
>>
>>
>>
>>
>>
>>>
>>> Does this mean I take a topology jar built with Maven for Storm and it
>>> runs fine in Heron? That would be cool.
>>>
>>
>>
>>> *Yes, you should be able to take the same jar that was build with maven
>>> and run it without the need to recompile or change anything. *
>>
>>
>> * An example of how submit an ECO topology is below*
>> heron submit local ~/path/to/jar-with-dependencies.jar \
>> com.twitter.heron.eco.Eco \
>> --eco-config-file /path/to/yaml/simple_wordcount.yaml
>>
>>>
>>>>
>>>> As of now, this will only work with Storm components. If you have any
>>> questions, please let me know!
>>>
>>> I was wondering about the YAML format to define a topology.
>>>
>>
>>
>>>
>>> Regards,
>>> Dave
>>>
>>>
>>>>
>>>> -Josh
>>>
>>>
>>
Re: Heron & Flux Style Topologies
Posted by Josh Fischer <jo...@joshfischer.io>.
Dave, sorry I missed your last inquiry:
You said "I was wondering about the YAML format to define a topology."
*At this time the yaml definition should be the exact same as Storm Flux
Topology. This way you shouldn't have to make any changes to migrate to
Heron with a Storm Flux topology*
On Fri, Jan 5, 2018 at 9:39 PM, Josh Fischer <jo...@joshfischer.io> wrote:
> My responses are inlined in *blue*
>
> On Fri, Jan 5, 2018 at 9:08 PM, Dave Fisher <da...@comcast.net> wrote:
>
>> Hi Josh,
>>
>> Sent from my iPhone
>>
>> > On Jan 5, 2018, at 6:30 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>> >
>> > Hi Dave,
>> >
>> > I am currently working on a Flux like way to deploy topologies for
>> Heron. We've decided to call it ECO. It's based off of the already
>> working code that Flux is built on. Within the next week I plan to open a
>> pull request to the core committers.
>>
>> Please let the email list know. I’d like to take a look.
>> *will do*
>> >
>> > So the idea is (hopefully) that you can take any Storm Flux topology
>> with spouts and bolts defined in Java and run it directly in Heron without
>> any code changes. So far I have this working with several topologies.
>> However, I would like the community to test it more as I am sure there are
>> edge cases I haven't accounted for yet.
>>
>> Does this handle Python based bolts?
>>
> *I have ran a topology that uses the FluxShellBolt and FluxShellSpout,
> each of them worked as expected. If you run the Flux Topologies from the
> Flux examples they should all work. I haven't extensively tested the
> multi-lang feature yet.*
>
>
>
>
>
>>
>> Does this mean I take a topology jar built with Maven for Storm and it
>> runs fine in Heron? That would be cool.
>>
>
>
>> *Yes, you should be able to take the same jar that was build with maven
>> and run it without the need to recompile or change anything. *
>
>
> * An example of how submit an ECO topology is below*
> heron submit local ~/path/to/jar-with-dependencies.jar \
> com.twitter.heron.eco.Eco \
> --eco-config-file /path/to/yaml/simple_wordcount.yaml
>
>>
>> >
>> > As of now, this will only work with Storm components. If you have any
>> questions, please let me know!
>>
>> I was wondering about the YAML format to define a topology.
>>
>
>
>>
>> Regards,
>> Dave
>>
>>
>> >
>> > -Josh
>>
>>
>
Re: Heron & Flux Style Topologies
Posted by Josh Fischer <jo...@joshfischer.io>.
My responses are inlined in *blue*
On Fri, Jan 5, 2018 at 9:08 PM, Dave Fisher <da...@comcast.net> wrote:
> Hi Josh,
>
> Sent from my iPhone
>
> > On Jan 5, 2018, at 6:30 PM, Josh Fischer <jo...@joshfischer.io> wrote:
> >
> > Hi Dave,
> >
> > I am currently working on a Flux like way to deploy topologies for
> Heron. We've decided to call it ECO. It's based off of the already
> working code that Flux is built on. Within the next week I plan to open a
> pull request to the core committers.
>
> Please let the email list know. I’d like to take a look.
> *will do*
> >
> > So the idea is (hopefully) that you can take any Storm Flux topology
> with spouts and bolts defined in Java and run it directly in Heron without
> any code changes. So far I have this working with several topologies.
> However, I would like the community to test it more as I am sure there are
> edge cases I haven't accounted for yet.
>
> Does this handle Python based bolts?
>
*I have ran a topology that uses the FluxShellBolt and FluxShellSpout,
each of them worked as expected. If you run the Flux Topologies from the
Flux examples they should all work. I haven't extensively tested the
multi-lang feature yet.*
>
> Does this mean I take a topology jar built with Maven for Storm and it
> runs fine in Heron? That would be cool.
>
> *Yes, you should be able to take the same jar that was build with maven
> and run it without the need to recompile or change anything. *
* An example of how submit an ECO topology is below*
heron submit local ~/path/to/jar-with-dependencies.jar \
com.twitter.heron.eco.Eco \
--eco-config-file /path/to/yaml/simple_wordcount.yaml
>
> >
> > As of now, this will only work with Storm components. If you have any
> questions, please let me know!
>
> I was wondering about the YAML format to define a topology.
>
>
> Regards,
> Dave
>
>
> >
> > -Josh
>
>
Re: Heron & Flux Style Topologies
Posted by Dave Fisher <da...@comcast.net>.
Hi Josh,
Sent from my iPhone
> On Jan 5, 2018, at 6:30 PM, Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Dave,
>
> I am currently working on a Flux like way to deploy topologies for Heron. We've decided to call it ECO. It's based off of the already working code that Flux is built on. Within the next week I plan to open a pull request to the core committers.
Please let the email list know. I’d like to take a look.
>
> So the idea is (hopefully) that you can take any Storm Flux topology with spouts and bolts defined in Java and run it directly in Heron without any code changes. So far I have this working with several topologies. However, I would like the community to test it more as I am sure there are edge cases I haven't accounted for yet.
Does this handle Python based bolts?
Does this mean I take a topology jar built with Maven for Storm and it runs fine in Heron? That would be cool.
>
> As of now, this will only work with Storm components. If you have any questions, please let me know!
I was wondering about the YAML format to define a topology.
Regards,
Dave
>
> -Josh