You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Nathan Leung <nc...@gmail.com> on 2014/11/17 23:37:00 UTC
Survey: dynamic topology construction
Hi,
Every time I see a thread like this (
http://mail-archives.apache.org/mod_mbox/storm-user/201411.mbox/browser),
it pains me. I have written code to be able to create a topology from json
and execute it, but I am not allowed to share it. So far I have resisted
the temptation to reproduce it in my own time because I don't have much
free time, and because rewriting something isn't always the most fun thing
to do. However, if enough people would use it, I would be more inclined to
invest the time.
So, two questions:
1) would you use this?
2) what features would you like to see (in addition to below)?
Proposed features:
a) pojo topology representation: leave choice of json or xml library up to
the user
b) support of all groupings, streams, via reflections
c) ability to specify configuration map via config file
d) reification of spout / bolt / dependency arguments
e) recursive iteration of d) for construction of complex types
Re: Survey: dynamic topology construction
Posted by Nathan Leung <nc...@gmail.com>.
Hi everyone,
Thanks for your feedback. I've created a jira:
https://issues.apache.org/jira/browse/STORM-561. I have yet to create a
branch; I need to get a dev environment suitable for storm setup first.
-Nathan
On Tue, Nov 18, 2014 at 11:27 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> + dev@
>
> I think this is a good idea, and I would support it. It seems there’s a
> lot of “wheel reinvention” going on. I’ve been thinking about implementing
> something along these lines as well.
>
> I’d propose opening a JIRA and corresponding feature branch to enable
> collaboration.
>
> One thing to keep in mind is that what’s being proposed (and what I have
> in mind) has a lot of similarities to what Yelp has done with pyleus [1].
> So we should consider reaching out to Yelp devs to see if they’d be
> interested in collaborating. Licensing/IP should also be kept in mind.
>
> I envision something a little more “API agnostic” — supporting JVM,
> multi-lang, and Trident.
>
> I’d be interested in hearing others’ ideas.
>
> -Taylor
>
> [1] https://github.com/Yelp/pyleus
>
> On Nov 18, 2014, at 11:37 AM, c_inconnu <c_...@yahoo.fr> wrote:
>
> Hi,
> I've also developed for my company a small framework to create a topology
> from JSON.
> I'm not sure if I'm allowed to open source it but it seems a common
> requirement.
> Maybe we can join forces.
> David
>
> 2014-11-18 16:42 GMT+01:00 Nick Katsipoulakis <ni...@gmail.com>:
>
>> Hello,
>>
>> I am also doing something similar at the moment. Nathan, I agree that
>> something like that would be extremely useful.Therefore, to answer your
>> questions:
>>
>> 1) Yes
>>
>> 2) Can not find any additional supported features at this point.
>>
>> Thanks,
>> Nikos
>>
>> 2014-11-18 10:15 GMT-05:00 Supun Kamburugamuva <su...@gmail.com>:
>>
>> Hi Nathan,
>>>
>>> I meant the algorithmic part of the middle bolts. We create the spouts
>>> and bolts that receive/send the messages using the configuration. We can
>>> create the full topology if the middle bolts are also specified but not the
>>> processing part (of course). Still we haven't implemented connecting the
>>> middle part because it seems our requirements are changing and we haven't
>>> finalized all the requirements etc.
>>>
>>> Overall I think this is a very useful feature.
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>>
>>> On Tue, Nov 18, 2014 at 9:41 AM, Nathan Leung <nc...@gmail.com> wrote:
>>>
>>>> Hi Supun, what kind of problems have you had with your middle bolts? I
>>>> have been creating production topologies fully from json but maybe there's
>>>> a feature I'm not using.
>>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>
>>
>> --
>> Nikolaos Romanos Katsipoulakis,
>> University of Pittsburgh, PhD candidate
>>
>
>
>
Re: Survey: dynamic topology construction
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
+ dev@
I think this is a good idea, and I would support it. It seems there’s a lot of “wheel reinvention” going on. I’ve been thinking about implementing something along these lines as well.
I’d propose opening a JIRA and corresponding feature branch to enable collaboration.
One thing to keep in mind is that what’s being proposed (and what I have in mind) has a lot of similarities to what Yelp has done with pyleus [1]. So we should consider reaching out to Yelp devs to see if they’d be interested in collaborating. Licensing/IP should also be kept in mind.
I envision something a little more “API agnostic” — supporting JVM, multi-lang, and Trident.
I’d be interested in hearing others’ ideas.
-Taylor
[1] https://github.com/Yelp/pyleus
On Nov 18, 2014, at 11:37 AM, c_inconnu <c_...@yahoo.fr> wrote:
> Hi,
> I've also developed for my company a small framework to create a topology from JSON.
> I'm not sure if I'm allowed to open source it but it seems a common requirement.
> Maybe we can join forces.
> David
>
> 2014-11-18 16:42 GMT+01:00 Nick Katsipoulakis <ni...@gmail.com>:
> Hello,
>
> I am also doing something similar at the moment. Nathan, I agree that something like that would be extremely useful.Therefore, to answer your questions:
>
> 1) Yes
>
> 2) Can not find any additional supported features at this point.
>
> Thanks,
> Nikos
>
> 2014-11-18 10:15 GMT-05:00 Supun Kamburugamuva <su...@gmail.com>:
>
> Hi Nathan,
>
> I meant the algorithmic part of the middle bolts. We create the spouts and bolts that receive/send the messages using the configuration. We can create the full topology if the middle bolts are also specified but not the processing part (of course). Still we haven't implemented connecting the middle part because it seems our requirements are changing and we haven't finalized all the requirements etc.
>
> Overall I think this is a very useful feature.
>
> Thanks,
> Supun..
>
>
>
> On Tue, Nov 18, 2014 at 9:41 AM, Nathan Leung <nc...@gmail.com> wrote:
> Hi Supun, what kind of problems have you had with your middle bolts? I have been creating production topologies fully from json but maybe there's a feature I'm not using.
>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
Re: Survey: dynamic topology construction
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
+ dev@
I think this is a good idea, and I would support it. It seems there’s a lot of “wheel reinvention” going on. I’ve been thinking about implementing something along these lines as well.
I’d propose opening a JIRA and corresponding feature branch to enable collaboration.
One thing to keep in mind is that what’s being proposed (and what I have in mind) has a lot of similarities to what Yelp has done with pyleus [1]. So we should consider reaching out to Yelp devs to see if they’d be interested in collaborating. Licensing/IP should also be kept in mind.
I envision something a little more “API agnostic” — supporting JVM, multi-lang, and Trident.
I’d be interested in hearing others’ ideas.
-Taylor
[1] https://github.com/Yelp/pyleus
On Nov 18, 2014, at 11:37 AM, c_inconnu <c_...@yahoo.fr> wrote:
> Hi,
> I've also developed for my company a small framework to create a topology from JSON.
> I'm not sure if I'm allowed to open source it but it seems a common requirement.
> Maybe we can join forces.
> David
>
> 2014-11-18 16:42 GMT+01:00 Nick Katsipoulakis <ni...@gmail.com>:
> Hello,
>
> I am also doing something similar at the moment. Nathan, I agree that something like that would be extremely useful.Therefore, to answer your questions:
>
> 1) Yes
>
> 2) Can not find any additional supported features at this point.
>
> Thanks,
> Nikos
>
> 2014-11-18 10:15 GMT-05:00 Supun Kamburugamuva <su...@gmail.com>:
>
> Hi Nathan,
>
> I meant the algorithmic part of the middle bolts. We create the spouts and bolts that receive/send the messages using the configuration. We can create the full topology if the middle bolts are also specified but not the processing part (of course). Still we haven't implemented connecting the middle part because it seems our requirements are changing and we haven't finalized all the requirements etc.
>
> Overall I think this is a very useful feature.
>
> Thanks,
> Supun..
>
>
>
> On Tue, Nov 18, 2014 at 9:41 AM, Nathan Leung <nc...@gmail.com> wrote:
> Hi Supun, what kind of problems have you had with your middle bolts? I have been creating production topologies fully from json but maybe there's a feature I'm not using.
>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
Re: Survey: dynamic topology construction
Posted by c_inconnu <c_...@yahoo.fr>.
Hi,
I've also developed for my company a small framework to create a topology
from JSON.
I'm not sure if I'm allowed to open source it but it seems a common
requirement.
Maybe we can join forces.
David
2014-11-18 16:42 GMT+01:00 Nick Katsipoulakis <ni...@gmail.com>:
> Hello,
>
> I am also doing something similar at the moment. Nathan, I agree that
> something like that would be extremely useful.Therefore, to answer your
> questions:
>
> 1) Yes
>
> 2) Can not find any additional supported features at this point.
>
> Thanks,
> Nikos
>
> 2014-11-18 10:15 GMT-05:00 Supun Kamburugamuva <su...@gmail.com>:
>
> Hi Nathan,
>>
>> I meant the algorithmic part of the middle bolts. We create the spouts
>> and bolts that receive/send the messages using the configuration. We can
>> create the full topology if the middle bolts are also specified but not the
>> processing part (of course). Still we haven't implemented connecting the
>> middle part because it seems our requirements are changing and we haven't
>> finalized all the requirements etc.
>>
>> Overall I think this is a very useful feature.
>>
>> Thanks,
>> Supun..
>>
>>
>>
>> On Tue, Nov 18, 2014 at 9:41 AM, Nathan Leung <nc...@gmail.com> wrote:
>>
>>> Hi Supun, what kind of problems have you had with your middle bolts? I
>>> have been creating production topologies fully from json but maybe there's
>>> a feature I'm not using.
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
Re: Survey: dynamic topology construction
Posted by Nick Katsipoulakis <ni...@gmail.com>.
Hello,
I am also doing something similar at the moment. Nathan, I agree that
something like that would be extremely useful.Therefore, to answer your
questions:
1) Yes
2) Can not find any additional supported features at this point.
Thanks,
Nikos
2014-11-18 10:15 GMT-05:00 Supun Kamburugamuva <su...@gmail.com>:
> Hi Nathan,
>
> I meant the algorithmic part of the middle bolts. We create the spouts and
> bolts that receive/send the messages using the configuration. We can create
> the full topology if the middle bolts are also specified but not the
> processing part (of course). Still we haven't implemented connecting the
> middle part because it seems our requirements are changing and we haven't
> finalized all the requirements etc.
>
> Overall I think this is a very useful feature.
>
> Thanks,
> Supun..
>
>
>
> On Tue, Nov 18, 2014 at 9:41 AM, Nathan Leung <nc...@gmail.com> wrote:
>
>> Hi Supun, what kind of problems have you had with your middle bolts? I
>> have been creating production topologies fully from json but maybe there's
>> a feature I'm not using.
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Survey: dynamic topology construction
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Nathan,
I meant the algorithmic part of the middle bolts. We create the spouts and
bolts that receive/send the messages using the configuration. We can create
the full topology if the middle bolts are also specified but not the
processing part (of course). Still we haven't implemented connecting the
middle part because it seems our requirements are changing and we haven't
finalized all the requirements etc.
Overall I think this is a very useful feature.
Thanks,
Supun..
On Tue, Nov 18, 2014 at 9:41 AM, Nathan Leung <nc...@gmail.com> wrote:
> Hi Supun, what kind of problems have you had with your middle bolts? I
> have been creating production topologies fully from json but maybe there's
> a feature I'm not using.
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Survey: dynamic topology construction
Posted by Nathan Leung <nc...@gmail.com>.
Hi Supun, what kind of problems have you had with your middle bolts? I have
been creating production topologies fully from json but maybe there's a
feature I'm not using.
Re: Survey: dynamic topology construction
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Nathan,
We are also creating topologies by looking at a YAML based configuration.
For our case we create the spouts and bolts that connect to external
systems directly using the configuration. The bolts in the middle are not
that easy to create by looking at a configuration.
Thanks,
Supun..
On Tue, Nov 18, 2014 at 1:51 AM, Kobi Salant <ko...@liveperson.com> wrote:
> Hi Nathan,
>
> We actually created our own code to create topology from json:-)
> I think it is great idea and we would probably switch to a more complete
> code if possible.
>
> We would like the code to be extendable so we can add our own logic in the
> creation processes other then supply our own spout and bolts classes.
>
> Thanks
> Kovi
>
> On Tue, Nov 18, 2014 at 12:37 AM, Nathan Leung <nc...@gmail.com> wrote:
>
>> Hi,
>>
>> Every time I see a thread like this (
>> http://mail-archives.apache.org/mod_mbox/storm-user/201411.mbox/browser),
>> it pains me. I have written code to be able to create a topology from json
>> and execute it, but I am not allowed to share it. So far I have resisted
>> the temptation to reproduce it in my own time because I don't have much
>> free time, and because rewriting something isn't always the most fun thing
>> to do. However, if enough people would use it, I would be more inclined to
>> invest the time.
>>
>> So, two questions:
>>
>> 1) would you use this?
>>
>> 2) what features would you like to see (in addition to below)?
>>
>> Proposed features:
>>
>> a) pojo topology representation: leave choice of json or xml library up
>> to the user
>>
>> b) support of all groupings, streams, via reflections
>>
>> c) ability to specify configuration map via config file
>>
>> d) reification of spout / bolt / dependency arguments
>>
>> e) recursive iteration of d) for construction of complex types
>>
>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Survey: dynamic topology construction
Posted by Kobi Salant <ko...@liveperson.com>.
Hi Nathan,
We actually created our own code to create topology from json:-)
I think it is great idea and we would probably switch to a more complete
code if possible.
We would like the code to be extendable so we can add our own logic in the
creation processes other then supply our own spout and bolts classes.
Thanks
Kovi
On Tue, Nov 18, 2014 at 12:37 AM, Nathan Leung <nc...@gmail.com> wrote:
> Hi,
>
> Every time I see a thread like this (
> http://mail-archives.apache.org/mod_mbox/storm-user/201411.mbox/browser),
> it pains me. I have written code to be able to create a topology from json
> and execute it, but I am not allowed to share it. So far I have resisted
> the temptation to reproduce it in my own time because I don't have much
> free time, and because rewriting something isn't always the most fun thing
> to do. However, if enough people would use it, I would be more inclined to
> invest the time.
>
> So, two questions:
>
> 1) would you use this?
>
> 2) what features would you like to see (in addition to below)?
>
> Proposed features:
>
> a) pojo topology representation: leave choice of json or xml library up to
> the user
>
> b) support of all groupings, streams, via reflections
>
> c) ability to specify configuration map via config file
>
> d) reification of spout / bolt / dependency arguments
>
> e) recursive iteration of d) for construction of complex types
>
--
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the
addressee you must not use, copy, disclose or take action based on this
message or any information herein.
If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Thank you.