Re: Storm multilang - .net core

[Stig Rohde Døssing <st...@gmail.com>]

Sorry about necro'ing this thread, but I think this is relevant again, due
to https://github.com/apache/storm/pull/2613.

I think it makes more sense to have adapters like this in separate
repositories from Storm, as they can then release fixes when they need to,
without being coupled to the Storm release schedule. For example,
https://github.com/Parsely/streamparse is released this way. Releasing
separately also makes it easier for users to tell whether the adapter is
still being actively developed.

Maybe we should add an "Ecosystem" section to the web page, similar to what
Kafka has here https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem.
It would help users find projects related to Storm, which is one of the few
advantages I can see to adding the adapter to the Storm repo (visibility).

2018-02-02 5:23 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:

> CORRECTION: I don't work on multilang for now. "had some works" on
> multilang and introduced small changes on multilang protocol.
>
> 2018년 2월 2일 (금) 오전 9:12, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> > Please remove or bcc. for company mail/mailing list which don't allow
> > sending mail outside of MS. My previous mail bounced. I just removed CCC
> > from recipient in current mail.
> >
> > 2018년 2월 2일 (금) 오전 9:08, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> >> Mauro,
> >>
> >> First of all, thanks for proposing contribution of valuable effort on
> >> Apache Storm! Really appreciated.
> >>
> >> I don't know about C#/.net but I have been working on the change of
> >> multilang, so adding my voice here.
> >>
> >> 1.
> >> Major concern from my side for code donation is sustainability. Not sure
> >> but according to my experience, most of Storm PMCs and contributors
> don't
> >> look using Windows at their dev. environment. It doesn't strictly mean
> they
> >> don't have experience with C#, but relatively higher chance. I'm sorry
> to
> >> the Azure team, but I recently found a forgotten major pull request on
> >> storm-eventhubs, which doesn't look like no one could review and
> maintain.
> >> It might become real pain if we receive the code which we can't/don't
> >> maintain, hence I'd rather not vote +1 unless at least two PMCs know C#
> >> well, understand the code completely, and willing to volunteer to
> maintain.
> >>
> >> 2.
> >> As you may know, all of multilang adapters in Storm repo are actually
> >> close to "example" of implementations. They're just implementations of
> the
> >> protocol, and don't provide any language specific optimization as well
> as
> >> language-standard code style. Most of Python users in Storm community
> would
> >> rather use StreamParse, and it is not uncommon to see StreamParse
> question
> >> in user group (whereas they have their own Github repo and issue as
> well).
> >> I would like to see adapter (and more) projects really looking
> attractive
> >> for other languages as well.
> >>
> >> 3.
> >> How it affect releasing Storm? We don't publish them as package in its
> >> language package environment. If NuGet is one of them, we need to add
> the
> >> sequence of release phase for NuGet while releasing Storm, which was not
> >> happened yet, and will become another pain. Moreover, if it's the case,
> I
> >> don't feel needs for having strict coupled between Storm and .net
> adapter
> >> package. For user side it's not a "battery-included" and there's no
> >> difference between maintaining inside/outside. You could freely use
> >> user/dev list to announce new .net adapter release and such
> announcements
> >> are happening from other projects as well.
> >>
> >> 4.
> >> It's completely a thought on my own, but I feel more needs of having
> more
> >> language-native way of supporting language instead of keep improving
> >> multilang. (Not meant to discontinue.) We will introduce streams API in
> >> Storm 2.0.0, a higher-level API like Trident but typed and
> record-to-record
> >> processing. We haven't supported Trident in multilang, but I'd like to
> see
> >> support streams API in non-JVM language, not only defining protocol, but
> >> also have it as first-class support (users should be able to construct
> >> their own topology with only using the language. there's thrift but not
> >> that convenient.). IMHO, according to Spark's "Lesson learned"
> (Databricks
> >> had a poll and published it), I think it's really clear that Python
> should
> >> be first (and only, R might be another good to have).
> >>
> >> Thanks for reading a wall of text. Regardless of whether we could
> include
> >> .net adapter into Storm core or not, thanks again for crafting .net
> adapter
> >> and proposing for donation.
> >>
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2018년 2월 2일 (금) 오전 2:03, Mauro Giusti <ma...@microsoft.com.invalid>님이
> >> 작성:
> >>
> >>> Hello Storm devs -
> >>>
> >>> We are working on a project that uses Storm and C# / .net core
> >>> components.
> >>>
> >>> As part of that, we would love to contribute to the Storm project with
> a
> >>> multilang protocol sample that uses our C# component/adapter.
> >>> The adapter will be a NuGet package, and we intend to publish the
> source
> >>> code of this component as well.
> >>>
> >>> With that in mind, I have two questions:
> >>>
> >>>   *   Would you prefer the code for the .net adapter (not the sample:
> >>> that will be on the Storm repo, just the adapter) to be hosted inside
> the
> >>> Storm repo or in a separate GitHub repo? We see pros and cons: we might
> >>> need to introduce .net core compilation in the Storm repo if we host
> the
> >>> adapter there, the code will be a bit more scattered and harder to
> test if
> >>> we host the adapter outside.
> >>>   *   Can you help us review / answer design questions about our
> adapter
> >>> and the multilang protocol? Is there a specific set of people that is
> more
> >>> knowledgeable about this and/or wants to help in this specific area?
> >>>
> >>> Thank you,
> >>> Mauro Giusti
> >>> Azure Team, Microsoft.
> >>>
> >>
>

Re: Storm multilang - .net core

["P. Taylor Goetz" <pt...@gmail.com>]

+1 I think that’s a good idea. It would also help to have some wording regarding how one requests a project be added to that page.

-Taylor

> On Apr 2, 2018, at 10:03 AM, Stig Rohde Døssing <st...@gmail.com> wrote:
> 
> Sorry about necro'ing this thread, but I think this is relevant again, due
> to https://github.com/apache/storm/pull/2613.
> 
> I think it makes more sense to have adapters like this in separate
> repositories from Storm, as they can then release fixes when they need to,
> without being coupled to the Storm release schedule. For example,
> https://github.com/Parsely/streamparse is released this way. Releasing
> separately also makes it easier for users to tell whether the adapter is
> still being actively developed.
> 
> Maybe we should add an "Ecosystem" section to the web page, similar to what
> Kafka has here https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem.
> It would help users find projects related to Storm, which is one of the few
> advantages I can see to adding the adapter to the Storm repo (visibility).
> 
> 2018-02-02 5:23 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
> 
>> CORRECTION: I don't work on multilang for now. "had some works" on
>> multilang and introduced small changes on multilang protocol.
>> 
>> 2018년 2월 2일 (금) 오전 9:12, Jungtaek Lim <ka...@gmail.com>님이 작성:
>> 
>>> Please remove or bcc. for company mail/mailing list which don't allow
>>> sending mail outside of MS. My previous mail bounced. I just removed CCC
>>> from recipient in current mail.
>>> 
>>> 2018년 2월 2일 (금) 오전 9:08, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>> 
>>>> Mauro,
>>>> 
>>>> First of all, thanks for proposing contribution of valuable effort on
>>>> Apache Storm! Really appreciated.
>>>> 
>>>> I don't know about C#/.net but I have been working on the change of
>>>> multilang, so adding my voice here.
>>>> 
>>>> 1.
>>>> Major concern from my side for code donation is sustainability. Not sure
>>>> but according to my experience, most of Storm PMCs and contributors
>> don't
>>>> look using Windows at their dev. environment. It doesn't strictly mean
>> they
>>>> don't have experience with C#, but relatively higher chance. I'm sorry
>> to
>>>> the Azure team, but I recently found a forgotten major pull request on
>>>> storm-eventhubs, which doesn't look like no one could review and
>> maintain.
>>>> It might become real pain if we receive the code which we can't/don't
>>>> maintain, hence I'd rather not vote +1 unless at least two PMCs know C#
>>>> well, understand the code completely, and willing to volunteer to
>> maintain.
>>>> 
>>>> 2.
>>>> As you may know, all of multilang adapters in Storm repo are actually
>>>> close to "example" of implementations. They're just implementations of
>> the
>>>> protocol, and don't provide any language specific optimization as well
>> as
>>>> language-standard code style. Most of Python users in Storm community
>> would
>>>> rather use StreamParse, and it is not uncommon to see StreamParse
>> question
>>>> in user group (whereas they have their own Github repo and issue as
>> well).
>>>> I would like to see adapter (and more) projects really looking
>> attractive
>>>> for other languages as well.
>>>> 
>>>> 3.
>>>> How it affect releasing Storm? We don't publish them as package in its
>>>> language package environment. If NuGet is one of them, we need to add
>> the
>>>> sequence of release phase for NuGet while releasing Storm, which was not
>>>> happened yet, and will become another pain. Moreover, if it's the case,
>> I
>>>> don't feel needs for having strict coupled between Storm and .net
>> adapter
>>>> package. For user side it's not a "battery-included" and there's no
>>>> difference between maintaining inside/outside. You could freely use
>>>> user/dev list to announce new .net adapter release and such
>> announcements
>>>> are happening from other projects as well.
>>>> 
>>>> 4.
>>>> It's completely a thought on my own, but I feel more needs of having
>> more
>>>> language-native way of supporting language instead of keep improving
>>>> multilang. (Not meant to discontinue.) We will introduce streams API in
>>>> Storm 2.0.0, a higher-level API like Trident but typed and
>> record-to-record
>>>> processing. We haven't supported Trident in multilang, but I'd like to
>> see
>>>> support streams API in non-JVM language, not only defining protocol, but
>>>> also have it as first-class support (users should be able to construct
>>>> their own topology with only using the language. there's thrift but not
>>>> that convenient.). IMHO, according to Spark's "Lesson learned"
>> (Databricks
>>>> had a poll and published it), I think it's really clear that Python
>> should
>>>> be first (and only, R might be another good to have).
>>>> 
>>>> Thanks for reading a wall of text. Regardless of whether we could
>> include
>>>> .net adapter into Storm core or not, thanks again for crafting .net
>> adapter
>>>> and proposing for donation.
>>>> 
>>>> Jungtaek Lim (HeartSaVioR)
>>>> 
>>>> 2018년 2월 2일 (금) 오전 2:03, Mauro Giusti <ma...@microsoft.com.invalid>님이
>>>> 작성:
>>>> 
>>>>> Hello Storm devs -
>>>>> 
>>>>> We are working on a project that uses Storm and C# / .net core
>>>>> components.
>>>>> 
>>>>> As part of that, we would love to contribute to the Storm project with
>> a
>>>>> multilang protocol sample that uses our C# component/adapter.
>>>>> The adapter will be a NuGet package, and we intend to publish the
>> source
>>>>> code of this component as well.
>>>>> 
>>>>> With that in mind, I have two questions:
>>>>> 
>>>>>  *   Would you prefer the code for the .net adapter (not the sample:
>>>>> that will be on the Storm repo, just the adapter) to be hosted inside
>> the
>>>>> Storm repo or in a separate GitHub repo? We see pros and cons: we might
>>>>> need to introduce .net core compilation in the Storm repo if we host
>> the
>>>>> adapter there, the code will be a bit more scattered and harder to
>> test if
>>>>> we host the adapter outside.
>>>>>  *   Can you help us review / answer design questions about our
>> adapter
>>>>> and the multilang protocol? Is there a specific set of people that is
>> more
>>>>> knowledgeable about this and/or wants to help in this specific area?
>>>>> 
>>>>> Thank you,
>>>>> Mauro Giusti
>>>>> Azure Team, Microsoft.
>>>>> 
>>>> 
>>