[DISCUSS] Prioritizing works in progress

[Jungtaek Lim <ka...@gmail.com>]

Hi devs,

I'm seeing lots of huge works in parallel, and we individual are busy
regarding each work so common works (review, release, documentation, etc.)
have been not made in progress for several months.

- Beam runner
- Metrics renewal
- Java port
- Storm SQL improvement (Streaming SQL in future)
- Stream API

IMHO, it would be better to set target versions for them, and set a roadmap
(per version), and prioritize based on roadmap.

Stream API (very first version), and Storm SQL improvement are waiting for
review, and personally I would like to publish them soon.

If we're OK to have 2.0.0 without adding much features, I'm in favor of
concentrating Java port work (postponing other things except releasing 1.x
version line) and moving to Apache Storm 2.0.0 really soon.
(I'm even OK we decide to postpone some clojure files to be addressed after
2.0.0.)
Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8 (2.x) which
is other reason to move to 2.x quickly.

I'd be really happy if we have metrics renewal and beam runner, but I'm not
sure when they're available to be published. Do we have any updates here?

What do you think? It might be ideal, and/or broader discussion but we
haven't discussed our plan / vision for a long time so better to give it a
try.

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: [DISCUSS] Prioritizing works in progress

[Jungtaek Lim <ka...@gmail.com>]

Updates: I reviewed PR for Stream API and think it would be good start.
I would be really appreciated if some of us also participate reviewing as
well. Thanks in advance.

- Jungtaek Lim (HeartSaVioR)

2017년 1월 4일 (수) 오전 4:00, Hugo Da Cruz Louro <hl...@hortonworks.com>님이 작성:

> Lambdas can also lead to clean code reuse. It is part of evolving to adopt
> new JDK functionalities, and I think that we should do it.
>
> It is expected that contributors starting with a new technology/paradigm
> may not always implement code that follows perfect practices. However, we
> should look at that as an opportunity to have more experienced community
> members give constructive feedback through code reviews, and sharing good
> resources where one can master such best practices.
>
> I don’t think that a few less than ideal implementations should lead us
> not to use this powerful language feature.
>
> Hugo
>
>
> > On Jan 3, 2017, at 10:19 AM, S G <sg...@gmail.com> wrote:
> >
> > Yes, I agree.
> >
> > +1s on both these lines:
> >
> > *Badly designed APIs can get more complicated when combined with lambdas*
> > *Lambdas and functional style can greatly improve the readability if used
> > properly*.
> >
> >
> >
> > On Mon, Jan 2, 2017 at 9:34 PM, Arun Mahadevan <ar...@apache.org> wrote:
> >
> >> Its more about getting used to and judiciously deciding when to use
> >> functional v/s procedural approach. Badly designed APIs can get more
> >> complicated when combined with lambdas, but overall what I observe is
> >> lambdas and functional style can greatly improve the readability (with
> many
> >> other benefits) if used properly.
> >>
> >> A simple word count pipeline like below can get too verbose when
> expressed
> >> without lambdas.
> >>
> >> Stream<String> stream = …
> >> stream
> >> .flatMap(s -> Arrays.asList(s.split(" ")))
> >> .mapToPair(w -> Pair.of(w, 1))
> >> .countByKey()
> >> .filter(x -> x.getSecond() >= 5)
> >> .print();
> >>
> >> *Without lambda*
> >>
> >> stream
> >> .flatMap(new FlatMapFunction<String, String>() {
> >>  @Override
> >>  public Iterable<String> apply(String s) {
> >>    return Arrays.asList(s.split(" "));
> >>  }
> >> })
> >> .mapToPair(new PairFunction<String, String, Integer>() {
> >>  @Override
> >>  public Pair<String, Integer> apply(String w) {
> >>    return Pair.of(w, 1);
> >>  }
> >> })
> >> .countByKey()
> >> .filter(new Predicate<Pair<String, Long>>() {
> >>  @Override
> >>  public boolean test(Pair<String, Long> x) {
> >>    return x.getSecond() >= 5;
> >>  }
> >> })
> >> .print();
> >>
> >>
> >> Thanks,
> >> Arun
> >>
> >> On 1/3/17, 10:21 AM, "S G" <sg...@gmail.com> wrote:
> >>
> >>> One bad usage I have found is missing types for lambda arguments.
> >>> When the types for arguments are not provided in the code itself,
> github
> >> or
> >>> vi-editor browsing of such code becomes impossible.
> >>>
> >>> You must have an IDE like Eclipse to infer the types for such cases and
> >>> that becomes a big limitation to browse such code. I have not looked
> into
> >>> the storm code-base to find such occurrences but the above is just a
> >>> general observation from the lambda code usages.
> >>>
> >>>
> >>> Secondly, lambdas tend to generate bad stack-traces on exceptions.
> >>>
> >>>
> >>> And lastly, functional programming is not natural to many Java
> developers.
> >>> https://zeroturnaround.com/rebellabs/how-to-avoid-
> >> ruining-your-world-with-lambdas-in-java-8/
> >>> hints at some of these problems. Java code is usually very easy to
> >>> understand. But once you throw in more than a single level of lambdas,
> >>> things tend to become more and more confusing for several developers.
> >> (This
> >>> last point reflects the thoughts from some of my fellow developers, not
> >>> representative of all the devs)
> >>>
> >>>
> >>> Thanks
> >>> SG
> >>>
> >>>
> >>> On Mon, Jan 2, 2017 at 8:26 PM, Jungtaek Lim <ka...@gmail.com>
> wrote:
> >>>
> >>>> Hi S G,
> >>>>
> >>>> I'd be happy if you could elaborate your opinion. Did you found bad
> >> usages
> >>>> from master branch of Storm code?
> >>>>
> >>>> Regarding comfortable of lambda, IMHO, I don't think many users are
> >>>> unfamiliar with lambda, since they should have been used it with
> various
> >>>> languages. We might not be comfortable with Java 8 lambda (since
> >> transition
> >>>> to Java 8 is going slowly), but it's just a matter of familiarizing.
> >>>>
> >>>> Are there kind of best practices for Java 8 lambda? We can refer these
> >> to
> >>>> construct some guides / restrictions for Storm project.
> >>>>
> >>>> Thanks,
> >>>> Jungtaek Lim (HeartSaVioR)
> >>>>
> >>>> 2017년 1월 3일 (화) 오후 12:26, S G <sg...@gmail.com>님이 작성:
> >>>>
> >>>>> I have found several bad usages of Java 8 lambdas and many developers
> >> are
> >>>>> not comfortable using them.
> >>>>>
> >>>>> So we should use them only if it really makes the code beautiful and
> >>>> easier
> >>>>> to understand.
> >>>>>
> >>>>> My 2c,
> >>>>> Thanks
> >>>>> SG
> >>>>>
> >>>>>
> >>>>> On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> The streams API implementation has limited usage of 1.8 features and
> >>>> can
> >>>>>> be easily ported to 1.7 if required. The examples are written in
> >> 1.8,
> >>>> the
> >>>>>> thought being users would stick to the Java 8 style usage (lambdas)
> >>>> from
> >>>>>> the beginning. If there is consensus we could also consider moving
> >> the
> >>>>> 1.x
> >>>>>> branch to JDK 8.
> >>>>>>
> >>>>>> Anyways would like interested folks to start reviewing the changes
> >> so
> >>>>> that
> >>>>>> we can take it forward.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Arun
> >>>>>>
> >>>>>>
> >>>>>> On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>>>>>
> >>>>>>> FYI, I've realized that internal of Stream API (pull request)
> >> relies
> >>>> on
> >>>>>> JDK
> >>>>>>> 8 (what I've found is 'static method in interface' and maybe more)
> >> so
> >>>>> for
> >>>>>>> now Stream API is expected to be included for at least Storm 2.0.0
> >>>>> unless
> >>>>>>> the PR is modified to fit to JDK 7.
> >>>>>>>
> >>>>>>> - Jungtaek Lim (HeartSaVioR)
> >>>>>>>
> >>>>>>> 2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>>>>>
> >>>>>>>> Thanks Manu and Taylor for giving your opinions.
> >>>>>>>>
> >>>>>>>> - Storm SQL improvement
> >>>>>>>>
> >>>>>>>> There're some huge PRs available but there're all about
> >> improvement
> >>>>>> which
> >>>>>>>> shouldn't be blocker for releasing 1.1.0. (I'd like to also
> >> include
> >>>>>> them to
> >>>>>>>> 1.1.0 but not sure it can be happen really soon.)
> >>>>>>>> I'll send a request for reviewing about pending Storm SQL PRs.
> >>>>>>>>
> >>>>>>>> Only one issue (STORM-2200) is linked to release 1.1.0 epic
> >> which is
> >>>>>>>> blocker for me.
> >>>>>>>>
> >>>>>>>> - Java port
> >>>>>>>>
> >>>>>>>> I also had some developers saying 'If core of Storm were written
> >> by
> >>>>>> Java,
> >>>>>>>> I could experiment and even contribute on something'. I was one
> >> of
> >>>>> them,
> >>>>>>>> and to be honest, I'm still a beginner of Clojure. Moving to
> >> Java 8
> >>>>> also
> >>>>>>>> gives great functionalities for us, so Java port is what I think
> >> the
> >>>>>> most
> >>>>>>>> important thing among the huge works now in progress. Ideally,
> >> and
> >>>>>>>> hopefully, I'd like to see us focus on this and make this happen
> >> at
> >>>>> the
> >>>>>>>> very early next year.
> >>>>>>>> (Yes we should do some manual tests and maybe some refactoring
> >> too.)
> >>>>>>>>
> >>>>>>>> - Metrics V2
> >>>>>>>>
> >>>>>>>> I'm not sure when we plan to release Storm 1.2.0, but given that
> >>>>>> there're
> >>>>>>>> only two things left (logviewer / ui) for completing port work
> >>>> (except
> >>>>>>>> tests) I guess Storm 2.0.0 might be happen earlier.
> >>>>>>>> Taylor, when do you expect metrics V2 will be available for
> >>>> reviewing?
> >>>>>>>>
> >>>>>>>> - Stream API
> >>>>>>>>
> >>>>>>>> With labeling as experiment or annotating with evolving, we could
> >>>>>> include
> >>>>>>>> the first version to next minor excluding 1.1.0. (We could even
> >>>>> include
> >>>>>>>> this to 1.1.0 if we start reviewing this very soon.)
> >>>>>>>>
> >>>>>>>> I'd like to hear others' opinions as well.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>>>>
> >>>>>>>> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이
> >>>> 작성:
> >>>>>>>>
> >>>>>>>> Hi Jungtaek,
> >>>>>>>>
> >>>>>>>>> - Beam runner
> >>>>>>>>
> >>>>>>>> There’s not been much activity around this, and I haven’t had
> >> much
> >>>>> time
> >>>>>> to
> >>>>>>>> work on it recently, but there’s a decent foundation to build
> >> upon.
> >>>> So
> >>>>>> it
> >>>>>>>> would be fairly easy for others to start contributing to that
> >>>> effort.
> >>>>>>>> There’s also interest from the Beam community in that runner, so
> >> one
> >>>>>>>> possibility is to move that effort to the Apache Beam project.
> >>>>>>>>
> >>>>>>>> This is very preliminary work, so I don’t have a good handle on
> >> what
> >>>>> the
> >>>>>>>> target release would be.
> >>>>>>>>
> >>>>>>>>> - Metrics renewal
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> This is what I’ve been referring to as “metrics_v2”. This is
> >>>>> progressing
> >>>>>>>> fairly well with support for multiple reporters (e.g. Graphite,
> >>>>> Ganglia,
> >>>>>>>> console, etc.), worker metrics, disruptor metrics, etc.
> >>>>>>>>
> >>>>>>>> I would like to target this work for 1.2.0.
> >>>>>>>>
> >>>>>>>>> - Java port
> >>>>>>>>
> >>>>>>>> This effort seems to have picked up (for example Bobby’s
> >> conversion
> >>>> of
> >>>>>>>> Nimbus, etc.) and is progressing steadily. It’s taken a lot
> >> longer
> >>>>> than
> >>>>>>>> initially thought, but a lot of that can be attributed to the ebb
> >>>> and
> >>>>>> flow
> >>>>>>>> of people’s availability to do the work.
> >>>>>>>>
> >>>>>>>>> - Storm SQL improvement (Streaming SQL in future)
> >>>>>>>>
> >>>>>>>> You’ve been spearheading most of the work here, so I’d delegate
> >> to
> >>>> you
> >>>>>> for
> >>>>>>>> your opinion on where it stands. If you need additional reviews,
> >>>> just
> >>>>>> ask
> >>>>>>>> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject
> >> line
> >>>>> might
> >>>>>>>> help get attention).
> >>>>>>>>
> >>>>>>>> My thinking has been that this could be included in the 1.1.0
> >>>> release.
> >>>>>> Is
> >>>>>>>> there a set of JIRA issues you would like to include in order to
> >>>> make
> >>>>>> that
> >>>>>>>> happen?
> >>>>>>>>
> >>>>>>>>> - Stream API
> >>>>>>>>
> >>>>>>>> This seems to have stalled a bit, though there seems to be a lot
> >> of
> >>>>>>>> interest around it. I think we all would agree that when
> >>>> introducing a
> >>>>>> new
> >>>>>>>> API for building topologies, it’s important that we get right
> >> from
> >>>> the
> >>>>>>>> start and have strong buy-in from the development community. I
> >> would
> >>>>>>>> encourage anyone interested in the Streams API to review the
> >>>> proposal
> >>>>>> and
> >>>>>>>> initial code.
> >>>>>>>>
> >>>>>>>> I think it is close, but I’m not sure what release to target.
> >>>> Possibly
> >>>>>> the
> >>>>>>>> 2.0 release?
> >>>>>>>>
> >>>>>>>> Re: 1.1.0 Release
> >>>>>>>>
> >>>>>>>> STORM-2176 is a fairly big concern of mine since the feature it
> >>>>> involves
> >>>>>>>> was introduced in 1.0.0 and did not work then nor in any
> >> subsequent
> >>>> or
> >>>>>>>> future releases (may not be a problem in 2.0). Unfortunately, as
> >>>>> you’ve
> >>>>>>>> seen, finding the root cause is elusive. That issue could
> >> definitely
> >>>>> use
> >>>>>>>> more eyes.
> >>>>>>>>
> >>>>>>>> -Taylor
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi devs,
> >>>>>>>>>
> >>>>>>>>> I'm seeing lots of huge works in parallel, and we individual
> >> are
> >>>>> busy
> >>>>>>>>> regarding each work so common works (review, release,
> >>>> documentation,
> >>>>>>>> etc.)
> >>>>>>>>> have been not made in progress for several months.
> >>>>>>>>>
> >>>>>>>>> - Beam runner
> >>>>>>>>> - Metrics renewal
> >>>>>>>>> - Java port
> >>>>>>>>> - Storm SQL improvement (Streaming SQL in future)
> >>>>>>>>> - Stream API
> >>>>>>>>>
> >>>>>>>>> IMHO, it would be better to set target versions for them, and
> >> set
> >>>> a
> >>>>>>>> roadmap
> >>>>>>>>> (per version), and prioritize based on roadmap.
> >>>>>>>>>
> >>>>>>>>> Stream API (very first version), and Storm SQL improvement are
> >>>>> waiting
> >>>>>>>> for
> >>>>>>>>> review, and personally I would like to publish them soon.
> >>>>>>>>>
> >>>>>>>>> If we're OK to have 2.0.0 without adding much features, I'm in
> >>>> favor
> >>>>>> of
> >>>>>>>>> concentrating Java port work (postponing other things except
> >>>>> releasing
> >>>>>>>> 1.x
> >>>>>>>>> version line) and moving to Apache Storm 2.0.0 really soon.
> >>>>>>>>> (I'm even OK we decide to postpone some clojure files to be
> >>>>> addressed
> >>>>>>>> after
> >>>>>>>>> 2.0.0.)
> >>>>>>>>> Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8
> >>>>> (2.x)
> >>>>>>>> which
> >>>>>>>>> is other reason to move to 2.x quickly.
> >>>>>>>>>
> >>>>>>>>> I'd be really happy if we have metrics renewal and beam runner,
> >>>> but
> >>>>>> I'm
> >>>>>>>> not
> >>>>>>>>> sure when they're available to be published. Do we have any
> >>>> updates
> >>>>>> here?
> >>>>>>>>>
> >>>>>>>>> What do you think? It might be ideal, and/or broader discussion
> >>>> but
> >>>>> we
> >>>>>>>>> haven't discussed our plan / vision for a long time so better
> >> to
> >>>>> give
> >>>>>> it
> >>>>>>>> a
> >>>>>>>>> try.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
> >>
>
>

Re: [DISCUSS] Prioritizing works in progress

[Hugo Da Cruz Louro <hl...@hortonworks.com>]

Lambdas can also lead to clean code reuse. It is part of evolving to adopt new JDK functionalities, and I think that we should do it.

It is expected that contributors starting with a new technology/paradigm may not always implement code that follows perfect practices. However, we should look at that as an opportunity to have more experienced community members give constructive feedback through code reviews, and sharing good resources where one can master such best practices. 

I don’t think that a few less than ideal implementations should lead us not to use this powerful language feature.

Hugo


> On Jan 3, 2017, at 10:19 AM, S G <sg...@gmail.com> wrote:
> 
> Yes, I agree.
> 
> +1s on both these lines:
> 
> *Badly designed APIs can get more complicated when combined with lambdas*
> *Lambdas and functional style can greatly improve the readability if used
> properly*.
> 
> 
> 
> On Mon, Jan 2, 2017 at 9:34 PM, Arun Mahadevan <ar...@apache.org> wrote:
> 
>> Its more about getting used to and judiciously deciding when to use
>> functional v/s procedural approach. Badly designed APIs can get more
>> complicated when combined with lambdas, but overall what I observe is
>> lambdas and functional style can greatly improve the readability (with many
>> other benefits) if used properly.
>> 
>> A simple word count pipeline like below can get too verbose when expressed
>> without lambdas.
>> 
>> Stream<String> stream = …
>> stream
>> .flatMap(s -> Arrays.asList(s.split(" ")))
>> .mapToPair(w -> Pair.of(w, 1))
>> .countByKey()
>> .filter(x -> x.getSecond() >= 5)
>> .print();
>> 
>> *Without lambda*
>> 
>> stream
>> .flatMap(new FlatMapFunction<String, String>() {
>>  @Override
>>  public Iterable<String> apply(String s) {
>>    return Arrays.asList(s.split(" "));
>>  }
>> })
>> .mapToPair(new PairFunction<String, String, Integer>() {
>>  @Override
>>  public Pair<String, Integer> apply(String w) {
>>    return Pair.of(w, 1);
>>  }
>> })
>> .countByKey()
>> .filter(new Predicate<Pair<String, Long>>() {
>>  @Override
>>  public boolean test(Pair<String, Long> x) {
>>    return x.getSecond() >= 5;
>>  }
>> })
>> .print();
>> 
>> 
>> Thanks,
>> Arun
>> 
>> On 1/3/17, 10:21 AM, "S G" <sg...@gmail.com> wrote:
>> 
>>> One bad usage I have found is missing types for lambda arguments.
>>> When the types for arguments are not provided in the code itself, github
>> or
>>> vi-editor browsing of such code becomes impossible.
>>> 
>>> You must have an IDE like Eclipse to infer the types for such cases and
>>> that becomes a big limitation to browse such code. I have not looked into
>>> the storm code-base to find such occurrences but the above is just a
>>> general observation from the lambda code usages.
>>> 
>>> 
>>> Secondly, lambdas tend to generate bad stack-traces on exceptions.
>>> 
>>> 
>>> And lastly, functional programming is not natural to many Java developers.
>>> https://zeroturnaround.com/rebellabs/how-to-avoid-
>> ruining-your-world-with-lambdas-in-java-8/
>>> hints at some of these problems. Java code is usually very easy to
>>> understand. But once you throw in more than a single level of lambdas,
>>> things tend to become more and more confusing for several developers.
>> (This
>>> last point reflects the thoughts from some of my fellow developers, not
>>> representative of all the devs)
>>> 
>>> 
>>> Thanks
>>> SG
>>> 
>>> 
>>> On Mon, Jan 2, 2017 at 8:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>>> 
>>>> Hi S G,
>>>> 
>>>> I'd be happy if you could elaborate your opinion. Did you found bad
>> usages
>>>> from master branch of Storm code?
>>>> 
>>>> Regarding comfortable of lambda, IMHO, I don't think many users are
>>>> unfamiliar with lambda, since they should have been used it with various
>>>> languages. We might not be comfortable with Java 8 lambda (since
>> transition
>>>> to Java 8 is going slowly), but it's just a matter of familiarizing.
>>>> 
>>>> Are there kind of best practices for Java 8 lambda? We can refer these
>> to
>>>> construct some guides / restrictions for Storm project.
>>>> 
>>>> Thanks,
>>>> Jungtaek Lim (HeartSaVioR)
>>>> 
>>>> 2017년 1월 3일 (화) 오후 12:26, S G <sg...@gmail.com>님이 작성:
>>>> 
>>>>> I have found several bad usages of Java 8 lambdas and many developers
>> are
>>>>> not comfortable using them.
>>>>> 
>>>>> So we should use them only if it really makes the code beautiful and
>>>> easier
>>>>> to understand.
>>>>> 
>>>>> My 2c,
>>>>> Thanks
>>>>> SG
>>>>> 
>>>>> 
>>>>> On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org>
>>>> wrote:
>>>>> 
>>>>>> The streams API implementation has limited usage of 1.8 features and
>>>> can
>>>>>> be easily ported to 1.7 if required. The examples are written in
>> 1.8,
>>>> the
>>>>>> thought being users would stick to the Java 8 style usage (lambdas)
>>>> from
>>>>>> the beginning. If there is consensus we could also consider moving
>> the
>>>>> 1.x
>>>>>> branch to JDK 8.
>>>>>> 
>>>>>> Anyways would like interested folks to start reviewing the changes
>> so
>>>>> that
>>>>>> we can take it forward.
>>>>>> 
>>>>>> Thanks,
>>>>>> Arun
>>>>>> 
>>>>>> 
>>>>>> On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>> 
>>>>>>> FYI, I've realized that internal of Stream API (pull request)
>> relies
>>>> on
>>>>>> JDK
>>>>>>> 8 (what I've found is 'static method in interface' and maybe more)
>> so
>>>>> for
>>>>>>> now Stream API is expected to be included for at least Storm 2.0.0
>>>>> unless
>>>>>>> the PR is modified to fit to JDK 7.
>>>>>>> 
>>>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>>> 
>>>>>>> 2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>>>> 
>>>>>>>> Thanks Manu and Taylor for giving your opinions.
>>>>>>>> 
>>>>>>>> - Storm SQL improvement
>>>>>>>> 
>>>>>>>> There're some huge PRs available but there're all about
>> improvement
>>>>>> which
>>>>>>>> shouldn't be blocker for releasing 1.1.0. (I'd like to also
>> include
>>>>>> them to
>>>>>>>> 1.1.0 but not sure it can be happen really soon.)
>>>>>>>> I'll send a request for reviewing about pending Storm SQL PRs.
>>>>>>>> 
>>>>>>>> Only one issue (STORM-2200) is linked to release 1.1.0 epic
>> which is
>>>>>>>> blocker for me.
>>>>>>>> 
>>>>>>>> - Java port
>>>>>>>> 
>>>>>>>> I also had some developers saying 'If core of Storm were written
>> by
>>>>>> Java,
>>>>>>>> I could experiment and even contribute on something'. I was one
>> of
>>>>> them,
>>>>>>>> and to be honest, I'm still a beginner of Clojure. Moving to
>> Java 8
>>>>> also
>>>>>>>> gives great functionalities for us, so Java port is what I think
>> the
>>>>>> most
>>>>>>>> important thing among the huge works now in progress. Ideally,
>> and
>>>>>>>> hopefully, I'd like to see us focus on this and make this happen
>> at
>>>>> the
>>>>>>>> very early next year.
>>>>>>>> (Yes we should do some manual tests and maybe some refactoring
>> too.)
>>>>>>>> 
>>>>>>>> - Metrics V2
>>>>>>>> 
>>>>>>>> I'm not sure when we plan to release Storm 1.2.0, but given that
>>>>>> there're
>>>>>>>> only two things left (logviewer / ui) for completing port work
>>>> (except
>>>>>>>> tests) I guess Storm 2.0.0 might be happen earlier.
>>>>>>>> Taylor, when do you expect metrics V2 will be available for
>>>> reviewing?
>>>>>>>> 
>>>>>>>> - Stream API
>>>>>>>> 
>>>>>>>> With labeling as experiment or annotating with evolving, we could
>>>>>> include
>>>>>>>> the first version to next minor excluding 1.1.0. (We could even
>>>>> include
>>>>>>>> this to 1.1.0 if we start reviewing this very soon.)
>>>>>>>> 
>>>>>>>> I'd like to hear others' opinions as well.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>> 
>>>>>>>> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이
>>>> 작성:
>>>>>>>> 
>>>>>>>> Hi Jungtaek,
>>>>>>>> 
>>>>>>>>> - Beam runner
>>>>>>>> 
>>>>>>>> There’s not been much activity around this, and I haven’t had
>> much
>>>>> time
>>>>>> to
>>>>>>>> work on it recently, but there’s a decent foundation to build
>> upon.
>>>> So
>>>>>> it
>>>>>>>> would be fairly easy for others to start contributing to that
>>>> effort.
>>>>>>>> There’s also interest from the Beam community in that runner, so
>> one
>>>>>>>> possibility is to move that effort to the Apache Beam project.
>>>>>>>> 
>>>>>>>> This is very preliminary work, so I don’t have a good handle on
>> what
>>>>> the
>>>>>>>> target release would be.
>>>>>>>> 
>>>>>>>>> - Metrics renewal
>>>>>>>> 
>>>>>>>> 
>>>>>>>> This is what I’ve been referring to as “metrics_v2”. This is
>>>>> progressing
>>>>>>>> fairly well with support for multiple reporters (e.g. Graphite,
>>>>> Ganglia,
>>>>>>>> console, etc.), worker metrics, disruptor metrics, etc.
>>>>>>>> 
>>>>>>>> I would like to target this work for 1.2.0.
>>>>>>>> 
>>>>>>>>> - Java port
>>>>>>>> 
>>>>>>>> This effort seems to have picked up (for example Bobby’s
>> conversion
>>>> of
>>>>>>>> Nimbus, etc.) and is progressing steadily. It’s taken a lot
>> longer
>>>>> than
>>>>>>>> initially thought, but a lot of that can be attributed to the ebb
>>>> and
>>>>>> flow
>>>>>>>> of people’s availability to do the work.
>>>>>>>> 
>>>>>>>>> - Storm SQL improvement (Streaming SQL in future)
>>>>>>>> 
>>>>>>>> You’ve been spearheading most of the work here, so I’d delegate
>> to
>>>> you
>>>>>> for
>>>>>>>> your opinion on where it stands. If you need additional reviews,
>>>> just
>>>>>> ask
>>>>>>>> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject
>> line
>>>>> might
>>>>>>>> help get attention).
>>>>>>>> 
>>>>>>>> My thinking has been that this could be included in the 1.1.0
>>>> release.
>>>>>> Is
>>>>>>>> there a set of JIRA issues you would like to include in order to
>>>> make
>>>>>> that
>>>>>>>> happen?
>>>>>>>> 
>>>>>>>>> - Stream API
>>>>>>>> 
>>>>>>>> This seems to have stalled a bit, though there seems to be a lot
>> of
>>>>>>>> interest around it. I think we all would agree that when
>>>> introducing a
>>>>>> new
>>>>>>>> API for building topologies, it’s important that we get right
>> from
>>>> the
>>>>>>>> start and have strong buy-in from the development community. I
>> would
>>>>>>>> encourage anyone interested in the Streams API to review the
>>>> proposal
>>>>>> and
>>>>>>>> initial code.
>>>>>>>> 
>>>>>>>> I think it is close, but I’m not sure what release to target.
>>>> Possibly
>>>>>> the
>>>>>>>> 2.0 release?
>>>>>>>> 
>>>>>>>> Re: 1.1.0 Release
>>>>>>>> 
>>>>>>>> STORM-2176 is a fairly big concern of mine since the feature it
>>>>> involves
>>>>>>>> was introduced in 1.0.0 and did not work then nor in any
>> subsequent
>>>> or
>>>>>>>> future releases (may not be a problem in 2.0). Unfortunately, as
>>>>> you’ve
>>>>>>>> seen, finding the root cause is elusive. That issue could
>> definitely
>>>>> use
>>>>>>>> more eyes.
>>>>>>>> 
>>>>>>>> -Taylor
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com>
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi devs,
>>>>>>>>> 
>>>>>>>>> I'm seeing lots of huge works in parallel, and we individual
>> are
>>>>> busy
>>>>>>>>> regarding each work so common works (review, release,
>>>> documentation,
>>>>>>>> etc.)
>>>>>>>>> have been not made in progress for several months.
>>>>>>>>> 
>>>>>>>>> - Beam runner
>>>>>>>>> - Metrics renewal
>>>>>>>>> - Java port
>>>>>>>>> - Storm SQL improvement (Streaming SQL in future)
>>>>>>>>> - Stream API
>>>>>>>>> 
>>>>>>>>> IMHO, it would be better to set target versions for them, and
>> set
>>>> a
>>>>>>>> roadmap
>>>>>>>>> (per version), and prioritize based on roadmap.
>>>>>>>>> 
>>>>>>>>> Stream API (very first version), and Storm SQL improvement are
>>>>> waiting
>>>>>>>> for
>>>>>>>>> review, and personally I would like to publish them soon.
>>>>>>>>> 
>>>>>>>>> If we're OK to have 2.0.0 without adding much features, I'm in
>>>> favor
>>>>>> of
>>>>>>>>> concentrating Java port work (postponing other things except
>>>>> releasing
>>>>>>>> 1.x
>>>>>>>>> version line) and moving to Apache Storm 2.0.0 really soon.
>>>>>>>>> (I'm even OK we decide to postpone some clojure files to be
>>>>> addressed
>>>>>>>> after
>>>>>>>>> 2.0.0.)
>>>>>>>>> Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8
>>>>> (2.x)
>>>>>>>> which
>>>>>>>>> is other reason to move to 2.x quickly.
>>>>>>>>> 
>>>>>>>>> I'd be really happy if we have metrics renewal and beam runner,
>>>> but
>>>>>> I'm
>>>>>>>> not
>>>>>>>>> sure when they're available to be published. Do we have any
>>>> updates
>>>>>> here?
>>>>>>>>> 
>>>>>>>>> What do you think? It might be ideal, and/or broader discussion
>>>> but
>>>>> we
>>>>>>>>> haven't discussed our plan / vision for a long time so better
>> to
>>>>> give
>>>>>> it
>>>>>>>> a
>>>>>>>>> try.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 
>> 

Re: [DISCUSS] Prioritizing works in progress

[S G <sg...@gmail.com>]

Yes, I agree.

+1s on both these lines:

*Badly designed APIs can get more complicated when combined with lambdas*
*Lambdas and functional style can greatly improve the readability if used
properly*.



On Mon, Jan 2, 2017 at 9:34 PM, Arun Mahadevan <ar...@apache.org> wrote:

> Its more about getting used to and judiciously deciding when to use
> functional v/s procedural approach. Badly designed APIs can get more
> complicated when combined with lambdas, but overall what I observe is
> lambdas and functional style can greatly improve the readability (with many
> other benefits) if used properly.
>
> A simple word count pipeline like below can get too verbose when expressed
> without lambdas.
>
> Stream<String> stream = …
> stream
> .flatMap(s -> Arrays.asList(s.split(" ")))
> .mapToPair(w -> Pair.of(w, 1))
> .countByKey()
> .filter(x -> x.getSecond() >= 5)
> .print();
>
> *Without lambda*
>
> stream
> .flatMap(new FlatMapFunction<String, String>() {
>   @Override
>   public Iterable<String> apply(String s) {
>     return Arrays.asList(s.split(" "));
>   }
> })
> .mapToPair(new PairFunction<String, String, Integer>() {
>   @Override
>   public Pair<String, Integer> apply(String w) {
>     return Pair.of(w, 1);
>   }
> })
> .countByKey()
> .filter(new Predicate<Pair<String, Long>>() {
>   @Override
>   public boolean test(Pair<String, Long> x) {
>     return x.getSecond() >= 5;
>   }
> })
> .print();
>
>
> Thanks,
> Arun
>
> On 1/3/17, 10:21 AM, "S G" <sg...@gmail.com> wrote:
>
> >One bad usage I have found is missing types for lambda arguments.
> >When the types for arguments are not provided in the code itself, github
> or
> >vi-editor browsing of such code becomes impossible.
> >
> >You must have an IDE like Eclipse to infer the types for such cases and
> >that becomes a big limitation to browse such code. I have not looked into
> >the storm code-base to find such occurrences but the above is just a
> >general observation from the lambda code usages.
> >
> >
> >Secondly, lambdas tend to generate bad stack-traces on exceptions.
> >
> >
> >And lastly, functional programming is not natural to many Java developers.
> >https://zeroturnaround.com/rebellabs/how-to-avoid-
> ruining-your-world-with-lambdas-in-java-8/
> >hints at some of these problems. Java code is usually very easy to
> >understand. But once you throw in more than a single level of lambdas,
> >things tend to become more and more confusing for several developers.
> (This
> >last point reflects the thoughts from some of my fellow developers, not
> >representative of all the devs)
> >
> >
> >Thanks
> >SG
> >
> >
> >On Mon, Jan 2, 2017 at 8:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> >
> >> Hi S G,
> >>
> >> I'd be happy if you could elaborate your opinion. Did you found bad
> usages
> >> from master branch of Storm code?
> >>
> >> Regarding comfortable of lambda, IMHO, I don't think many users are
> >> unfamiliar with lambda, since they should have been used it with various
> >> languages. We might not be comfortable with Java 8 lambda (since
> transition
> >> to Java 8 is going slowly), but it's just a matter of familiarizing.
> >>
> >> Are there kind of best practices for Java 8 lambda? We can refer these
> to
> >> construct some guides / restrictions for Storm project.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 1월 3일 (화) 오후 12:26, S G <sg...@gmail.com>님이 작성:
> >>
> >> > I have found several bad usages of Java 8 lambdas and many developers
> are
> >> > not comfortable using them.
> >> >
> >> > So we should use them only if it really makes the code beautiful and
> >> easier
> >> > to understand.
> >> >
> >> > My 2c,
> >> > Thanks
> >> > SG
> >> >
> >> >
> >> > On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org>
> >> wrote:
> >> >
> >> > > The streams API implementation has limited usage of 1.8 features and
> >> can
> >> > > be easily ported to 1.7 if required. The examples are written in
> 1.8,
> >> the
> >> > > thought being users would stick to the Java 8 style usage (lambdas)
> >> from
> >> > > the beginning. If there is consensus we could also consider moving
> the
> >> > 1.x
> >> > > branch to JDK 8.
> >> > >
> >> > > Anyways would like interested folks to start reviewing the changes
> so
> >> > that
> >> > > we can take it forward.
> >> > >
> >> > > Thanks,
> >> > > Arun
> >> > >
> >> > >
> >> > > On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >> > >
> >> > > >FYI, I've realized that internal of Stream API (pull request)
> relies
> >> on
> >> > > JDK
> >> > > >8 (what I've found is 'static method in interface' and maybe more)
> so
> >> > for
> >> > > >now Stream API is expected to be included for at least Storm 2.0.0
> >> > unless
> >> > > >the PR is modified to fit to JDK 7.
> >> > > >
> >> > > >- Jungtaek Lim (HeartSaVioR)
> >> > > >
> >> > > >2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >> > > >
> >> > > >> Thanks Manu and Taylor for giving your opinions.
> >> > > >>
> >> > > >> - Storm SQL improvement
> >> > > >>
> >> > > >> There're some huge PRs available but there're all about
> improvement
> >> > > which
> >> > > >> shouldn't be blocker for releasing 1.1.0. (I'd like to also
> include
> >> > > them to
> >> > > >> 1.1.0 but not sure it can be happen really soon.)
> >> > > >> I'll send a request for reviewing about pending Storm SQL PRs.
> >> > > >>
> >> > > >> Only one issue (STORM-2200) is linked to release 1.1.0 epic
> which is
> >> > > >> blocker for me.
> >> > > >>
> >> > > >> - Java port
> >> > > >>
> >> > > >> I also had some developers saying 'If core of Storm were written
> by
> >> > > Java,
> >> > > >> I could experiment and even contribute on something'. I was one
> of
> >> > them,
> >> > > >> and to be honest, I'm still a beginner of Clojure. Moving to
> Java 8
> >> > also
> >> > > >> gives great functionalities for us, so Java port is what I think
> the
> >> > > most
> >> > > >> important thing among the huge works now in progress. Ideally,
> and
> >> > > >> hopefully, I'd like to see us focus on this and make this happen
> at
> >> > the
> >> > > >> very early next year.
> >> > > >> (Yes we should do some manual tests and maybe some refactoring
> too.)
> >> > > >>
> >> > > >> - Metrics V2
> >> > > >>
> >> > > >> I'm not sure when we plan to release Storm 1.2.0, but given that
> >> > > there're
> >> > > >> only two things left (logviewer / ui) for completing port work
> >> (except
> >> > > >> tests) I guess Storm 2.0.0 might be happen earlier.
> >> > > >> Taylor, when do you expect metrics V2 will be available for
> >> reviewing?
> >> > > >>
> >> > > >> - Stream API
> >> > > >>
> >> > > >> With labeling as experiment or annotating with evolving, we could
> >> > > include
> >> > > >> the first version to next minor excluding 1.1.0. (We could even
> >> > include
> >> > > >> this to 1.1.0 if we start reviewing this very soon.)
> >> > > >>
> >> > > >> I'd like to hear others' opinions as well.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Jungtaek Lim (HeartSaVioR)
> >> > > >>
> >> > > >> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이
> >> 작성:
> >> > > >>
> >> > > >> Hi Jungtaek,
> >> > > >>
> >> > > >> > - Beam runner
> >> > > >>
> >> > > >> There’s not been much activity around this, and I haven’t had
> much
> >> > time
> >> > > to
> >> > > >> work on it recently, but there’s a decent foundation to build
> upon.
> >> So
> >> > > it
> >> > > >> would be fairly easy for others to start contributing to that
> >> effort.
> >> > > >> There’s also interest from the Beam community in that runner, so
> one
> >> > > >> possibility is to move that effort to the Apache Beam project.
> >> > > >>
> >> > > >> This is very preliminary work, so I don’t have a good handle on
> what
> >> > the
> >> > > >> target release would be.
> >> > > >>
> >> > > >> > - Metrics renewal
> >> > > >>
> >> > > >>
> >> > > >> This is what I’ve been referring to as “metrics_v2”. This is
> >> > progressing
> >> > > >> fairly well with support for multiple reporters (e.g. Graphite,
> >> > Ganglia,
> >> > > >> console, etc.), worker metrics, disruptor metrics, etc.
> >> > > >>
> >> > > >> I would like to target this work for 1.2.0.
> >> > > >>
> >> > > >> > - Java port
> >> > > >>
> >> > > >> This effort seems to have picked up (for example Bobby’s
> conversion
> >> of
> >> > > >> Nimbus, etc.) and is progressing steadily. It’s taken a lot
> longer
> >> > than
> >> > > >> initially thought, but a lot of that can be attributed to the ebb
> >> and
> >> > > flow
> >> > > >> of people’s availability to do the work.
> >> > > >>
> >> > > >> > - Storm SQL improvement (Streaming SQL in future)
> >> > > >>
> >> > > >> You’ve been spearheading most of the work here, so I’d delegate
> to
> >> you
> >> > > for
> >> > > >> your opinion on where it stands. If you need additional reviews,
> >> just
> >> > > ask
> >> > > >> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject
> line
> >> > might
> >> > > >> help get attention).
> >> > > >>
> >> > > >> My thinking has been that this could be included in the 1.1.0
> >> release.
> >> > > Is
> >> > > >> there a set of JIRA issues you would like to include in order to
> >> make
> >> > > that
> >> > > >> happen?
> >> > > >>
> >> > > >> > - Stream API
> >> > > >>
> >> > > >> This seems to have stalled a bit, though there seems to be a lot
> of
> >> > > >> interest around it. I think we all would agree that when
> >> introducing a
> >> > > new
> >> > > >> API for building topologies, it’s important that we get right
> from
> >> the
> >> > > >> start and have strong buy-in from the development community. I
> would
> >> > > >> encourage anyone interested in the Streams API to review the
> >> proposal
> >> > > and
> >> > > >> initial code.
> >> > > >>
> >> > > >> I think it is close, but I’m not sure what release to target.
> >> Possibly
> >> > > the
> >> > > >> 2.0 release?
> >> > > >>
> >> > > >> Re: 1.1.0 Release
> >> > > >>
> >> > > >> STORM-2176 is a fairly big concern of mine since the feature it
> >> > involves
> >> > > >> was introduced in 1.0.0 and did not work then nor in any
> subsequent
> >> or
> >> > > >> future releases (may not be a problem in 2.0). Unfortunately, as
> >> > you’ve
> >> > > >> seen, finding the root cause is elusive. That issue could
> definitely
> >> > use
> >> > > >> more eyes.
> >> > > >>
> >> > > >> -Taylor
> >> > > >>
> >> > > >>
> >> > > >> > On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com>
> >> > wrote:
> >> > > >> >
> >> > > >> > Hi devs,
> >> > > >> >
> >> > > >> > I'm seeing lots of huge works in parallel, and we individual
> are
> >> > busy
> >> > > >> > regarding each work so common works (review, release,
> >> documentation,
> >> > > >> etc.)
> >> > > >> > have been not made in progress for several months.
> >> > > >> >
> >> > > >> > - Beam runner
> >> > > >> > - Metrics renewal
> >> > > >> > - Java port
> >> > > >> > - Storm SQL improvement (Streaming SQL in future)
> >> > > >> > - Stream API
> >> > > >> >
> >> > > >> > IMHO, it would be better to set target versions for them, and
> set
> >> a
> >> > > >> roadmap
> >> > > >> > (per version), and prioritize based on roadmap.
> >> > > >> >
> >> > > >> > Stream API (very first version), and Storm SQL improvement are
> >> > waiting
> >> > > >> for
> >> > > >> > review, and personally I would like to publish them soon.
> >> > > >> >
> >> > > >> > If we're OK to have 2.0.0 without adding much features, I'm in
> >> favor
> >> > > of
> >> > > >> > concentrating Java port work (postponing other things except
> >> > releasing
> >> > > >> 1.x
> >> > > >> > version line) and moving to Apache Storm 2.0.0 really soon.
> >> > > >> > (I'm even OK we decide to postpone some clojure files to be
> >> > addressed
> >> > > >> after
> >> > > >> > 2.0.0.)
> >> > > >> > Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8
> >> > (2.x)
> >> > > >> which
> >> > > >> > is other reason to move to 2.x quickly.
> >> > > >> >
> >> > > >> > I'd be really happy if we have metrics renewal and beam runner,
> >> but
> >> > > I'm
> >> > > >> not
> >> > > >> > sure when they're available to be published. Do we have any
> >> updates
> >> > > here?
> >> > > >> >
> >> > > >> > What do you think? It might be ideal, and/or broader discussion
> >> but
> >> > we
> >> > > >> > haven't discussed our plan / vision for a long time so better
> to
> >> > give
> >> > > it
> >> > > >> a
> >> > > >> > try.
> >> > > >> >
> >> > > >> > Thanks,
> >> > > >> > Jungtaek Lim (HeartSaVioR)
> >> > > >>
> >> > > >>
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
>
>

Re: [DISCUSS] Prioritizing works in progress

[Arun Mahadevan <ar...@apache.org>]

Its more about getting used to and judiciously deciding when to use functional v/s procedural approach. Badly designed APIs can get more complicated when combined with lambdas, but overall what I observe is lambdas and functional style can greatly improve the readability (with many other benefits) if used properly.

A simple word count pipeline like below can get too verbose when expressed without lambdas.

Stream<String> stream = …
stream
.flatMap(s -> Arrays.asList(s.split(" ")))
.mapToPair(w -> Pair.of(w, 1))
.countByKey()
.filter(x -> x.getSecond() >= 5)
.print();

*Without lambda*

stream
.flatMap(new FlatMapFunction<String, String>() {
  @Override
  public Iterable<String> apply(String s) {
    return Arrays.asList(s.split(" "));
  }
})
.mapToPair(new PairFunction<String, String, Integer>() {
  @Override
  public Pair<String, Integer> apply(String w) {
    return Pair.of(w, 1);
  }
})
.countByKey()
.filter(new Predicate<Pair<String, Long>>() {
  @Override
  public boolean test(Pair<String, Long> x) {
    return x.getSecond() >= 5;
  }
})
.print();


Thanks,
Arun

On 1/3/17, 10:21 AM, "S G" <sg...@gmail.com> wrote:

>One bad usage I have found is missing types for lambda arguments.
>When the types for arguments are not provided in the code itself, github or
>vi-editor browsing of such code becomes impossible.
>
>You must have an IDE like Eclipse to infer the types for such cases and
>that becomes a big limitation to browse such code. I have not looked into
>the storm code-base to find such occurrences but the above is just a
>general observation from the lambda code usages.
>
>
>Secondly, lambdas tend to generate bad stack-traces on exceptions.
>
>
>And lastly, functional programming is not natural to many Java developers.
>https://zeroturnaround.com/rebellabs/how-to-avoid-ruining-your-world-with-lambdas-in-java-8/
>hints at some of these problems. Java code is usually very easy to
>understand. But once you throw in more than a single level of lambdas,
>things tend to become more and more confusing for several developers. (This
>last point reflects the thoughts from some of my fellow developers, not
>representative of all the devs)
>
>
>Thanks
>SG
>
>
>On Mon, Jan 2, 2017 at 8:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
>> Hi S G,
>>
>> I'd be happy if you could elaborate your opinion. Did you found bad usages
>> from master branch of Storm code?
>>
>> Regarding comfortable of lambda, IMHO, I don't think many users are
>> unfamiliar with lambda, since they should have been used it with various
>> languages. We might not be comfortable with Java 8 lambda (since transition
>> to Java 8 is going slowly), but it's just a matter of familiarizing.
>>
>> Are there kind of best practices for Java 8 lambda? We can refer these to
>> construct some guides / restrictions for Storm project.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2017년 1월 3일 (화) 오후 12:26, S G <sg...@gmail.com>님이 작성:
>>
>> > I have found several bad usages of Java 8 lambdas and many developers are
>> > not comfortable using them.
>> >
>> > So we should use them only if it really makes the code beautiful and
>> easier
>> > to understand.
>> >
>> > My 2c,
>> > Thanks
>> > SG
>> >
>> >
>> > On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org>
>> wrote:
>> >
>> > > The streams API implementation has limited usage of 1.8 features and
>> can
>> > > be easily ported to 1.7 if required. The examples are written in 1.8,
>> the
>> > > thought being users would stick to the Java 8 style usage (lambdas)
>> from
>> > > the beginning. If there is consensus we could also consider moving the
>> > 1.x
>> > > branch to JDK 8.
>> > >
>> > > Anyways would like interested folks to start reviewing the changes so
>> > that
>> > > we can take it forward.
>> > >
>> > > Thanks,
>> > > Arun
>> > >
>> > >
>> > > On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>> > >
>> > > >FYI, I've realized that internal of Stream API (pull request) relies
>> on
>> > > JDK
>> > > >8 (what I've found is 'static method in interface' and maybe more) so
>> > for
>> > > >now Stream API is expected to be included for at least Storm 2.0.0
>> > unless
>> > > >the PR is modified to fit to JDK 7.
>> > > >
>> > > >- Jungtaek Lim (HeartSaVioR)
>> > > >
>> > > >2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
>> > > >
>> > > >> Thanks Manu and Taylor for giving your opinions.
>> > > >>
>> > > >> - Storm SQL improvement
>> > > >>
>> > > >> There're some huge PRs available but there're all about improvement
>> > > which
>> > > >> shouldn't be blocker for releasing 1.1.0. (I'd like to also include
>> > > them to
>> > > >> 1.1.0 but not sure it can be happen really soon.)
>> > > >> I'll send a request for reviewing about pending Storm SQL PRs.
>> > > >>
>> > > >> Only one issue (STORM-2200) is linked to release 1.1.0 epic which is
>> > > >> blocker for me.
>> > > >>
>> > > >> - Java port
>> > > >>
>> > > >> I also had some developers saying 'If core of Storm were written by
>> > > Java,
>> > > >> I could experiment and even contribute on something'. I was one of
>> > them,
>> > > >> and to be honest, I'm still a beginner of Clojure. Moving to Java 8
>> > also
>> > > >> gives great functionalities for us, so Java port is what I think the
>> > > most
>> > > >> important thing among the huge works now in progress. Ideally, and
>> > > >> hopefully, I'd like to see us focus on this and make this happen at
>> > the
>> > > >> very early next year.
>> > > >> (Yes we should do some manual tests and maybe some refactoring too.)
>> > > >>
>> > > >> - Metrics V2
>> > > >>
>> > > >> I'm not sure when we plan to release Storm 1.2.0, but given that
>> > > there're
>> > > >> only two things left (logviewer / ui) for completing port work
>> (except
>> > > >> tests) I guess Storm 2.0.0 might be happen earlier.
>> > > >> Taylor, when do you expect metrics V2 will be available for
>> reviewing?
>> > > >>
>> > > >> - Stream API
>> > > >>
>> > > >> With labeling as experiment or annotating with evolving, we could
>> > > include
>> > > >> the first version to next minor excluding 1.1.0. (We could even
>> > include
>> > > >> this to 1.1.0 if we start reviewing this very soon.)
>> > > >>
>> > > >> I'd like to hear others' opinions as well.
>> > > >>
>> > > >> Thanks,
>> > > >> Jungtaek Lim (HeartSaVioR)
>> > > >>
>> > > >> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이
>> 작성:
>> > > >>
>> > > >> Hi Jungtaek,
>> > > >>
>> > > >> > - Beam runner
>> > > >>
>> > > >> There’s not been much activity around this, and I haven’t had much
>> > time
>> > > to
>> > > >> work on it recently, but there’s a decent foundation to build upon.
>> So
>> > > it
>> > > >> would be fairly easy for others to start contributing to that
>> effort.
>> > > >> There’s also interest from the Beam community in that runner, so one
>> > > >> possibility is to move that effort to the Apache Beam project.
>> > > >>
>> > > >> This is very preliminary work, so I don’t have a good handle on what
>> > the
>> > > >> target release would be.
>> > > >>
>> > > >> > - Metrics renewal
>> > > >>
>> > > >>
>> > > >> This is what I’ve been referring to as “metrics_v2”. This is
>> > progressing
>> > > >> fairly well with support for multiple reporters (e.g. Graphite,
>> > Ganglia,
>> > > >> console, etc.), worker metrics, disruptor metrics, etc.
>> > > >>
>> > > >> I would like to target this work for 1.2.0.
>> > > >>
>> > > >> > - Java port
>> > > >>
>> > > >> This effort seems to have picked up (for example Bobby’s conversion
>> of
>> > > >> Nimbus, etc.) and is progressing steadily. It’s taken a lot longer
>> > than
>> > > >> initially thought, but a lot of that can be attributed to the ebb
>> and
>> > > flow
>> > > >> of people’s availability to do the work.
>> > > >>
>> > > >> > - Storm SQL improvement (Streaming SQL in future)
>> > > >>
>> > > >> You’ve been spearheading most of the work here, so I’d delegate to
>> you
>> > > for
>> > > >> your opinion on where it stands. If you need additional reviews,
>> just
>> > > ask
>> > > >> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line
>> > might
>> > > >> help get attention).
>> > > >>
>> > > >> My thinking has been that this could be included in the 1.1.0
>> release.
>> > > Is
>> > > >> there a set of JIRA issues you would like to include in order to
>> make
>> > > that
>> > > >> happen?
>> > > >>
>> > > >> > - Stream API
>> > > >>
>> > > >> This seems to have stalled a bit, though there seems to be a lot of
>> > > >> interest around it. I think we all would agree that when
>> introducing a
>> > > new
>> > > >> API for building topologies, it’s important that we get right from
>> the
>> > > >> start and have strong buy-in from the development community. I would
>> > > >> encourage anyone interested in the Streams API to review the
>> proposal
>> > > and
>> > > >> initial code.
>> > > >>
>> > > >> I think it is close, but I’m not sure what release to target.
>> Possibly
>> > > the
>> > > >> 2.0 release?
>> > > >>
>> > > >> Re: 1.1.0 Release
>> > > >>
>> > > >> STORM-2176 is a fairly big concern of mine since the feature it
>> > involves
>> > > >> was introduced in 1.0.0 and did not work then nor in any subsequent
>> or
>> > > >> future releases (may not be a problem in 2.0). Unfortunately, as
>> > you’ve
>> > > >> seen, finding the root cause is elusive. That issue could definitely
>> > use
>> > > >> more eyes.
>> > > >>
>> > > >> -Taylor
>> > > >>
>> > > >>
>> > > >> > On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com>
>> > wrote:
>> > > >> >
>> > > >> > Hi devs,
>> > > >> >
>> > > >> > I'm seeing lots of huge works in parallel, and we individual are
>> > busy
>> > > >> > regarding each work so common works (review, release,
>> documentation,
>> > > >> etc.)
>> > > >> > have been not made in progress for several months.
>> > > >> >
>> > > >> > - Beam runner
>> > > >> > - Metrics renewal
>> > > >> > - Java port
>> > > >> > - Storm SQL improvement (Streaming SQL in future)
>> > > >> > - Stream API
>> > > >> >
>> > > >> > IMHO, it would be better to set target versions for them, and set
>> a
>> > > >> roadmap
>> > > >> > (per version), and prioritize based on roadmap.
>> > > >> >
>> > > >> > Stream API (very first version), and Storm SQL improvement are
>> > waiting
>> > > >> for
>> > > >> > review, and personally I would like to publish them soon.
>> > > >> >
>> > > >> > If we're OK to have 2.0.0 without adding much features, I'm in
>> favor
>> > > of
>> > > >> > concentrating Java port work (postponing other things except
>> > releasing
>> > > >> 1.x
>> > > >> > version line) and moving to Apache Storm 2.0.0 really soon.
>> > > >> > (I'm even OK we decide to postpone some clojure files to be
>> > addressed
>> > > >> after
>> > > >> > 2.0.0.)
>> > > >> > Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8
>> > (2.x)
>> > > >> which
>> > > >> > is other reason to move to 2.x quickly.
>> > > >> >
>> > > >> > I'd be really happy if we have metrics renewal and beam runner,
>> but
>> > > I'm
>> > > >> not
>> > > >> > sure when they're available to be published. Do we have any
>> updates
>> > > here?
>> > > >> >
>> > > >> > What do you think? It might be ideal, and/or broader discussion
>> but
>> > we
>> > > >> > haven't discussed our plan / vision for a long time so better to
>> > give
>> > > it
>> > > >> a
>> > > >> > try.
>> > > >> >
>> > > >> > Thanks,
>> > > >> > Jungtaek Lim (HeartSaVioR)
>> > > >>
>> > > >>
>> > >
>> > >
>> > >
>> >
>>

Re: [DISCUSS] Prioritizing works in progress

[S G <sg...@gmail.com>]

One bad usage I have found is missing types for lambda arguments.
When the types for arguments are not provided in the code itself, github or
vi-editor browsing of such code becomes impossible.

You must have an IDE like Eclipse to infer the types for such cases and
that becomes a big limitation to browse such code. I have not looked into
the storm code-base to find such occurrences but the above is just a
general observation from the lambda code usages.


Secondly, lambdas tend to generate bad stack-traces on exceptions.


And lastly, functional programming is not natural to many Java developers.
https://zeroturnaround.com/rebellabs/how-to-avoid-ruining-your-world-with-lambdas-in-java-8/
hints at some of these problems. Java code is usually very easy to
understand. But once you throw in more than a single level of lambdas,
things tend to become more and more confusing for several developers. (This
last point reflects the thoughts from some of my fellow developers, not
representative of all the devs)


Thanks
SG


On Mon, Jan 2, 2017 at 8:26 PM, Jungtaek Lim <ka...@gmail.com> wrote:

> Hi S G,
>
> I'd be happy if you could elaborate your opinion. Did you found bad usages
> from master branch of Storm code?
>
> Regarding comfortable of lambda, IMHO, I don't think many users are
> unfamiliar with lambda, since they should have been used it with various
> languages. We might not be comfortable with Java 8 lambda (since transition
> to Java 8 is going slowly), but it's just a matter of familiarizing.
>
> Are there kind of best practices for Java 8 lambda? We can refer these to
> construct some guides / restrictions for Storm project.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2017년 1월 3일 (화) 오후 12:26, S G <sg...@gmail.com>님이 작성:
>
> > I have found several bad usages of Java 8 lambdas and many developers are
> > not comfortable using them.
> >
> > So we should use them only if it really makes the code beautiful and
> easier
> > to understand.
> >
> > My 2c,
> > Thanks
> > SG
> >
> >
> > On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org>
> wrote:
> >
> > > The streams API implementation has limited usage of 1.8 features and
> can
> > > be easily ported to 1.7 if required. The examples are written in 1.8,
> the
> > > thought being users would stick to the Java 8 style usage (lambdas)
> from
> > > the beginning. If there is consensus we could also consider moving the
> > 1.x
> > > branch to JDK 8.
> > >
> > > Anyways would like interested folks to start reviewing the changes so
> > that
> > > we can take it forward.
> > >
> > > Thanks,
> > > Arun
> > >
> > >
> > > On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> > >
> > > >FYI, I've realized that internal of Stream API (pull request) relies
> on
> > > JDK
> > > >8 (what I've found is 'static method in interface' and maybe more) so
> > for
> > > >now Stream API is expected to be included for at least Storm 2.0.0
> > unless
> > > >the PR is modified to fit to JDK 7.
> > > >
> > > >- Jungtaek Lim (HeartSaVioR)
> > > >
> > > >2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > > >
> > > >> Thanks Manu and Taylor for giving your opinions.
> > > >>
> > > >> - Storm SQL improvement
> > > >>
> > > >> There're some huge PRs available but there're all about improvement
> > > which
> > > >> shouldn't be blocker for releasing 1.1.0. (I'd like to also include
> > > them to
> > > >> 1.1.0 but not sure it can be happen really soon.)
> > > >> I'll send a request for reviewing about pending Storm SQL PRs.
> > > >>
> > > >> Only one issue (STORM-2200) is linked to release 1.1.0 epic which is
> > > >> blocker for me.
> > > >>
> > > >> - Java port
> > > >>
> > > >> I also had some developers saying 'If core of Storm were written by
> > > Java,
> > > >> I could experiment and even contribute on something'. I was one of
> > them,
> > > >> and to be honest, I'm still a beginner of Clojure. Moving to Java 8
> > also
> > > >> gives great functionalities for us, so Java port is what I think the
> > > most
> > > >> important thing among the huge works now in progress. Ideally, and
> > > >> hopefully, I'd like to see us focus on this and make this happen at
> > the
> > > >> very early next year.
> > > >> (Yes we should do some manual tests and maybe some refactoring too.)
> > > >>
> > > >> - Metrics V2
> > > >>
> > > >> I'm not sure when we plan to release Storm 1.2.0, but given that
> > > there're
> > > >> only two things left (logviewer / ui) for completing port work
> (except
> > > >> tests) I guess Storm 2.0.0 might be happen earlier.
> > > >> Taylor, when do you expect metrics V2 will be available for
> reviewing?
> > > >>
> > > >> - Stream API
> > > >>
> > > >> With labeling as experiment or annotating with evolving, we could
> > > include
> > > >> the first version to next minor excluding 1.1.0. (We could even
> > include
> > > >> this to 1.1.0 if we start reviewing this very soon.)
> > > >>
> > > >> I'd like to hear others' opinions as well.
> > > >>
> > > >> Thanks,
> > > >> Jungtaek Lim (HeartSaVioR)
> > > >>
> > > >> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이
> 작성:
> > > >>
> > > >> Hi Jungtaek,
> > > >>
> > > >> > - Beam runner
> > > >>
> > > >> There’s not been much activity around this, and I haven’t had much
> > time
> > > to
> > > >> work on it recently, but there’s a decent foundation to build upon.
> So
> > > it
> > > >> would be fairly easy for others to start contributing to that
> effort.
> > > >> There’s also interest from the Beam community in that runner, so one
> > > >> possibility is to move that effort to the Apache Beam project.
> > > >>
> > > >> This is very preliminary work, so I don’t have a good handle on what
> > the
> > > >> target release would be.
> > > >>
> > > >> > - Metrics renewal
> > > >>
> > > >>
> > > >> This is what I’ve been referring to as “metrics_v2”. This is
> > progressing
> > > >> fairly well with support for multiple reporters (e.g. Graphite,
> > Ganglia,
> > > >> console, etc.), worker metrics, disruptor metrics, etc.
> > > >>
> > > >> I would like to target this work for 1.2.0.
> > > >>
> > > >> > - Java port
> > > >>
> > > >> This effort seems to have picked up (for example Bobby’s conversion
> of
> > > >> Nimbus, etc.) and is progressing steadily. It’s taken a lot longer
> > than
> > > >> initially thought, but a lot of that can be attributed to the ebb
> and
> > > flow
> > > >> of people’s availability to do the work.
> > > >>
> > > >> > - Storm SQL improvement (Streaming SQL in future)
> > > >>
> > > >> You’ve been spearheading most of the work here, so I’d delegate to
> you
> > > for
> > > >> your opinion on where it stands. If you need additional reviews,
> just
> > > ask
> > > >> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line
> > might
> > > >> help get attention).
> > > >>
> > > >> My thinking has been that this could be included in the 1.1.0
> release.
> > > Is
> > > >> there a set of JIRA issues you would like to include in order to
> make
> > > that
> > > >> happen?
> > > >>
> > > >> > - Stream API
> > > >>
> > > >> This seems to have stalled a bit, though there seems to be a lot of
> > > >> interest around it. I think we all would agree that when
> introducing a
> > > new
> > > >> API for building topologies, it’s important that we get right from
> the
> > > >> start and have strong buy-in from the development community. I would
> > > >> encourage anyone interested in the Streams API to review the
> proposal
> > > and
> > > >> initial code.
> > > >>
> > > >> I think it is close, but I’m not sure what release to target.
> Possibly
> > > the
> > > >> 2.0 release?
> > > >>
> > > >> Re: 1.1.0 Release
> > > >>
> > > >> STORM-2176 is a fairly big concern of mine since the feature it
> > involves
> > > >> was introduced in 1.0.0 and did not work then nor in any subsequent
> or
> > > >> future releases (may not be a problem in 2.0). Unfortunately, as
> > you’ve
> > > >> seen, finding the root cause is elusive. That issue could definitely
> > use
> > > >> more eyes.
> > > >>
> > > >> -Taylor
> > > >>
> > > >>
> > > >> > On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com>
> > wrote:
> > > >> >
> > > >> > Hi devs,
> > > >> >
> > > >> > I'm seeing lots of huge works in parallel, and we individual are
> > busy
> > > >> > regarding each work so common works (review, release,
> documentation,
> > > >> etc.)
> > > >> > have been not made in progress for several months.
> > > >> >
> > > >> > - Beam runner
> > > >> > - Metrics renewal
> > > >> > - Java port
> > > >> > - Storm SQL improvement (Streaming SQL in future)
> > > >> > - Stream API
> > > >> >
> > > >> > IMHO, it would be better to set target versions for them, and set
> a
> > > >> roadmap
> > > >> > (per version), and prioritize based on roadmap.
> > > >> >
> > > >> > Stream API (very first version), and Storm SQL improvement are
> > waiting
> > > >> for
> > > >> > review, and personally I would like to publish them soon.
> > > >> >
> > > >> > If we're OK to have 2.0.0 without adding much features, I'm in
> favor
> > > of
> > > >> > concentrating Java port work (postponing other things except
> > releasing
> > > >> 1.x
> > > >> > version line) and moving to Apache Storm 2.0.0 really soon.
> > > >> > (I'm even OK we decide to postpone some clojure files to be
> > addressed
> > > >> after
> > > >> > 2.0.0.)
> > > >> > Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8
> > (2.x)
> > > >> which
> > > >> > is other reason to move to 2.x quickly.
> > > >> >
> > > >> > I'd be really happy if we have metrics renewal and beam runner,
> but
> > > I'm
> > > >> not
> > > >> > sure when they're available to be published. Do we have any
> updates
> > > here?
> > > >> >
> > > >> > What do you think? It might be ideal, and/or broader discussion
> but
> > we
> > > >> > haven't discussed our plan / vision for a long time so better to
> > give
> > > it
> > > >> a
> > > >> > try.
> > > >> >
> > > >> > Thanks,
> > > >> > Jungtaek Lim (HeartSaVioR)
> > > >>
> > > >>
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Prioritizing works in progress

[Jungtaek Lim <ka...@gmail.com>]

Hi S G,

I'd be happy if you could elaborate your opinion. Did you found bad usages
from master branch of Storm code?

Regarding comfortable of lambda, IMHO, I don't think many users are
unfamiliar with lambda, since they should have been used it with various
languages. We might not be comfortable with Java 8 lambda (since transition
to Java 8 is going slowly), but it's just a matter of familiarizing.

Are there kind of best practices for Java 8 lambda? We can refer these to
construct some guides / restrictions for Storm project.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 1월 3일 (화) 오후 12:26, S G <sg...@gmail.com>님이 작성:

> I have found several bad usages of Java 8 lambdas and many developers are
> not comfortable using them.
>
> So we should use them only if it really makes the code beautiful and easier
> to understand.
>
> My 2c,
> Thanks
> SG
>
>
> On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org> wrote:
>
> > The streams API implementation has limited usage of 1.8 features and can
> > be easily ported to 1.7 if required. The examples are written in 1.8, the
> > thought being users would stick to the Java 8 style usage (lambdas) from
> > the beginning. If there is consensus we could also consider moving the
> 1.x
> > branch to JDK 8.
> >
> > Anyways would like interested folks to start reviewing the changes so
> that
> > we can take it forward.
> >
> > Thanks,
> > Arun
> >
> >
> > On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >
> > >FYI, I've realized that internal of Stream API (pull request) relies on
> > JDK
> > >8 (what I've found is 'static method in interface' and maybe more) so
> for
> > >now Stream API is expected to be included for at least Storm 2.0.0
> unless
> > >the PR is modified to fit to JDK 7.
> > >
> > >- Jungtaek Lim (HeartSaVioR)
> > >
> > >2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >
> > >> Thanks Manu and Taylor for giving your opinions.
> > >>
> > >> - Storm SQL improvement
> > >>
> > >> There're some huge PRs available but there're all about improvement
> > which
> > >> shouldn't be blocker for releasing 1.1.0. (I'd like to also include
> > them to
> > >> 1.1.0 but not sure it can be happen really soon.)
> > >> I'll send a request for reviewing about pending Storm SQL PRs.
> > >>
> > >> Only one issue (STORM-2200) is linked to release 1.1.0 epic which is
> > >> blocker for me.
> > >>
> > >> - Java port
> > >>
> > >> I also had some developers saying 'If core of Storm were written by
> > Java,
> > >> I could experiment and even contribute on something'. I was one of
> them,
> > >> and to be honest, I'm still a beginner of Clojure. Moving to Java 8
> also
> > >> gives great functionalities for us, so Java port is what I think the
> > most
> > >> important thing among the huge works now in progress. Ideally, and
> > >> hopefully, I'd like to see us focus on this and make this happen at
> the
> > >> very early next year.
> > >> (Yes we should do some manual tests and maybe some refactoring too.)
> > >>
> > >> - Metrics V2
> > >>
> > >> I'm not sure when we plan to release Storm 1.2.0, but given that
> > there're
> > >> only two things left (logviewer / ui) for completing port work (except
> > >> tests) I guess Storm 2.0.0 might be happen earlier.
> > >> Taylor, when do you expect metrics V2 will be available for reviewing?
> > >>
> > >> - Stream API
> > >>
> > >> With labeling as experiment or annotating with evolving, we could
> > include
> > >> the first version to next minor excluding 1.1.0. (We could even
> include
> > >> this to 1.1.0 if we start reviewing this very soon.)
> > >>
> > >> I'd like to hear others' opinions as well.
> > >>
> > >> Thanks,
> > >> Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> > >>
> > >> Hi Jungtaek,
> > >>
> > >> > - Beam runner
> > >>
> > >> There’s not been much activity around this, and I haven’t had much
> time
> > to
> > >> work on it recently, but there’s a decent foundation to build upon. So
> > it
> > >> would be fairly easy for others to start contributing to that effort.
> > >> There’s also interest from the Beam community in that runner, so one
> > >> possibility is to move that effort to the Apache Beam project.
> > >>
> > >> This is very preliminary work, so I don’t have a good handle on what
> the
> > >> target release would be.
> > >>
> > >> > - Metrics renewal
> > >>
> > >>
> > >> This is what I’ve been referring to as “metrics_v2”. This is
> progressing
> > >> fairly well with support for multiple reporters (e.g. Graphite,
> Ganglia,
> > >> console, etc.), worker metrics, disruptor metrics, etc.
> > >>
> > >> I would like to target this work for 1.2.0.
> > >>
> > >> > - Java port
> > >>
> > >> This effort seems to have picked up (for example Bobby’s conversion of
> > >> Nimbus, etc.) and is progressing steadily. It’s taken a lot longer
> than
> > >> initially thought, but a lot of that can be attributed to the ebb and
> > flow
> > >> of people’s availability to do the work.
> > >>
> > >> > - Storm SQL improvement (Streaming SQL in future)
> > >>
> > >> You’ve been spearheading most of the work here, so I’d delegate to you
> > for
> > >> your opinion on where it stands. If you need additional reviews, just
> > ask
> > >> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line
> might
> > >> help get attention).
> > >>
> > >> My thinking has been that this could be included in the 1.1.0 release.
> > Is
> > >> there a set of JIRA issues you would like to include in order to make
> > that
> > >> happen?
> > >>
> > >> > - Stream API
> > >>
> > >> This seems to have stalled a bit, though there seems to be a lot of
> > >> interest around it. I think we all would agree that when introducing a
> > new
> > >> API for building topologies, it’s important that we get right from the
> > >> start and have strong buy-in from the development community. I would
> > >> encourage anyone interested in the Streams API to review the proposal
> > and
> > >> initial code.
> > >>
> > >> I think it is close, but I’m not sure what release to target. Possibly
> > the
> > >> 2.0 release?
> > >>
> > >> Re: 1.1.0 Release
> > >>
> > >> STORM-2176 is a fairly big concern of mine since the feature it
> involves
> > >> was introduced in 1.0.0 and did not work then nor in any subsequent or
> > >> future releases (may not be a problem in 2.0). Unfortunately, as
> you’ve
> > >> seen, finding the root cause is elusive. That issue could definitely
> use
> > >> more eyes.
> > >>
> > >> -Taylor
> > >>
> > >>
> > >> > On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com>
> wrote:
> > >> >
> > >> > Hi devs,
> > >> >
> > >> > I'm seeing lots of huge works in parallel, and we individual are
> busy
> > >> > regarding each work so common works (review, release, documentation,
> > >> etc.)
> > >> > have been not made in progress for several months.
> > >> >
> > >> > - Beam runner
> > >> > - Metrics renewal
> > >> > - Java port
> > >> > - Storm SQL improvement (Streaming SQL in future)
> > >> > - Stream API
> > >> >
> > >> > IMHO, it would be better to set target versions for them, and set a
> > >> roadmap
> > >> > (per version), and prioritize based on roadmap.
> > >> >
> > >> > Stream API (very first version), and Storm SQL improvement are
> waiting
> > >> for
> > >> > review, and personally I would like to publish them soon.
> > >> >
> > >> > If we're OK to have 2.0.0 without adding much features, I'm in favor
> > of
> > >> > concentrating Java port work (postponing other things except
> releasing
> > >> 1.x
> > >> > version line) and moving to Apache Storm 2.0.0 really soon.
> > >> > (I'm even OK we decide to postpone some clojure files to be
> addressed
> > >> after
> > >> > 2.0.0.)
> > >> > Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8
> (2.x)
> > >> which
> > >> > is other reason to move to 2.x quickly.
> > >> >
> > >> > I'd be really happy if we have metrics renewal and beam runner, but
> > I'm
> > >> not
> > >> > sure when they're available to be published. Do we have any updates
> > here?
> > >> >
> > >> > What do you think? It might be ideal, and/or broader discussion but
> we
> > >> > haven't discussed our plan / vision for a long time so better to
> give
> > it
> > >> a
> > >> > try.
> > >> >
> > >> > Thanks,
> > >> > Jungtaek Lim (HeartSaVioR)
> > >>
> > >>
> >
> >
> >
>

Re: [DISCUSS] Prioritizing works in progress

[S G <sg...@gmail.com>]

I have found several bad usages of Java 8 lambdas and many developers are
not comfortable using them.

So we should use them only if it really makes the code beautiful and easier
to understand.

My 2c,
Thanks
SG


On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan <ar...@apache.org> wrote:

> The streams API implementation has limited usage of 1.8 features and can
> be easily ported to 1.7 if required. The examples are written in 1.8, the
> thought being users would stick to the Java 8 style usage (lambdas) from
> the beginning. If there is consensus we could also consider moving the 1.x
> branch to JDK 8.
>
> Anyways would like interested folks to start reviewing the changes so that
> we can take it forward.
>
> Thanks,
> Arun
>
>
> On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>
> >FYI, I've realized that internal of Stream API (pull request) relies on
> JDK
> >8 (what I've found is 'static method in interface' and maybe more) so for
> >now Stream API is expected to be included for at least Storm 2.0.0 unless
> >the PR is modified to fit to JDK 7.
> >
> >- Jungtaek Lim (HeartSaVioR)
> >
> >2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> >> Thanks Manu and Taylor for giving your opinions.
> >>
> >> - Storm SQL improvement
> >>
> >> There're some huge PRs available but there're all about improvement
> which
> >> shouldn't be blocker for releasing 1.1.0. (I'd like to also include
> them to
> >> 1.1.0 but not sure it can be happen really soon.)
> >> I'll send a request for reviewing about pending Storm SQL PRs.
> >>
> >> Only one issue (STORM-2200) is linked to release 1.1.0 epic which is
> >> blocker for me.
> >>
> >> - Java port
> >>
> >> I also had some developers saying 'If core of Storm were written by
> Java,
> >> I could experiment and even contribute on something'. I was one of them,
> >> and to be honest, I'm still a beginner of Clojure. Moving to Java 8 also
> >> gives great functionalities for us, so Java port is what I think the
> most
> >> important thing among the huge works now in progress. Ideally, and
> >> hopefully, I'd like to see us focus on this and make this happen at the
> >> very early next year.
> >> (Yes we should do some manual tests and maybe some refactoring too.)
> >>
> >> - Metrics V2
> >>
> >> I'm not sure when we plan to release Storm 1.2.0, but given that
> there're
> >> only two things left (logviewer / ui) for completing port work (except
> >> tests) I guess Storm 2.0.0 might be happen earlier.
> >> Taylor, when do you expect metrics V2 will be available for reviewing?
> >>
> >> - Stream API
> >>
> >> With labeling as experiment or annotating with evolving, we could
> include
> >> the first version to next minor excluding 1.1.0. (We could even include
> >> this to 1.1.0 if we start reviewing this very soon.)
> >>
> >> I'd like to hear others' opinions as well.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> >>
> >> Hi Jungtaek,
> >>
> >> > - Beam runner
> >>
> >> There’s not been much activity around this, and I haven’t had much time
> to
> >> work on it recently, but there’s a decent foundation to build upon. So
> it
> >> would be fairly easy for others to start contributing to that effort.
> >> There’s also interest from the Beam community in that runner, so one
> >> possibility is to move that effort to the Apache Beam project.
> >>
> >> This is very preliminary work, so I don’t have a good handle on what the
> >> target release would be.
> >>
> >> > - Metrics renewal
> >>
> >>
> >> This is what I’ve been referring to as “metrics_v2”. This is progressing
> >> fairly well with support for multiple reporters (e.g. Graphite, Ganglia,
> >> console, etc.), worker metrics, disruptor metrics, etc.
> >>
> >> I would like to target this work for 1.2.0.
> >>
> >> > - Java port
> >>
> >> This effort seems to have picked up (for example Bobby’s conversion of
> >> Nimbus, etc.) and is progressing steadily. It’s taken a lot longer than
> >> initially thought, but a lot of that can be attributed to the ebb and
> flow
> >> of people’s availability to do the work.
> >>
> >> > - Storm SQL improvement (Streaming SQL in future)
> >>
> >> You’ve been spearheading most of the work here, so I’d delegate to you
> for
> >> your opinion on where it stands. If you need additional reviews, just
> ask
> >> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line might
> >> help get attention).
> >>
> >> My thinking has been that this could be included in the 1.1.0 release.
> Is
> >> there a set of JIRA issues you would like to include in order to make
> that
> >> happen?
> >>
> >> > - Stream API
> >>
> >> This seems to have stalled a bit, though there seems to be a lot of
> >> interest around it. I think we all would agree that when introducing a
> new
> >> API for building topologies, it’s important that we get right from the
> >> start and have strong buy-in from the development community. I would
> >> encourage anyone interested in the Streams API to review the proposal
> and
> >> initial code.
> >>
> >> I think it is close, but I’m not sure what release to target. Possibly
> the
> >> 2.0 release?
> >>
> >> Re: 1.1.0 Release
> >>
> >> STORM-2176 is a fairly big concern of mine since the feature it involves
> >> was introduced in 1.0.0 and did not work then nor in any subsequent or
> >> future releases (may not be a problem in 2.0). Unfortunately, as you’ve
> >> seen, finding the root cause is elusive. That issue could definitely use
> >> more eyes.
> >>
> >> -Taylor
> >>
> >>
> >> > On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> >> >
> >> > Hi devs,
> >> >
> >> > I'm seeing lots of huge works in parallel, and we individual are busy
> >> > regarding each work so common works (review, release, documentation,
> >> etc.)
> >> > have been not made in progress for several months.
> >> >
> >> > - Beam runner
> >> > - Metrics renewal
> >> > - Java port
> >> > - Storm SQL improvement (Streaming SQL in future)
> >> > - Stream API
> >> >
> >> > IMHO, it would be better to set target versions for them, and set a
> >> roadmap
> >> > (per version), and prioritize based on roadmap.
> >> >
> >> > Stream API (very first version), and Storm SQL improvement are waiting
> >> for
> >> > review, and personally I would like to publish them soon.
> >> >
> >> > If we're OK to have 2.0.0 without adding much features, I'm in favor
> of
> >> > concentrating Java port work (postponing other things except releasing
> >> 1.x
> >> > version line) and moving to Apache Storm 2.0.0 really soon.
> >> > (I'm even OK we decide to postpone some clojure files to be addressed
> >> after
> >> > 2.0.0.)
> >> > Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8 (2.x)
> >> which
> >> > is other reason to move to 2.x quickly.
> >> >
> >> > I'd be really happy if we have metrics renewal and beam runner, but
> I'm
> >> not
> >> > sure when they're available to be published. Do we have any updates
> here?
> >> >
> >> > What do you think? It might be ideal, and/or broader discussion but we
> >> > haven't discussed our plan / vision for a long time so better to give
> it
> >> a
> >> > try.
> >> >
> >> > Thanks,
> >> > Jungtaek Lim (HeartSaVioR)
> >>
> >>
>
>
>

Re: [DISCUSS] Prioritizing works in progress

[Arun Mahadevan <ar...@apache.org>]

The streams API implementation has limited usage of 1.8 features and can be easily ported to 1.7 if required. The examples are written in 1.8, the thought being users would stick to the Java 8 style usage (lambdas) from the beginning. If there is consensus we could also consider moving the 1.x branch to JDK 8. 

Anyways would like interested folks to start reviewing the changes so that we can take it forward.

Thanks,
Arun


On 12/23/16, 10:09 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:

>FYI, I've realized that internal of Stream API (pull request) relies on JDK
>8 (what I've found is 'static method in interface' and maybe more) so for
>now Stream API is expected to be included for at least Storm 2.0.0 unless
>the PR is modified to fit to JDK 7.
>
>- Jungtaek Lim (HeartSaVioR)
>
>2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
>> Thanks Manu and Taylor for giving your opinions.
>>
>> - Storm SQL improvement
>>
>> There're some huge PRs available but there're all about improvement which
>> shouldn't be blocker for releasing 1.1.0. (I'd like to also include them to
>> 1.1.0 but not sure it can be happen really soon.)
>> I'll send a request for reviewing about pending Storm SQL PRs.
>>
>> Only one issue (STORM-2200) is linked to release 1.1.0 epic which is
>> blocker for me.
>>
>> - Java port
>>
>> I also had some developers saying 'If core of Storm were written by Java,
>> I could experiment and even contribute on something'. I was one of them,
>> and to be honest, I'm still a beginner of Clojure. Moving to Java 8 also
>> gives great functionalities for us, so Java port is what I think the most
>> important thing among the huge works now in progress. Ideally, and
>> hopefully, I'd like to see us focus on this and make this happen at the
>> very early next year.
>> (Yes we should do some manual tests and maybe some refactoring too.)
>>
>> - Metrics V2
>>
>> I'm not sure when we plan to release Storm 1.2.0, but given that there're
>> only two things left (logviewer / ui) for completing port work (except
>> tests) I guess Storm 2.0.0 might be happen earlier.
>> Taylor, when do you expect metrics V2 will be available for reviewing?
>>
>> - Stream API
>>
>> With labeling as experiment or annotating with evolving, we could include
>> the first version to next minor excluding 1.1.0. (We could even include
>> this to 1.1.0 if we start reviewing this very soon.)
>>
>> I'd like to hear others' opinions as well.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>>
>> Hi Jungtaek,
>>
>> > - Beam runner
>>
>> There’s not been much activity around this, and I haven’t had much time to
>> work on it recently, but there’s a decent foundation to build upon. So it
>> would be fairly easy for others to start contributing to that effort.
>> There’s also interest from the Beam community in that runner, so one
>> possibility is to move that effort to the Apache Beam project.
>>
>> This is very preliminary work, so I don’t have a good handle on what the
>> target release would be.
>>
>> > - Metrics renewal
>>
>>
>> This is what I’ve been referring to as “metrics_v2”. This is progressing
>> fairly well with support for multiple reporters (e.g. Graphite, Ganglia,
>> console, etc.), worker metrics, disruptor metrics, etc.
>>
>> I would like to target this work for 1.2.0.
>>
>> > - Java port
>>
>> This effort seems to have picked up (for example Bobby’s conversion of
>> Nimbus, etc.) and is progressing steadily. It’s taken a lot longer than
>> initially thought, but a lot of that can be attributed to the ebb and flow
>> of people’s availability to do the work.
>>
>> > - Storm SQL improvement (Streaming SQL in future)
>>
>> You’ve been spearheading most of the work here, so I’d delegate to you for
>> your opinion on where it stands. If you need additional reviews, just ask
>> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line might
>> help get attention).
>>
>> My thinking has been that this could be included in the 1.1.0 release. Is
>> there a set of JIRA issues you would like to include in order to make that
>> happen?
>>
>> > - Stream API
>>
>> This seems to have stalled a bit, though there seems to be a lot of
>> interest around it. I think we all would agree that when introducing a new
>> API for building topologies, it’s important that we get right from the
>> start and have strong buy-in from the development community. I would
>> encourage anyone interested in the Streams API to review the proposal and
>> initial code.
>>
>> I think it is close, but I’m not sure what release to target. Possibly the
>> 2.0 release?
>>
>> Re: 1.1.0 Release
>>
>> STORM-2176 is a fairly big concern of mine since the feature it involves
>> was introduced in 1.0.0 and did not work then nor in any subsequent or
>> future releases (may not be a problem in 2.0). Unfortunately, as you’ve
>> seen, finding the root cause is elusive. That issue could definitely use
>> more eyes.
>>
>> -Taylor
>>
>>
>> > On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>> >
>> > Hi devs,
>> >
>> > I'm seeing lots of huge works in parallel, and we individual are busy
>> > regarding each work so common works (review, release, documentation,
>> etc.)
>> > have been not made in progress for several months.
>> >
>> > - Beam runner
>> > - Metrics renewal
>> > - Java port
>> > - Storm SQL improvement (Streaming SQL in future)
>> > - Stream API
>> >
>> > IMHO, it would be better to set target versions for them, and set a
>> roadmap
>> > (per version), and prioritize based on roadmap.
>> >
>> > Stream API (very first version), and Storm SQL improvement are waiting
>> for
>> > review, and personally I would like to publish them soon.
>> >
>> > If we're OK to have 2.0.0 without adding much features, I'm in favor of
>> > concentrating Java port work (postponing other things except releasing
>> 1.x
>> > version line) and moving to Apache Storm 2.0.0 really soon.
>> > (I'm even OK we decide to postpone some clojure files to be addressed
>> after
>> > 2.0.0.)
>> > Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8 (2.x)
>> which
>> > is other reason to move to 2.x quickly.
>> >
>> > I'd be really happy if we have metrics renewal and beam runner, but I'm
>> not
>> > sure when they're available to be published. Do we have any updates here?
>> >
>> > What do you think? It might be ideal, and/or broader discussion but we
>> > haven't discussed our plan / vision for a long time so better to give it
>> a
>> > try.
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>>
>>

Re: [DISCUSS] Prioritizing works in progress

[Jungtaek Lim <ka...@gmail.com>]

FYI, I've realized that internal of Stream API (pull request) relies on JDK
8 (what I've found is 'static method in interface' and maybe more) so for
now Stream API is expected to be included for at least Storm 2.0.0 unless
the PR is modified to fit to JDK 7.

- Jungtaek Lim (HeartSaVioR)

2016년 12월 21일 (수) 오전 9:40, Jungtaek Lim <ka...@gmail.com>님이 작성:

> Thanks Manu and Taylor for giving your opinions.
>
> - Storm SQL improvement
>
> There're some huge PRs available but there're all about improvement which
> shouldn't be blocker for releasing 1.1.0. (I'd like to also include them to
> 1.1.0 but not sure it can be happen really soon.)
> I'll send a request for reviewing about pending Storm SQL PRs.
>
> Only one issue (STORM-2200) is linked to release 1.1.0 epic which is
> blocker for me.
>
> - Java port
>
> I also had some developers saying 'If core of Storm were written by Java,
> I could experiment and even contribute on something'. I was one of them,
> and to be honest, I'm still a beginner of Clojure. Moving to Java 8 also
> gives great functionalities for us, so Java port is what I think the most
> important thing among the huge works now in progress. Ideally, and
> hopefully, I'd like to see us focus on this and make this happen at the
> very early next year.
> (Yes we should do some manual tests and maybe some refactoring too.)
>
> - Metrics V2
>
> I'm not sure when we plan to release Storm 1.2.0, but given that there're
> only two things left (logviewer / ui) for completing port work (except
> tests) I guess Storm 2.0.0 might be happen earlier.
> Taylor, when do you expect metrics V2 will be available for reviewing?
>
> - Stream API
>
> With labeling as experiment or annotating with evolving, we could include
> the first version to next minor excluding 1.1.0. (We could even include
> this to 1.1.0 if we start reviewing this very soon.)
>
> I'd like to hear others' opinions as well.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>
> Hi Jungtaek,
>
> > - Beam runner
>
> There’s not been much activity around this, and I haven’t had much time to
> work on it recently, but there’s a decent foundation to build upon. So it
> would be fairly easy for others to start contributing to that effort.
> There’s also interest from the Beam community in that runner, so one
> possibility is to move that effort to the Apache Beam project.
>
> This is very preliminary work, so I don’t have a good handle on what the
> target release would be.
>
> > - Metrics renewal
>
>
> This is what I’ve been referring to as “metrics_v2”. This is progressing
> fairly well with support for multiple reporters (e.g. Graphite, Ganglia,
> console, etc.), worker metrics, disruptor metrics, etc.
>
> I would like to target this work for 1.2.0.
>
> > - Java port
>
> This effort seems to have picked up (for example Bobby’s conversion of
> Nimbus, etc.) and is progressing steadily. It’s taken a lot longer than
> initially thought, but a lot of that can be attributed to the ebb and flow
> of people’s availability to do the work.
>
> > - Storm SQL improvement (Streaming SQL in future)
>
> You’ve been spearheading most of the work here, so I’d delegate to you for
> your opinion on where it stands. If you need additional reviews, just ask
> on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line might
> help get attention).
>
> My thinking has been that this could be included in the 1.1.0 release. Is
> there a set of JIRA issues you would like to include in order to make that
> happen?
>
> > - Stream API
>
> This seems to have stalled a bit, though there seems to be a lot of
> interest around it. I think we all would agree that when introducing a new
> API for building topologies, it’s important that we get right from the
> start and have strong buy-in from the development community. I would
> encourage anyone interested in the Streams API to review the proposal and
> initial code.
>
> I think it is close, but I’m not sure what release to target. Possibly the
> 2.0 release?
>
> Re: 1.1.0 Release
>
> STORM-2176 is a fairly big concern of mine since the feature it involves
> was introduced in 1.0.0 and did not work then nor in any subsequent or
> future releases (may not be a problem in 2.0). Unfortunately, as you’ve
> seen, finding the root cause is elusive. That issue could definitely use
> more eyes.
>
> -Taylor
>
>
> > On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> >
> > Hi devs,
> >
> > I'm seeing lots of huge works in parallel, and we individual are busy
> > regarding each work so common works (review, release, documentation,
> etc.)
> > have been not made in progress for several months.
> >
> > - Beam runner
> > - Metrics renewal
> > - Java port
> > - Storm SQL improvement (Streaming SQL in future)
> > - Stream API
> >
> > IMHO, it would be better to set target versions for them, and set a
> roadmap
> > (per version), and prioritize based on roadmap.
> >
> > Stream API (very first version), and Storm SQL improvement are waiting
> for
> > review, and personally I would like to publish them soon.
> >
> > If we're OK to have 2.0.0 without adding much features, I'm in favor of
> > concentrating Java port work (postponing other things except releasing
> 1.x
> > version line) and moving to Apache Storm 2.0.0 really soon.
> > (I'm even OK we decide to postpone some clojure files to be addressed
> after
> > 2.0.0.)
> > Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8 (2.x)
> which
> > is other reason to move to 2.x quickly.
> >
> > I'd be really happy if we have metrics renewal and beam runner, but I'm
> not
> > sure when they're available to be published. Do we have any updates here?
> >
> > What do you think? It might be ideal, and/or broader discussion but we
> > haven't discussed our plan / vision for a long time so better to give it
> a
> > try.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
>
>

Re: [DISCUSS] Prioritizing works in progress

[Jungtaek Lim <ka...@gmail.com>]

Thanks Manu and Taylor for giving your opinions.

- Storm SQL improvement

There're some huge PRs available but there're all about improvement which
shouldn't be blocker for releasing 1.1.0. (I'd like to also include them to
1.1.0 but not sure it can be happen really soon.)
I'll send a request for reviewing about pending Storm SQL PRs.

Only one issue (STORM-2200) is linked to release 1.1.0 epic which is
blocker for me.

- Java port

I also had some developers saying 'If core of Storm were written by Java, I
could experiment and even contribute on something'. I was one of them, and
to be honest, I'm still a beginner of Clojure. Moving to Java 8 also gives
great functionalities for us, so Java port is what I think the most
important thing among the huge works now in progress. Ideally, and
hopefully, I'd like to see us focus on this and make this happen at the
very early next year.
(Yes we should do some manual tests and maybe some refactoring too.)

- Metrics V2

I'm not sure when we plan to release Storm 1.2.0, but given that there're
only two things left (logviewer / ui) for completing port work (except
tests) I guess Storm 2.0.0 might be happen earlier.
Taylor, when do you expect metrics V2 will be available for reviewing?

- Stream API

With labeling as experiment or annotating with evolving, we could include
the first version to next minor excluding 1.1.0. (We could even include
this to 1.1.0 if we start reviewing this very soon.)

I'd like to hear others' opinions as well.

Thanks,
Jungtaek Lim (HeartSaVioR)

2016년 12월 21일 (수) 오전 7:33, P. Taylor Goetz <pt...@gmail.com>님이 작성:

Hi Jungtaek,

> - Beam runner

There’s not been much activity around this, and I haven’t had much time to
work on it recently, but there’s a decent foundation to build upon. So it
would be fairly easy for others to start contributing to that effort.
There’s also interest from the Beam community in that runner, so one
possibility is to move that effort to the Apache Beam project.

This is very preliminary work, so I don’t have a good handle on what the
target release would be.

> - Metrics renewal


This is what I’ve been referring to as “metrics_v2”. This is progressing
fairly well with support for multiple reporters (e.g. Graphite, Ganglia,
console, etc.), worker metrics, disruptor metrics, etc.

I would like to target this work for 1.2.0.

> - Java port

This effort seems to have picked up (for example Bobby’s conversion of
Nimbus, etc.) and is progressing steadily. It’s taken a lot longer than
initially thought, but a lot of that can be attributed to the ebb and flow
of people’s availability to do the work.

> - Storm SQL improvement (Streaming SQL in future)

You’ve been spearheading most of the work here, so I’d delegate to you for
your opinion on where it stands. If you need additional reviews, just ask
on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line might
help get attention).

My thinking has been that this could be included in the 1.1.0 release. Is
there a set of JIRA issues you would like to include in order to make that
happen?

> - Stream API

This seems to have stalled a bit, though there seems to be a lot of
interest around it. I think we all would agree that when introducing a new
API for building topologies, it’s important that we get right from the
start and have strong buy-in from the development community. I would
encourage anyone interested in the Streams API to review the proposal and
initial code.

I think it is close, but I’m not sure what release to target. Possibly the
2.0 release?

Re: 1.1.0 Release

STORM-2176 is a fairly big concern of mine since the feature it involves
was introduced in 1.0.0 and did not work then nor in any subsequent or
future releases (may not be a problem in 2.0). Unfortunately, as you’ve
seen, finding the root cause is elusive. That issue could definitely use
more eyes.

-Taylor


> On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Hi devs,
>
> I'm seeing lots of huge works in parallel, and we individual are busy
> regarding each work so common works (review, release, documentation, etc.)
> have been not made in progress for several months.
>
> - Beam runner
> - Metrics renewal
> - Java port
> - Storm SQL improvement (Streaming SQL in future)
> - Stream API
>
> IMHO, it would be better to set target versions for them, and set a
roadmap
> (per version), and prioritize based on roadmap.
>
> Stream API (very first version), and Storm SQL improvement are waiting for
> review, and personally I would like to publish them soon.
>
> If we're OK to have 2.0.0 without adding much features, I'm in favor of
> concentrating Java port work (postponing other things except releasing 1.x
> version line) and moving to Apache Storm 2.0.0 really soon.
> (I'm even OK we decide to postpone some clojure files to be addressed
after
> 2.0.0.)
> Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8 (2.x)
which
> is other reason to move to 2.x quickly.
>
> I'd be really happy if we have metrics renewal and beam runner, but I'm
not
> sure when they're available to be published. Do we have any updates here?
>
> What do you think? It might be ideal, and/or broader discussion but we
> haven't discussed our plan / vision for a long time so better to give it a
> try.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)

Re: [DISCUSS] Prioritizing works in progress

["P. Taylor Goetz" <pt...@gmail.com>]

Hi Jungtaek,

> - Beam runner

There’s not been much activity around this, and I haven’t had much time to work on it recently, but there’s a decent foundation to build upon. So it would be fairly easy for others to start contributing to that effort. There’s also interest from the Beam community in that runner, so one possibility is to move that effort to the Apache Beam project.

This is very preliminary work, so I don’t have a good handle on what the target release would be.

> - Metrics renewal


This is what I’ve been referring to as “metrics_v2”. This is progressing fairly well with support for multiple reporters (e.g. Graphite, Ganglia, console, etc.), worker metrics, disruptor metrics, etc.

I would like to target this work for 1.2.0.

> - Java port

This effort seems to have picked up (for example Bobby’s conversion of Nimbus, etc.) and is progressing steadily. It’s taken a lot longer than initially thought, but a lot of that can be attributed to the ebb and flow of people’s availability to do the work.

> - Storm SQL improvement (Streaming SQL in future)

You’ve been spearheading most of the work here, so I’d delegate to you for your opinion on where it stands. If you need additional reviews, just ask on list or via GitHub (e.g. “[REVIEW REQUEST]” in the subject line might help get attention).

My thinking has been that this could be included in the 1.1.0 release. Is there a set of JIRA issues you would like to include in order to make that happen?

> - Stream API

This seems to have stalled a bit, though there seems to be a lot of interest around it. I think we all would agree that when introducing a new API for building topologies, it’s important that we get right from the start and have strong buy-in from the development community. I would encourage anyone interested in the Streams API to review the proposal and initial code.

I think it is close, but I’m not sure what release to target. Possibly the 2.0 release?

Re: 1.1.0 Release

STORM-2176 is a fairly big concern of mine since the feature it involves was introduced in 1.0.0 and did not work then nor in any subsequent or future releases (may not be a problem in 2.0). Unfortunately, as you’ve seen, finding the root cause is elusive. That issue could definitely use more eyes.

-Taylor


> On Dec 20, 2016, at 2:19 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> Hi devs,
> 
> I'm seeing lots of huge works in parallel, and we individual are busy
> regarding each work so common works (review, release, documentation, etc.)
> have been not made in progress for several months.
> 
> - Beam runner
> - Metrics renewal
> - Java port
> - Storm SQL improvement (Streaming SQL in future)
> - Stream API
> 
> IMHO, it would be better to set target versions for them, and set a roadmap
> (per version), and prioritize based on roadmap.
> 
> Stream API (very first version), and Storm SQL improvement are waiting for
> review, and personally I would like to publish them soon.
> 
> If we're OK to have 2.0.0 without adding much features, I'm in favor of
> concentrating Java port work (postponing other things except releasing 1.x
> version line) and moving to Apache Storm 2.0.0 really soon.
> (I'm even OK we decide to postpone some clojure files to be addressed after
> 2.0.0.)
> Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8 (2.x) which
> is other reason to move to 2.x quickly.
> 
> I'd be really happy if we have metrics renewal and beam runner, but I'm not
> sure when they're available to be published. Do we have any updates here?
> 
> What do you think? It might be ideal, and/or broader discussion but we
> haven't discussed our plan / vision for a long time so better to give it a
> try.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)

Re: [DISCUSS] Prioritizing works in progress

[Manu Zhang <ow...@gmail.com>]

Hi,

Please prioritize 2.0.0 release if possible. We have users in China we want
it strongly and I think that will also bring in a lot of contributions
since the core is Java then.
Metrics is another highly valued feature. But if we can only do one thing
now, I hope it is 2.0.0 release

Thanks,
Manu Zhang

On Tue, Dec 20, 2016 at 3:19 PM Jungtaek Lim <ka...@gmail.com> wrote:

> Hi devs,
>
> I'm seeing lots of huge works in parallel, and we individual are busy
> regarding each work so common works (review, release, documentation, etc.)
> have been not made in progress for several months.
>
> - Beam runner
> - Metrics renewal
> - Java port
> - Storm SQL improvement (Streaming SQL in future)
> - Stream API
>
> IMHO, it would be better to set target versions for them, and set a roadmap
> (per version), and prioritize based on roadmap.
>
> Stream API (very first version), and Storm SQL improvement are waiting for
> review, and personally I would like to publish them soon.
>
> If we're OK to have 2.0.0 without adding much features, I'm in favor of
> concentrating Java port work (postponing other things except releasing 1.x
> version line) and moving to Apache Storm 2.0.0 really soon.
> (I'm even OK we decide to postpone some clojure files to be addressed after
> 2.0.0.)
> Actually we're suffering other annoying issue: JDK 7 (1.x) vs 8 (2.x) which
> is other reason to move to 2.x quickly.
>
> I'd be really happy if we have metrics renewal and beam runner, but I'm not
> sure when they're available to be published. Do we have any updates here?
>
> What do you think? It might be ideal, and/or broader discussion but we
> haven't discussed our plan / vision for a long time so better to give it a
> try.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>