You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Jungtaek Lim <ka...@gmail.com> on 2018/02/01 01:48:29 UTC
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
This discussion got 4 +1 (binding) and no -1. Moreover two active
maintainers for storm-kafka-client (Hugo and Stig) voted +1.
Do we want to hold on for hearing more voices, or treating above opinions
as consensus and reflect the change?
Btw, I think we need to sort out the sequences between two topics:
separating storm-kafka-client as independent release cycle, and this. I
guess some of us agreed former topic doesn't related to current RC, but I
think this topic can be (should be) reflected to current RC ongoing.
-Jungtaek Lim (HeartSaVioR)
2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hl...@hortonworks.com>님이 작성:
> +1 to replace storm-kafka-client in 1.0.x branch.
> Hugo
>
> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <st...@gmail.com>
> wrote:
> >
> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
> > versioning is really nasty, but I think it is the lesser evil in this
> case.
> >
> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> >
> >> +1 to replace storm-kafka-client in 1.0.x branch
> >> -Harsha
> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> >>> Bump up this thread so that we could reach consensus earlier. Given
> that
> >> we
> >>> got concern related to this, I think it is ideal to release 1.1.x/1.0.x
> >>> with making decision and applying the change if we want.
> >>>
> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>
> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use Storm
> >> 1.1.0
> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
> leaving
> >>>> storm-kafka-client in 1.0.x in inconsistent state.
> >>>>
> >>>> IMHO, breaking backward compatibility is worse, but leaving broken
> >> thing
> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may bring
> >>>> backward incompatibility in release announce.
> >>>>
> >>>> -Jungtaek Lim (HeartSaVioR)
> >>>>
> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> >>>>
> >>>>> As I mentioned else thread I’m open to this but would defer to
> >> community
> >>>>> consensus.
> >>>>>
> >>>>> If there’s concern about doing this for 1.0.x, one option would be
> >> skip
> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> >>>>>
> >>>>> -Taylor
> >>>>>
> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <ka...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>> Hi devs,
> >>>>>>
> >>>>>> This is initial post to separate out discussion topic from vote
> >> thread,
> >>>>> and
> >>>>>> continue discussing.
> >>>>>>
> >>>>>> Background of the topic:
> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> >> (relatively)
> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x because
> >>>>> they're
> >>>>>> diverged too much.
> >>>>>>
> >>>>>> Downside:
> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x. Not
> >> sure for
> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client version is
> >>>>>> different, and if my memory is right, we already applied backward
> >>>>>> incompatible change into storm-kafka-client 1.1.0.
> >>>>>>
> >>>>>> Please put your opinion regarding topic. You're encouraged to copy
> >> your
> >>>>>> previous post in vote thread which helps to centralize opinions in
> >>>>> current
> >>>>>> thread.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>
> >>>>>
> >>
>
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Jungtaek Lim <ka...@gmail.com>.
Update: I also merged the patch for 1.0.x-branch. Thanks all for taking
valuable efforts and time to participate this topic and making it possible
so quickly. Now release phase of both 1.1.2 and 1.0.6 can be restart.
2018년 2월 8일 (목) 오전 6:30, Jungtaek Lim <ka...@gmail.com>님이 작성:
> Update: I just merged the patch for 1.1.x-branch so release phase of Storm
> 1.1.2 can be restarted. Patch for 1.0.x-branch from Erik is available and
> got some +1s but waiting for 24hrs rule.
>
> 2018년 2월 7일 (수) 오전 5:03, Stig Rohde Døssing <st...@gmail.com>님이 작성:
>
>> Took a look at backporting to 1.0.x. We'll have to update the time
>> simulation code (Time.java in storm-core) to support nanoseconds, as Erik
>> noted, but this isn't a breaking change and only affects tests.
>>
>> This PR https://github.com/apache/storm/pull/1995/files#diff-
>> 72647db30ffd6005dc01c4d1f75d2c68 made a breaking change to
>> IOpaquePartitionedTridentSpoutExecutor, so we'll have to do the same on
>> 1.0.x.
>>
>> 2018-02-06 19:13 GMT+01:00 P. Taylor Goetz <pt...@gmail.com>:
>>
>> > Just a heads up: While this gets sorted out I’m going to proceed with a
>> > 1.2.0 RC.
>> >
>> > -Taylor
>> >
>> > > On Feb 5, 2018, at 10:46 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>> > >
>> > > UPDATE: Submitted a pull request https://github.com/apache/
>> > storm/pull/2549 for
>> > > STORM-2936 (against 1.1.x-branch)
>> > >
>> > > Erik, please change the status to "IN PROGRESS" if someone is working
>> > on. I
>> > > would find the free time and just do it if there's no one working in
>> > > progress.
>> > >
>> > > 2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <ka...@gmail.com>님이 작성:
>> > >
>> > >> Thanks for quick response Erik!
>> > >>
>> > >> Just filed two issues :
>> > >> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
>> > >> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
>> > >>
>> > >> We have another discussion around making storm-kafka-client be
>> > experiment
>> > >> of managing separately (independent of Storm release). So the three
>> > >> versions which are in release phase might be the last releases of
>> > >> "battery-included" of storm-kafka-client if our experiment works
>> well.
>> > >>
>> > >> If we would want to make the change for storm-kafka-client, it might
>> be
>> > >> better to put the change and release before start experimenting, but
>> > that's
>> > >> just a thought on my own. In opposite way, we could even start
>> > experiment
>> > >> and make change of storm-core of 1.0.x-branch to be compatible with
>> that
>> > >> version of storm-kafka-client. We could even do it for 1.1.x-branch,
>> but
>> > >> the change is almost done so it doesn't look like needed to postpone
>> it.
>> > >>
>> > >> Would like to here everyone's voice on this.
>> > >>
>> > >> -Jungtaek Lim (HeartSaVioR)
>> > >>
>> > >> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers
>> <eweathers@groupon.com.invalid
>> > >님이
>> > >> 작성:
>> > >>
>> > >>> Thanks for the quick response Jungtaek!
>> > >>>
>> > >>> Yes, my teammates and myself would like to help on this. Is there
>> an
>> > >>> existing JIRA for the work you've been doing on the other branches?
>> > >>>
>> > >>> I propose we don't make this block 1.0.6 -- we can just release
>> 1.0.7
>> > >>> quickly when the backport is done, if that is amenable.
>> > >>> That strategy also might be cleaner since it would avoid other
>> changes
>> > in
>> > >>> 1.0.6 being lumped together with this.
>> > >>>
>> > >>> - Erik
>> > >>>
>> > >>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <ka...@gmail.com>
>> > wrote:
>> > >>>
>> > >>>> UPDATE: I've finished working on overwriting storm-kafka-client
>> > >>> 1.x-branch
>> > >>>> to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork
>> > first
>> > >>> to
>> > >>>> trigger Travis CI to see how the build goes well.
>> > >>>>
>> > >>>> https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
>> > >>>> 612e87da8589f5723f05218a
>> > >>>> https://travis-ci.org/HeartSaVioR/storm/builds/337819430
>> > >>>>
>> > >>>> Thanks for the input regarding 1.0.x version, Erik. I guess then we
>> > >>> have no
>> > >>>> alternative here: someone has to fix storm-kafka-client as well as
>> > >>>> storm-core, since including shaded storm-core doesn't make sense
>> for
>> > >>>> official Storm release.
>> > >>>>
>> > >>>> I guess it doesn't take many hour(s), hence may not worth to sync
>> and
>> > >>> talk
>> > >>>> offline. I just wanted to judge whether we are OK to make change of
>> > >>>> storm-core in bugfix version lines, but maybe the judgement itself
>> can
>> > >>> be
>> > >>>> possible after finishing the change, so I'll just go ahead making
>> the
>> > >>>> change.
>> > >>>> Since this is blocking release candidate, we should get it ASAP.
>> > That's
>> > >>> why
>> > >>>> I'm eager to go ahead making the change. If you could spend time
>> now
>> > >>>> helping with making the change ASAP, please leave short notice
>> (maybe
>> > >>> with
>> > >>>> JIRA issue?) and go ahead.
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Jungtaek Lim (HeartSaVioR)
>> > >>>>
>> > >>>> 2018년 2월 6일 (화) 오전 9:41, Erik Weathers
>> <eweathers@groupon.com.invalid
>> > >>>> 님이
>> > >>>> 작성:
>> > >>>>
>> > >>>>> hey Jungtaek,
>> > >>>>>
>> > >>>>> Thanks for continuing to pursue this!
>> > >>>>>
>> > >>>>> The issue for Storm not working on Mesos is due to a fundamental
>> > >>> change
>> > >>>> to
>> > >>>>> the core scheduling logic in Storm:
>> > >>>>>
>> > >>>>> -
>> > >>>>>
>> > >>>>> https://issues.apache.org/jira/browse/STORM-2126?focusedComm
>> > >>>> entId=16136150&page=com.atlassian.jira.plugin.system.
>> > >>>> issuetabpanels%3Acomment-tabpanel#comment-16136150
>> > >>>>>
>> > >>>>> The yet-to-be-ironed-out solution that Bobby was brainstorming
>> about
>> > >>>> isn't
>> > >>>>> a short term fix as far as I understand it. I believe it to be
>> many
>> > >>> many
>> > >>>>> months (years?) out for it to actually be workable. Per my naive
>> > >>>>> understanding of the proposal, we'd probably have to completely
>> > >>> rewrite
>> > >>>> the
>> > >>>>> Storm-on-Mesos framework. So it's probably the right long-term
>> > >>> solution,
>> > >>>>> but it isn't anything that should impact this discussion.
>> > >>>>>
>> > >>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
>> > >>>> include
>> > >>>>> it into their topology jar, it will also not work with Storm
>> 1.0.x.
>> > It
>> > >>>>> even can't
>> > >>>>> compile.
>> > >>>>>
>> > >>>>> FWIW, I'm pretty sure that I was able to successfully run
>> > >>>>> storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after
>> > >>> shading
>> > >>>>> in storm-core-1.1.x to the topology uber jar. There was *at
>> least*
>> > a
>> > >>>>> change to some timer-related class in storm-core in 1.1.x
>> (something
>> > >>>> about
>> > >>>>> milliseconds IIRC -- it's been 1.5 months since I did it, need to
>> > >>> revisit
>> > >>>>> the process I followed).
>> > >>>>>
>> > >>>>> I'm happy to help with backporting / stomping storm-kafka-client
>> in
>> > >>>> 1.0.x.
>> > >>>>> Maybe we can talk offline about it?
>> > >>>>>
>> > >>>>> - Erik
>> > >>>>>
>> > >>>>> On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com>
>> > >>> wrote:
>> > >>>>>
>> > >>>>>> UPDATE: Looks like we changed some parts of storm-core while
>> fixing
>> > >>>>>> storm-kafka-client issues (especially went in 1.1.0), hence
>> > >>> overwriting
>> > >>>>>> also incurs changes of storm-core. It doesn't look like a big
>> deal
>> > >>> for
>> > >>>>>> 1.1.x-branch, but there looks like needed many changes for
>> > >>>> 1.0.x-branch.
>> > >>>>>>
>> > >>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
>> > >>>> include
>> > >>>>> it
>> > >>>>>> into their topology jar, it will also not work with Storm 1.0.x.
>> It
>> > >>>> even
>> > >>>>>> can't compile.
>> > >>>>>>
>> > >>>>>> 1.0.x version line was long lived (22 months) even we released
>> Storm
>> > >>>>> 1.1.0
>> > >>>>>> at 11 months ago. Instead of struggling 1.0.x-branch to up to
>> date,
>> > >>> I'd
>> > >>>>>> like to suggest that we define 1.0.x-branch as deprecated with
>> > >>> guiding
>> > >>>> to
>> > >>>>>> update to latest 1.1.x version or 1.2.0 (after release), and try
>> to
>> > >>>>> resolve
>> > >>>>>> storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's
>> concern.
>> > >>>>>>
>> > >>>>>> Makes sense? I'll continue working on 1.1.x-branch and update
>> > >>> anyway.
>> > >>>>>>
>> > >>>>>> -Jungtaek Lim (HeartSaVioR)
>> > >>>>>>
>> > >>>>>> 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
>> > >>>>>>
>> > >>>>>>> OK. No more opinion/vote in 5 days. I'll treat consensus was
>> made,
>> > >>>> and
>> > >>>>> go
>> > >>>>>>> ahead making change: overwrite storm-kafka-client 1.2.0 to two
>> > >>>> branches
>> > >>>>>>> 1.1.x/1.0.x.
>> > >>>>>>>
>> > >>>>>>> -Jungtaek Lim (HeartSaVioR)
>> > >>>>>>>
>> > >>>>>>> 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이
>> 작성:
>> > >>>>>>>
>> > >>>>>>>> This discussion got 4 +1 (binding) and no -1. Moreover two
>> active
>> > >>>>>>>> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
>> > >>>>>>>>
>> > >>>>>>>> Do we want to hold on for hearing more voices, or treating
>> above
>> > >>>>>> opinions
>> > >>>>>>>> as consensus and reflect the change?
>> > >>>>>>>>
>> > >>>>>>>> Btw, I think we need to sort out the sequences between two
>> > >>> topics:
>> > >>>>>>>> separating storm-kafka-client as independent release cycle, and
>> > >>>> this.
>> > >>>>> I
>> > >>>>>>>> guess some of us agreed former topic doesn't related to current
>> > >>> RC,
>> > >>>>> but
>> > >>>>>> I
>> > >>>>>>>> think this topic can be (should be) reflected to current RC
>> > >>> ongoing.
>> > >>>>>>>>
>> > >>>>>>>> -Jungtaek Lim (HeartSaVioR)
>> > >>>>>>>>
>> > >>>>>>>> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <
>> > >>> hlouro@hortonworks.com
>> > >>>>>> 님이
>> > >>>>>>>> 작성:
>> > >>>>>>>>
>> > >>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch.
>> > >>>>>>>>> Hugo
>> > >>>>>>>>>
>> > >>>>>>>>>> On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
>> > >>>>>>>>> stigdoessing@gmail.com> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch. Breaking
>> > >>>> semantic
>> > >>>>>>>>>> versioning is really nasty, but I think it is the lesser evil
>> > >>> in
>> > >>>>> this
>> > >>>>>>>>> case.
>> > >>>>>>>>>>
>> > >>>>>>>>>> 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch
>> > >>>>>>>>>>> -Harsha
>> > >>>>>>>>>>> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
>> > >>>>>>>>>>>> Bump up this thread so that we could reach consensus
>> > >>> earlier.
>> > >>>>> Given
>> > >>>>>>>>> that
>> > >>>>>>>>>>> we
>> > >>>>>>>>>>>> got concern related to this, I think it is ideal to release
>> > >>>>>>>>> 1.1.x/1.0.x
>> > >>>>>>>>>>>> with making decision and applying the change if we want.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabhwan@gmail.com
>> > >>>> 님이
>> > >>>> 작성:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> Erik's concern brought from 1.0.6 RC1, because they can't
>> > >>> use
>> > >>>>>> Storm
>> > >>>>>>>>>>> 1.1.0
>> > >>>>>>>>>>>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could
>> > >>>> take
>> > >>>>> an
>> > >>>>>>>>>>>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
>> > >>>>> decide
>> > >>>>>> to
>> > >>>>>>>>>>>>> replace) with Storm 1.0.6, it would be better if we don't
>> > >>>> allow
>> > >>>>>>>>> leaving
>> > >>>>>>>>>>>>> storm-kafka-client in 1.0.x in inconsistent state.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> IMHO, breaking backward compatibility is worse, but
>> leaving
>> > >>>>> broken
>> > >>>>>>>>>>> thing
>> > >>>>>>>>>>>>> is worst. Hence I'm +1 to replace all, with noticing that
>> > >>> it
>> > >>>> may
>> > >>>>>>>>> bring
>> > >>>>>>>>>>>>> backward incompatibility in release announce.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> -Jungtaek Lim (HeartSaVioR)
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <
>> > >>> ptgoetz@gmail.com
>> > >>>>> 님이
>> > >>>>>> 작성:
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> As I mentioned else thread I’m open to this but would
>> > >>> defer
>> > >>>> to
>> > >>>>>>>>>>> community
>> > >>>>>>>>>>>>>> consensus.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> If there’s concern about doing this for 1.0.x, one option
>> > >>>> would
>> > >>>>>> be
>> > >>>>>>>>>>> skip
>> > >>>>>>>>>>>>>> that version line and only apply it to 1.2.0 and 1.1.x.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> -Taylor
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <
>> > >>>> kabhwan@gmail.com
>> > >>>>>>
>> > >>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Hi devs,
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> This is initial post to separate out discussion topic
>> > >>> from
>> > >>>>> vote
>> > >>>>>>>>>>> thread,
>> > >>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>> continue discussing.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Background of the topic:
>> > >>>>>>>>>>>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
>> > >>>>>>>>>>> (relatively)
>> > >>>>>>>>>>>>>>> 2. We would avoid to port back patches to 1.1.x and
>> 1.0.x
>> > >>>>>> because
>> > >>>>>>>>>>>>>> they're
>> > >>>>>>>>>>>>>>> diverged too much.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Downside:
>> > >>>>>>>>>>>>>>> Backward compatibility might be broken for 1.1.x and
>> > >>> 1.0.x.
>> > >>>>> Not
>> > >>>>>>>>>>> sure for
>> > >>>>>>>>>>>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
>> > >>>>> version
>> > >>>>>> is
>> > >>>>>>>>>>>>>>> different, and if my memory is right, we already applied
>> > >>>>>> backward
>> > >>>>>>>>>>>>>>> incompatible change into storm-kafka-client 1.1.0.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Please put your opinion regarding topic. You're
>> > >>> encouraged
>> > >>>> to
>> > >>>>>> copy
>> > >>>>>>>>>>> your
>> > >>>>>>>>>>>>>>> previous post in vote thread which helps to centralize
>> > >>>>> opinions
>> > >>>>>> in
>> > >>>>>>>>>>>>>> current
>> > >>>>>>>>>>>>>>> thread.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Thanks,
>> > >>>>>>>>>>>>>>> Jungtaek Lim (HeartSaVioR)
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>
>> > >>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> >
>>
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Jungtaek Lim <ka...@gmail.com>.
Update: I just merged the patch for 1.1.x-branch so release phase of Storm
1.1.2 can be restarted. Patch for 1.0.x-branch from Erik is available and
got some +1s but waiting for 24hrs rule.
2018년 2월 7일 (수) 오전 5:03, Stig Rohde Døssing <st...@gmail.com>님이 작성:
> Took a look at backporting to 1.0.x. We'll have to update the time
> simulation code (Time.java in storm-core) to support nanoseconds, as Erik
> noted, but this isn't a breaking change and only affects tests.
>
> This PR https://github.com/apache/storm/pull/1995/files#diff-
> 72647db30ffd6005dc01c4d1f75d2c68 made a breaking change to
> IOpaquePartitionedTridentSpoutExecutor, so we'll have to do the same on
> 1.0.x.
>
> 2018-02-06 19:13 GMT+01:00 P. Taylor Goetz <pt...@gmail.com>:
>
> > Just a heads up: While this gets sorted out I’m going to proceed with a
> > 1.2.0 RC.
> >
> > -Taylor
> >
> > > On Feb 5, 2018, at 10:46 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> > >
> > > UPDATE: Submitted a pull request https://github.com/apache/
> > storm/pull/2549 for
> > > STORM-2936 (against 1.1.x-branch)
> > >
> > > Erik, please change the status to "IN PROGRESS" if someone is working
> > on. I
> > > would find the free time and just do it if there's no one working in
> > > progress.
> > >
> > > 2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >
> > >> Thanks for quick response Erik!
> > >>
> > >> Just filed two issues :
> > >> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
> > >> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
> > >>
> > >> We have another discussion around making storm-kafka-client be
> > experiment
> > >> of managing separately (independent of Storm release). So the three
> > >> versions which are in release phase might be the last releases of
> > >> "battery-included" of storm-kafka-client if our experiment works well.
> > >>
> > >> If we would want to make the change for storm-kafka-client, it might
> be
> > >> better to put the change and release before start experimenting, but
> > that's
> > >> just a thought on my own. In opposite way, we could even start
> > experiment
> > >> and make change of storm-core of 1.0.x-branch to be compatible with
> that
> > >> version of storm-kafka-client. We could even do it for 1.1.x-branch,
> but
> > >> the change is almost done so it doesn't look like needed to postpone
> it.
> > >>
> > >> Would like to here everyone's voice on this.
> > >>
> > >> -Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers <eweathers@groupon.com.invalid
> > >님이
> > >> 작성:
> > >>
> > >>> Thanks for the quick response Jungtaek!
> > >>>
> > >>> Yes, my teammates and myself would like to help on this. Is there an
> > >>> existing JIRA for the work you've been doing on the other branches?
> > >>>
> > >>> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
> > >>> quickly when the backport is done, if that is amenable.
> > >>> That strategy also might be cleaner since it would avoid other
> changes
> > in
> > >>> 1.0.6 being lumped together with this.
> > >>>
> > >>> - Erik
> > >>>
> > >>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <ka...@gmail.com>
> > wrote:
> > >>>
> > >>>> UPDATE: I've finished working on overwriting storm-kafka-client
> > >>> 1.x-branch
> > >>>> to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork
> > first
> > >>> to
> > >>>> trigger Travis CI to see how the build goes well.
> > >>>>
> > >>>> https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
> > >>>> 612e87da8589f5723f05218a
> > >>>> https://travis-ci.org/HeartSaVioR/storm/builds/337819430
> > >>>>
> > >>>> Thanks for the input regarding 1.0.x version, Erik. I guess then we
> > >>> have no
> > >>>> alternative here: someone has to fix storm-kafka-client as well as
> > >>>> storm-core, since including shaded storm-core doesn't make sense for
> > >>>> official Storm release.
> > >>>>
> > >>>> I guess it doesn't take many hour(s), hence may not worth to sync
> and
> > >>> talk
> > >>>> offline. I just wanted to judge whether we are OK to make change of
> > >>>> storm-core in bugfix version lines, but maybe the judgement itself
> can
> > >>> be
> > >>>> possible after finishing the change, so I'll just go ahead making
> the
> > >>>> change.
> > >>>> Since this is blocking release candidate, we should get it ASAP.
> > That's
> > >>> why
> > >>>> I'm eager to go ahead making the change. If you could spend time now
> > >>>> helping with making the change ASAP, please leave short notice
> (maybe
> > >>> with
> > >>>> JIRA issue?) and go ahead.
> > >>>>
> > >>>> Thanks,
> > >>>> Jungtaek Lim (HeartSaVioR)
> > >>>>
> > >>>> 2018년 2월 6일 (화) 오전 9:41, Erik Weathers
> <eweathers@groupon.com.invalid
> > >>>> 님이
> > >>>> 작성:
> > >>>>
> > >>>>> hey Jungtaek,
> > >>>>>
> > >>>>> Thanks for continuing to pursue this!
> > >>>>>
> > >>>>> The issue for Storm not working on Mesos is due to a fundamental
> > >>> change
> > >>>> to
> > >>>>> the core scheduling logic in Storm:
> > >>>>>
> > >>>>> -
> > >>>>>
> > >>>>> https://issues.apache.org/jira/browse/STORM-2126?focusedComm
> > >>>> entId=16136150&page=com.atlassian.jira.plugin.system.
> > >>>> issuetabpanels%3Acomment-tabpanel#comment-16136150
> > >>>>>
> > >>>>> The yet-to-be-ironed-out solution that Bobby was brainstorming
> about
> > >>>> isn't
> > >>>>> a short term fix as far as I understand it. I believe it to be
> many
> > >>> many
> > >>>>> months (years?) out for it to actually be workable. Per my naive
> > >>>>> understanding of the proposal, we'd probably have to completely
> > >>> rewrite
> > >>>> the
> > >>>>> Storm-on-Mesos framework. So it's probably the right long-term
> > >>> solution,
> > >>>>> but it isn't anything that should impact this discussion.
> > >>>>>
> > >>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> > >>>> include
> > >>>>> it into their topology jar, it will also not work with Storm 1.0.x.
> > It
> > >>>>> even can't
> > >>>>> compile.
> > >>>>>
> > >>>>> FWIW, I'm pretty sure that I was able to successfully run
> > >>>>> storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after
> > >>> shading
> > >>>>> in storm-core-1.1.x to the topology uber jar. There was *at
> least*
> > a
> > >>>>> change to some timer-related class in storm-core in 1.1.x
> (something
> > >>>> about
> > >>>>> milliseconds IIRC -- it's been 1.5 months since I did it, need to
> > >>> revisit
> > >>>>> the process I followed).
> > >>>>>
> > >>>>> I'm happy to help with backporting / stomping storm-kafka-client in
> > >>>> 1.0.x.
> > >>>>> Maybe we can talk offline about it?
> > >>>>>
> > >>>>> - Erik
> > >>>>>
> > >>>>> On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>>> UPDATE: Looks like we changed some parts of storm-core while
> fixing
> > >>>>>> storm-kafka-client issues (especially went in 1.1.0), hence
> > >>> overwriting
> > >>>>>> also incurs changes of storm-core. It doesn't look like a big deal
> > >>> for
> > >>>>>> 1.1.x-branch, but there looks like needed many changes for
> > >>>> 1.0.x-branch.
> > >>>>>>
> > >>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> > >>>> include
> > >>>>> it
> > >>>>>> into their topology jar, it will also not work with Storm 1.0.x.
> It
> > >>>> even
> > >>>>>> can't compile.
> > >>>>>>
> > >>>>>> 1.0.x version line was long lived (22 months) even we released
> Storm
> > >>>>> 1.1.0
> > >>>>>> at 11 months ago. Instead of struggling 1.0.x-branch to up to
> date,
> > >>> I'd
> > >>>>>> like to suggest that we define 1.0.x-branch as deprecated with
> > >>> guiding
> > >>>> to
> > >>>>>> update to latest 1.1.x version or 1.2.0 (after release), and try
> to
> > >>>>> resolve
> > >>>>>> storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
> > >>>>>>
> > >>>>>> Makes sense? I'll continue working on 1.1.x-branch and update
> > >>> anyway.
> > >>>>>>
> > >>>>>> -Jungtaek Lim (HeartSaVioR)
> > >>>>>>
> > >>>>>> 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >>>>>>
> > >>>>>>> OK. No more opinion/vote in 5 days. I'll treat consensus was
> made,
> > >>>> and
> > >>>>> go
> > >>>>>>> ahead making change: overwrite storm-kafka-client 1.2.0 to two
> > >>>> branches
> > >>>>>>> 1.1.x/1.0.x.
> > >>>>>>>
> > >>>>>>> -Jungtaek Lim (HeartSaVioR)
> > >>>>>>>
> > >>>>>>> 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >>>>>>>
> > >>>>>>>> This discussion got 4 +1 (binding) and no -1. Moreover two
> active
> > >>>>>>>> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
> > >>>>>>>>
> > >>>>>>>> Do we want to hold on for hearing more voices, or treating above
> > >>>>>> opinions
> > >>>>>>>> as consensus and reflect the change?
> > >>>>>>>>
> > >>>>>>>> Btw, I think we need to sort out the sequences between two
> > >>> topics:
> > >>>>>>>> separating storm-kafka-client as independent release cycle, and
> > >>>> this.
> > >>>>> I
> > >>>>>>>> guess some of us agreed former topic doesn't related to current
> > >>> RC,
> > >>>>> but
> > >>>>>> I
> > >>>>>>>> think this topic can be (should be) reflected to current RC
> > >>> ongoing.
> > >>>>>>>>
> > >>>>>>>> -Jungtaek Lim (HeartSaVioR)
> > >>>>>>>>
> > >>>>>>>> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <
> > >>> hlouro@hortonworks.com
> > >>>>>> 님이
> > >>>>>>>> 작성:
> > >>>>>>>>
> > >>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch.
> > >>>>>>>>> Hugo
> > >>>>>>>>>
> > >>>>>>>>>> On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
> > >>>>>>>>> stigdoessing@gmail.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch. Breaking
> > >>>> semantic
> > >>>>>>>>>> versioning is really nasty, but I think it is the lesser evil
> > >>> in
> > >>>>> this
> > >>>>>>>>> case.
> > >>>>>>>>>>
> > >>>>>>>>>> 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> > >>>>>>>>>>
> > >>>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch
> > >>>>>>>>>>> -Harsha
> > >>>>>>>>>>> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> > >>>>>>>>>>>> Bump up this thread so that we could reach consensus
> > >>> earlier.
> > >>>>> Given
> > >>>>>>>>> that
> > >>>>>>>>>>> we
> > >>>>>>>>>>>> got concern related to this, I think it is ideal to release
> > >>>>>>>>> 1.1.x/1.0.x
> > >>>>>>>>>>>> with making decision and applying the change if we want.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabhwan@gmail.com
> > >>>> 님이
> > >>>> 작성:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Erik's concern brought from 1.0.6 RC1, because they can't
> > >>> use
> > >>>>>> Storm
> > >>>>>>>>>>> 1.1.0
> > >>>>>>>>>>>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could
> > >>>> take
> > >>>>> an
> > >>>>>>>>>>>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
> > >>>>> decide
> > >>>>>> to
> > >>>>>>>>>>>>> replace) with Storm 1.0.6, it would be better if we don't
> > >>>> allow
> > >>>>>>>>> leaving
> > >>>>>>>>>>>>> storm-kafka-client in 1.0.x in inconsistent state.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> IMHO, breaking backward compatibility is worse, but leaving
> > >>>>> broken
> > >>>>>>>>>>> thing
> > >>>>>>>>>>>>> is worst. Hence I'm +1 to replace all, with noticing that
> > >>> it
> > >>>> may
> > >>>>>>>>> bring
> > >>>>>>>>>>>>> backward incompatibility in release announce.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> -Jungtaek Lim (HeartSaVioR)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <
> > >>> ptgoetz@gmail.com
> > >>>>> 님이
> > >>>>>> 작성:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> As I mentioned else thread I’m open to this but would
> > >>> defer
> > >>>> to
> > >>>>>>>>>>> community
> > >>>>>>>>>>>>>> consensus.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If there’s concern about doing this for 1.0.x, one option
> > >>>> would
> > >>>>>> be
> > >>>>>>>>>>> skip
> > >>>>>>>>>>>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -Taylor
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <
> > >>>> kabhwan@gmail.com
> > >>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi devs,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This is initial post to separate out discussion topic
> > >>> from
> > >>>>> vote
> > >>>>>>>>>>> thread,
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>> continue discussing.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Background of the topic:
> > >>>>>>>>>>>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> > >>>>>>>>>>> (relatively)
> > >>>>>>>>>>>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
> > >>>>>> because
> > >>>>>>>>>>>>>> they're
> > >>>>>>>>>>>>>>> diverged too much.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Downside:
> > >>>>>>>>>>>>>>> Backward compatibility might be broken for 1.1.x and
> > >>> 1.0.x.
> > >>>>> Not
> > >>>>>>>>>>> sure for
> > >>>>>>>>>>>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
> > >>>>> version
> > >>>>>> is
> > >>>>>>>>>>>>>>> different, and if my memory is right, we already applied
> > >>>>>> backward
> > >>>>>>>>>>>>>>> incompatible change into storm-kafka-client 1.1.0.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Please put your opinion regarding topic. You're
> > >>> encouraged
> > >>>> to
> > >>>>>> copy
> > >>>>>>>>>>> your
> > >>>>>>>>>>>>>>> previous post in vote thread which helps to centralize
> > >>>>> opinions
> > >>>>>> in
> > >>>>>>>>>>>>>> current
> > >>>>>>>>>>>>>>> thread.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>> Jungtaek Lim (HeartSaVioR)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Stig Rohde Døssing <st...@gmail.com>.
Took a look at backporting to 1.0.x. We'll have to update the time
simulation code (Time.java in storm-core) to support nanoseconds, as Erik
noted, but this isn't a breaking change and only affects tests.
This PR https://github.com/apache/storm/pull/1995/files#diff-
72647db30ffd6005dc01c4d1f75d2c68 made a breaking change to
IOpaquePartitionedTridentSpoutExecutor, so we'll have to do the same on
1.0.x.
2018-02-06 19:13 GMT+01:00 P. Taylor Goetz <pt...@gmail.com>:
> Just a heads up: While this gets sorted out I’m going to proceed with a
> 1.2.0 RC.
>
> -Taylor
>
> > On Feb 5, 2018, at 10:46 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> >
> > UPDATE: Submitted a pull request https://github.com/apache/
> storm/pull/2549 for
> > STORM-2936 (against 1.1.x-branch)
> >
> > Erik, please change the status to "IN PROGRESS" if someone is working
> on. I
> > would find the free time and just do it if there's no one working in
> > progress.
> >
> > 2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> >> Thanks for quick response Erik!
> >>
> >> Just filed two issues :
> >> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
> >> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
> >>
> >> We have another discussion around making storm-kafka-client be
> experiment
> >> of managing separately (independent of Storm release). So the three
> >> versions which are in release phase might be the last releases of
> >> "battery-included" of storm-kafka-client if our experiment works well.
> >>
> >> If we would want to make the change for storm-kafka-client, it might be
> >> better to put the change and release before start experimenting, but
> that's
> >> just a thought on my own. In opposite way, we could even start
> experiment
> >> and make change of storm-core of 1.0.x-branch to be compatible with that
> >> version of storm-kafka-client. We could even do it for 1.1.x-branch, but
> >> the change is almost done so it doesn't look like needed to postpone it.
> >>
> >> Would like to here everyone's voice on this.
> >>
> >> -Jungtaek Lim (HeartSaVioR)
> >>
> >> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers <eweathers@groupon.com.invalid
> >님이
> >> 작성:
> >>
> >>> Thanks for the quick response Jungtaek!
> >>>
> >>> Yes, my teammates and myself would like to help on this. Is there an
> >>> existing JIRA for the work you've been doing on the other branches?
> >>>
> >>> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
> >>> quickly when the backport is done, if that is amenable.
> >>> That strategy also might be cleaner since it would avoid other changes
> in
> >>> 1.0.6 being lumped together with this.
> >>>
> >>> - Erik
> >>>
> >>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <ka...@gmail.com>
> wrote:
> >>>
> >>>> UPDATE: I've finished working on overwriting storm-kafka-client
> >>> 1.x-branch
> >>>> to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork
> first
> >>> to
> >>>> trigger Travis CI to see how the build goes well.
> >>>>
> >>>> https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
> >>>> 612e87da8589f5723f05218a
> >>>> https://travis-ci.org/HeartSaVioR/storm/builds/337819430
> >>>>
> >>>> Thanks for the input regarding 1.0.x version, Erik. I guess then we
> >>> have no
> >>>> alternative here: someone has to fix storm-kafka-client as well as
> >>>> storm-core, since including shaded storm-core doesn't make sense for
> >>>> official Storm release.
> >>>>
> >>>> I guess it doesn't take many hour(s), hence may not worth to sync and
> >>> talk
> >>>> offline. I just wanted to judge whether we are OK to make change of
> >>>> storm-core in bugfix version lines, but maybe the judgement itself can
> >>> be
> >>>> possible after finishing the change, so I'll just go ahead making the
> >>>> change.
> >>>> Since this is blocking release candidate, we should get it ASAP.
> That's
> >>> why
> >>>> I'm eager to go ahead making the change. If you could spend time now
> >>>> helping with making the change ASAP, please leave short notice (maybe
> >>> with
> >>>> JIRA issue?) and go ahead.
> >>>>
> >>>> Thanks,
> >>>> Jungtaek Lim (HeartSaVioR)
> >>>>
> >>>> 2018년 2월 6일 (화) 오전 9:41, Erik Weathers <eweathers@groupon.com.invalid
> >>>> 님이
> >>>> 작성:
> >>>>
> >>>>> hey Jungtaek,
> >>>>>
> >>>>> Thanks for continuing to pursue this!
> >>>>>
> >>>>> The issue for Storm not working on Mesos is due to a fundamental
> >>> change
> >>>> to
> >>>>> the core scheduling logic in Storm:
> >>>>>
> >>>>> -
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/STORM-2126?focusedComm
> >>>> entId=16136150&page=com.atlassian.jira.plugin.system.
> >>>> issuetabpanels%3Acomment-tabpanel#comment-16136150
> >>>>>
> >>>>> The yet-to-be-ironed-out solution that Bobby was brainstorming about
> >>>> isn't
> >>>>> a short term fix as far as I understand it. I believe it to be many
> >>> many
> >>>>> months (years?) out for it to actually be workable. Per my naive
> >>>>> understanding of the proposal, we'd probably have to completely
> >>> rewrite
> >>>> the
> >>>>> Storm-on-Mesos framework. So it's probably the right long-term
> >>> solution,
> >>>>> but it isn't anything that should impact this discussion.
> >>>>>
> >>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> >>>> include
> >>>>> it into their topology jar, it will also not work with Storm 1.0.x.
> It
> >>>>> even can't
> >>>>> compile.
> >>>>>
> >>>>> FWIW, I'm pretty sure that I was able to successfully run
> >>>>> storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after
> >>> shading
> >>>>> in storm-core-1.1.x to the topology uber jar. There was *at least*
> a
> >>>>> change to some timer-related class in storm-core in 1.1.x (something
> >>>> about
> >>>>> milliseconds IIRC -- it's been 1.5 months since I did it, need to
> >>> revisit
> >>>>> the process I followed).
> >>>>>
> >>>>> I'm happy to help with backporting / stomping storm-kafka-client in
> >>>> 1.0.x.
> >>>>> Maybe we can talk offline about it?
> >>>>>
> >>>>> - Erik
> >>>>>
> >>>>> On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> UPDATE: Looks like we changed some parts of storm-core while fixing
> >>>>>> storm-kafka-client issues (especially went in 1.1.0), hence
> >>> overwriting
> >>>>>> also incurs changes of storm-core. It doesn't look like a big deal
> >>> for
> >>>>>> 1.1.x-branch, but there looks like needed many changes for
> >>>> 1.0.x-branch.
> >>>>>>
> >>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> >>>> include
> >>>>> it
> >>>>>> into their topology jar, it will also not work with Storm 1.0.x. It
> >>>> even
> >>>>>> can't compile.
> >>>>>>
> >>>>>> 1.0.x version line was long lived (22 months) even we released Storm
> >>>>> 1.1.0
> >>>>>> at 11 months ago. Instead of struggling 1.0.x-branch to up to date,
> >>> I'd
> >>>>>> like to suggest that we define 1.0.x-branch as deprecated with
> >>> guiding
> >>>> to
> >>>>>> update to latest 1.1.x version or 1.2.0 (after release), and try to
> >>>>> resolve
> >>>>>> storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
> >>>>>>
> >>>>>> Makes sense? I'll continue working on 1.1.x-branch and update
> >>> anyway.
> >>>>>>
> >>>>>> -Jungtaek Lim (HeartSaVioR)
> >>>>>>
> >>>>>> 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>>>>
> >>>>>>> OK. No more opinion/vote in 5 days. I'll treat consensus was made,
> >>>> and
> >>>>> go
> >>>>>>> ahead making change: overwrite storm-kafka-client 1.2.0 to two
> >>>> branches
> >>>>>>> 1.1.x/1.0.x.
> >>>>>>>
> >>>>>>> -Jungtaek Lim (HeartSaVioR)
> >>>>>>>
> >>>>>>> 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>>>>>>
> >>>>>>>> This discussion got 4 +1 (binding) and no -1. Moreover two active
> >>>>>>>> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
> >>>>>>>>
> >>>>>>>> Do we want to hold on for hearing more voices, or treating above
> >>>>>> opinions
> >>>>>>>> as consensus and reflect the change?
> >>>>>>>>
> >>>>>>>> Btw, I think we need to sort out the sequences between two
> >>> topics:
> >>>>>>>> separating storm-kafka-client as independent release cycle, and
> >>>> this.
> >>>>> I
> >>>>>>>> guess some of us agreed former topic doesn't related to current
> >>> RC,
> >>>>> but
> >>>>>> I
> >>>>>>>> think this topic can be (should be) reflected to current RC
> >>> ongoing.
> >>>>>>>>
> >>>>>>>> -Jungtaek Lim (HeartSaVioR)
> >>>>>>>>
> >>>>>>>> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <
> >>> hlouro@hortonworks.com
> >>>>>> 님이
> >>>>>>>> 작성:
> >>>>>>>>
> >>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch.
> >>>>>>>>> Hugo
> >>>>>>>>>
> >>>>>>>>>> On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
> >>>>>>>>> stigdoessing@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch. Breaking
> >>>> semantic
> >>>>>>>>>> versioning is really nasty, but I think it is the lesser evil
> >>> in
> >>>>> this
> >>>>>>>>> case.
> >>>>>>>>>>
> >>>>>>>>>> 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> >>>>>>>>>>
> >>>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch
> >>>>>>>>>>> -Harsha
> >>>>>>>>>>> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> >>>>>>>>>>>> Bump up this thread so that we could reach consensus
> >>> earlier.
> >>>>> Given
> >>>>>>>>> that
> >>>>>>>>>>> we
> >>>>>>>>>>>> got concern related to this, I think it is ideal to release
> >>>>>>>>> 1.1.x/1.0.x
> >>>>>>>>>>>> with making decision and applying the change if we want.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabhwan@gmail.com
> >>>> 님이
> >>>> 작성:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Erik's concern brought from 1.0.6 RC1, because they can't
> >>> use
> >>>>>> Storm
> >>>>>>>>>>> 1.1.0
> >>>>>>>>>>>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could
> >>>> take
> >>>>> an
> >>>>>>>>>>>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
> >>>>> decide
> >>>>>> to
> >>>>>>>>>>>>> replace) with Storm 1.0.6, it would be better if we don't
> >>>> allow
> >>>>>>>>> leaving
> >>>>>>>>>>>>> storm-kafka-client in 1.0.x in inconsistent state.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> IMHO, breaking backward compatibility is worse, but leaving
> >>>>> broken
> >>>>>>>>>>> thing
> >>>>>>>>>>>>> is worst. Hence I'm +1 to replace all, with noticing that
> >>> it
> >>>> may
> >>>>>>>>> bring
> >>>>>>>>>>>>> backward incompatibility in release announce.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Jungtaek Lim (HeartSaVioR)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <
> >>> ptgoetz@gmail.com
> >>>>> 님이
> >>>>>> 작성:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> As I mentioned else thread I’m open to this but would
> >>> defer
> >>>> to
> >>>>>>>>>>> community
> >>>>>>>>>>>>>> consensus.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If there’s concern about doing this for 1.0.x, one option
> >>>> would
> >>>>>> be
> >>>>>>>>>>> skip
> >>>>>>>>>>>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Taylor
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <
> >>>> kabhwan@gmail.com
> >>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi devs,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This is initial post to separate out discussion topic
> >>> from
> >>>>> vote
> >>>>>>>>>>> thread,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> continue discussing.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Background of the topic:
> >>>>>>>>>>>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> >>>>>>>>>>> (relatively)
> >>>>>>>>>>>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
> >>>>>> because
> >>>>>>>>>>>>>> they're
> >>>>>>>>>>>>>>> diverged too much.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Downside:
> >>>>>>>>>>>>>>> Backward compatibility might be broken for 1.1.x and
> >>> 1.0.x.
> >>>>> Not
> >>>>>>>>>>> sure for
> >>>>>>>>>>>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
> >>>>> version
> >>>>>> is
> >>>>>>>>>>>>>>> different, and if my memory is right, we already applied
> >>>>>> backward
> >>>>>>>>>>>>>>> incompatible change into storm-kafka-client 1.1.0.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Please put your opinion regarding topic. You're
> >>> encouraged
> >>>> to
> >>>>>> copy
> >>>>>>>>>>> your
> >>>>>>>>>>>>>>> previous post in vote thread which helps to centralize
> >>>>> opinions
> >>>>>> in
> >>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>> thread.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Just a heads up: While this gets sorted out I’m going to proceed with a 1.2.0 RC.
-Taylor
> On Feb 5, 2018, at 10:46 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> UPDATE: Submitted a pull request https://github.com/apache/storm/pull/2549 for
> STORM-2936 (against 1.1.x-branch)
>
> Erik, please change the status to "IN PROGRESS" if someone is working on. I
> would find the free time and just do it if there's no one working in
> progress.
>
> 2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
>> Thanks for quick response Erik!
>>
>> Just filed two issues :
>> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
>> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
>>
>> We have another discussion around making storm-kafka-client be experiment
>> of managing separately (independent of Storm release). So the three
>> versions which are in release phase might be the last releases of
>> "battery-included" of storm-kafka-client if our experiment works well.
>>
>> If we would want to make the change for storm-kafka-client, it might be
>> better to put the change and release before start experimenting, but that's
>> just a thought on my own. In opposite way, we could even start experiment
>> and make change of storm-core of 1.0.x-branch to be compatible with that
>> version of storm-kafka-client. We could even do it for 1.1.x-branch, but
>> the change is almost done so it doesn't look like needed to postpone it.
>>
>> Would like to here everyone's voice on this.
>>
>> -Jungtaek Lim (HeartSaVioR)
>>
>> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers <ew...@groupon.com.invalid>님이
>> 작성:
>>
>>> Thanks for the quick response Jungtaek!
>>>
>>> Yes, my teammates and myself would like to help on this. Is there an
>>> existing JIRA for the work you've been doing on the other branches?
>>>
>>> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
>>> quickly when the backport is done, if that is amenable.
>>> That strategy also might be cleaner since it would avoid other changes in
>>> 1.0.6 being lumped together with this.
>>>
>>> - Erik
>>>
>>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>>>
>>>> UPDATE: I've finished working on overwriting storm-kafka-client
>>> 1.x-branch
>>>> to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first
>>> to
>>>> trigger Travis CI to see how the build goes well.
>>>>
>>>> https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
>>>> 612e87da8589f5723f05218a
>>>> https://travis-ci.org/HeartSaVioR/storm/builds/337819430
>>>>
>>>> Thanks for the input regarding 1.0.x version, Erik. I guess then we
>>> have no
>>>> alternative here: someone has to fix storm-kafka-client as well as
>>>> storm-core, since including shaded storm-core doesn't make sense for
>>>> official Storm release.
>>>>
>>>> I guess it doesn't take many hour(s), hence may not worth to sync and
>>> talk
>>>> offline. I just wanted to judge whether we are OK to make change of
>>>> storm-core in bugfix version lines, but maybe the judgement itself can
>>> be
>>>> possible after finishing the change, so I'll just go ahead making the
>>>> change.
>>>> Since this is blocking release candidate, we should get it ASAP. That's
>>> why
>>>> I'm eager to go ahead making the change. If you could spend time now
>>>> helping with making the change ASAP, please leave short notice (maybe
>>> with
>>>> JIRA issue?) and go ahead.
>>>>
>>>> Thanks,
>>>> Jungtaek Lim (HeartSaVioR)
>>>>
>>>> 2018년 2월 6일 (화) 오전 9:41, Erik Weathers <eweathers@groupon.com.invalid
>>>> 님이
>>>> 작성:
>>>>
>>>>> hey Jungtaek,
>>>>>
>>>>> Thanks for continuing to pursue this!
>>>>>
>>>>> The issue for Storm not working on Mesos is due to a fundamental
>>> change
>>>> to
>>>>> the core scheduling logic in Storm:
>>>>>
>>>>> -
>>>>>
>>>>> https://issues.apache.org/jira/browse/STORM-2126?focusedComm
>>>> entId=16136150&page=com.atlassian.jira.plugin.system.
>>>> issuetabpanels%3Acomment-tabpanel#comment-16136150
>>>>>
>>>>> The yet-to-be-ironed-out solution that Bobby was brainstorming about
>>>> isn't
>>>>> a short term fix as far as I understand it. I believe it to be many
>>> many
>>>>> months (years?) out for it to actually be workable. Per my naive
>>>>> understanding of the proposal, we'd probably have to completely
>>> rewrite
>>>> the
>>>>> Storm-on-Mesos framework. So it's probably the right long-term
>>> solution,
>>>>> but it isn't anything that should impact this discussion.
>>>>>
>>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
>>>> include
>>>>> it into their topology jar, it will also not work with Storm 1.0.x. It
>>>>> even can't
>>>>> compile.
>>>>>
>>>>> FWIW, I'm pretty sure that I was able to successfully run
>>>>> storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after
>>> shading
>>>>> in storm-core-1.1.x to the topology uber jar. There was *at least* a
>>>>> change to some timer-related class in storm-core in 1.1.x (something
>>>> about
>>>>> milliseconds IIRC -- it's been 1.5 months since I did it, need to
>>> revisit
>>>>> the process I followed).
>>>>>
>>>>> I'm happy to help with backporting / stomping storm-kafka-client in
>>>> 1.0.x.
>>>>> Maybe we can talk offline about it?
>>>>>
>>>>> - Erik
>>>>>
>>>>> On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com>
>>> wrote:
>>>>>
>>>>>> UPDATE: Looks like we changed some parts of storm-core while fixing
>>>>>> storm-kafka-client issues (especially went in 1.1.0), hence
>>> overwriting
>>>>>> also incurs changes of storm-core. It doesn't look like a big deal
>>> for
>>>>>> 1.1.x-branch, but there looks like needed many changes for
>>>> 1.0.x-branch.
>>>>>>
>>>>>> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
>>>> include
>>>>> it
>>>>>> into their topology jar, it will also not work with Storm 1.0.x. It
>>>> even
>>>>>> can't compile.
>>>>>>
>>>>>> 1.0.x version line was long lived (22 months) even we released Storm
>>>>> 1.1.0
>>>>>> at 11 months ago. Instead of struggling 1.0.x-branch to up to date,
>>> I'd
>>>>>> like to suggest that we define 1.0.x-branch as deprecated with
>>> guiding
>>>> to
>>>>>> update to latest 1.1.x version or 1.2.0 (after release), and try to
>>>>> resolve
>>>>>> storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
>>>>>>
>>>>>> Makes sense? I'll continue working on 1.1.x-branch and update
>>> anyway.
>>>>>>
>>>>>> -Jungtaek Lim (HeartSaVioR)
>>>>>>
>>>>>> 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>>>
>>>>>>> OK. No more opinion/vote in 5 days. I'll treat consensus was made,
>>>> and
>>>>> go
>>>>>>> ahead making change: overwrite storm-kafka-client 1.2.0 to two
>>>> branches
>>>>>>> 1.1.x/1.0.x.
>>>>>>>
>>>>>>> -Jungtaek Lim (HeartSaVioR)
>>>>>>>
>>>>>>> 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>>>>
>>>>>>>> This discussion got 4 +1 (binding) and no -1. Moreover two active
>>>>>>>> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
>>>>>>>>
>>>>>>>> Do we want to hold on for hearing more voices, or treating above
>>>>>> opinions
>>>>>>>> as consensus and reflect the change?
>>>>>>>>
>>>>>>>> Btw, I think we need to sort out the sequences between two
>>> topics:
>>>>>>>> separating storm-kafka-client as independent release cycle, and
>>>> this.
>>>>> I
>>>>>>>> guess some of us agreed former topic doesn't related to current
>>> RC,
>>>>> but
>>>>>> I
>>>>>>>> think this topic can be (should be) reflected to current RC
>>> ongoing.
>>>>>>>>
>>>>>>>> -Jungtaek Lim (HeartSaVioR)
>>>>>>>>
>>>>>>>> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <
>>> hlouro@hortonworks.com
>>>>>> 님이
>>>>>>>> 작성:
>>>>>>>>
>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch.
>>>>>>>>> Hugo
>>>>>>>>>
>>>>>>>>>> On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
>>>>>>>>> stigdoessing@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch. Breaking
>>>> semantic
>>>>>>>>>> versioning is really nasty, but I think it is the lesser evil
>>> in
>>>>> this
>>>>>>>>> case.
>>>>>>>>>>
>>>>>>>>>> 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
>>>>>>>>>>
>>>>>>>>>>> +1 to replace storm-kafka-client in 1.0.x branch
>>>>>>>>>>> -Harsha
>>>>>>>>>>> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
>>>>>>>>>>>> Bump up this thread so that we could reach consensus
>>> earlier.
>>>>> Given
>>>>>>>>> that
>>>>>>>>>>> we
>>>>>>>>>>>> got concern related to this, I think it is ideal to release
>>>>>>>>> 1.1.x/1.0.x
>>>>>>>>>>>> with making decision and applying the change if we want.
>>>>>>>>>>>>
>>>>>>>>>>>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabhwan@gmail.com
>>>> 님이
>>>> 작성:
>>>>>>>>>>>>
>>>>>>>>>>>>> Erik's concern brought from 1.0.6 RC1, because they can't
>>> use
>>>>>> Storm
>>>>>>>>>>> 1.1.0
>>>>>>>>>>>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could
>>>> take
>>>>> an
>>>>>>>>>>>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
>>>>> decide
>>>>>> to
>>>>>>>>>>>>> replace) with Storm 1.0.6, it would be better if we don't
>>>> allow
>>>>>>>>> leaving
>>>>>>>>>>>>> storm-kafka-client in 1.0.x in inconsistent state.
>>>>>>>>>>>>>
>>>>>>>>>>>>> IMHO, breaking backward compatibility is worse, but leaving
>>>>> broken
>>>>>>>>>>> thing
>>>>>>>>>>>>> is worst. Hence I'm +1 to replace all, with noticing that
>>> it
>>>> may
>>>>>>>>> bring
>>>>>>>>>>>>> backward incompatibility in release announce.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Jungtaek Lim (HeartSaVioR)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <
>>> ptgoetz@gmail.com
>>>>> 님이
>>>>>> 작성:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> As I mentioned else thread I’m open to this but would
>>> defer
>>>> to
>>>>>>>>>>> community
>>>>>>>>>>>>>> consensus.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If there’s concern about doing this for 1.0.x, one option
>>>> would
>>>>>> be
>>>>>>>>>>> skip
>>>>>>>>>>>>>> that version line and only apply it to 1.2.0 and 1.1.x.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Taylor
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <
>>>> kabhwan@gmail.com
>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi devs,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is initial post to separate out discussion topic
>>> from
>>>>> vote
>>>>>>>>>>> thread,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> continue discussing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Background of the topic:
>>>>>>>>>>>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
>>>>>>>>>>> (relatively)
>>>>>>>>>>>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
>>>>>> because
>>>>>>>>>>>>>> they're
>>>>>>>>>>>>>>> diverged too much.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Downside:
>>>>>>>>>>>>>>> Backward compatibility might be broken for 1.1.x and
>>> 1.0.x.
>>>>> Not
>>>>>>>>>>> sure for
>>>>>>>>>>>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
>>>>> version
>>>>>> is
>>>>>>>>>>>>>>> different, and if my memory is right, we already applied
>>>>>> backward
>>>>>>>>>>>>>>> incompatible change into storm-kafka-client 1.1.0.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please put your opinion regarding topic. You're
>>> encouraged
>>>> to
>>>>>> copy
>>>>>>>>>>> your
>>>>>>>>>>>>>>> previous post in vote thread which helps to centralize
>>>>> opinions
>>>>>> in
>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Jungtaek Lim <ka...@gmail.com>.
UPDATE: Submitted a pull request https://github.com/apache/storm/pull/2549 for
STORM-2936 (against 1.1.x-branch)
Erik, please change the status to "IN PROGRESS" if someone is working on. I
would find the free time and just do it if there's no one working in
progress.
2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <ka...@gmail.com>님이 작성:
> Thanks for quick response Erik!
>
> Just filed two issues :
> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
>
> We have another discussion around making storm-kafka-client be experiment
> of managing separately (independent of Storm release). So the three
> versions which are in release phase might be the last releases of
> "battery-included" of storm-kafka-client if our experiment works well.
>
> If we would want to make the change for storm-kafka-client, it might be
> better to put the change and release before start experimenting, but that's
> just a thought on my own. In opposite way, we could even start experiment
> and make change of storm-core of 1.0.x-branch to be compatible with that
> version of storm-kafka-client. We could even do it for 1.1.x-branch, but
> the change is almost done so it doesn't look like needed to postpone it.
>
> Would like to here everyone's voice on this.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers <ew...@groupon.com.invalid>님이
> 작성:
>
>> Thanks for the quick response Jungtaek!
>>
>> Yes, my teammates and myself would like to help on this. Is there an
>> existing JIRA for the work you've been doing on the other branches?
>>
>> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
>> quickly when the backport is done, if that is amenable.
>> That strategy also might be cleaner since it would avoid other changes in
>> 1.0.6 being lumped together with this.
>>
>> - Erik
>>
>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>>
>> > UPDATE: I've finished working on overwriting storm-kafka-client
>> 1.x-branch
>> > to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first
>> to
>> > trigger Travis CI to see how the build goes well.
>> >
>> > https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
>> > 612e87da8589f5723f05218a
>> > https://travis-ci.org/HeartSaVioR/storm/builds/337819430
>> >
>> > Thanks for the input regarding 1.0.x version, Erik. I guess then we
>> have no
>> > alternative here: someone has to fix storm-kafka-client as well as
>> > storm-core, since including shaded storm-core doesn't make sense for
>> > official Storm release.
>> >
>> > I guess it doesn't take many hour(s), hence may not worth to sync and
>> talk
>> > offline. I just wanted to judge whether we are OK to make change of
>> > storm-core in bugfix version lines, but maybe the judgement itself can
>> be
>> > possible after finishing the change, so I'll just go ahead making the
>> > change.
>> > Since this is blocking release candidate, we should get it ASAP. That's
>> why
>> > I'm eager to go ahead making the change. If you could spend time now
>> > helping with making the change ASAP, please leave short notice (maybe
>> with
>> > JIRA issue?) and go ahead.
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>> >
>> > 2018년 2월 6일 (화) 오전 9:41, Erik Weathers <eweathers@groupon.com.invalid
>> >님이
>> > 작성:
>> >
>> > > hey Jungtaek,
>> > >
>> > > Thanks for continuing to pursue this!
>> > >
>> > > The issue for Storm not working on Mesos is due to a fundamental
>> change
>> > to
>> > > the core scheduling logic in Storm:
>> > >
>> > > -
>> > >
>> > > https://issues.apache.org/jira/browse/STORM-2126?focusedComm
>> > entId=16136150&page=com.atlassian.jira.plugin.system.
>> > issuetabpanels%3Acomment-tabpanel#comment-16136150
>> > >
>> > > The yet-to-be-ironed-out solution that Bobby was brainstorming about
>> > isn't
>> > > a short term fix as far as I understand it. I believe it to be many
>> many
>> > > months (years?) out for it to actually be workable. Per my naive
>> > > understanding of the proposal, we'd probably have to completely
>> rewrite
>> > the
>> > > Storm-on-Mesos framework. So it's probably the right long-term
>> solution,
>> > > but it isn't anything that should impact this discussion.
>> > >
>> > > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
>> > include
>> > > it into their topology jar, it will also not work with Storm 1.0.x. It
>> > > even can't
>> > > compile.
>> > >
>> > > FWIW, I'm pretty sure that I was able to successfully run
>> > > storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after
>> shading
>> > > in storm-core-1.1.x to the topology uber jar. There was *at least* a
>> > > change to some timer-related class in storm-core in 1.1.x (something
>> > about
>> > > milliseconds IIRC -- it's been 1.5 months since I did it, need to
>> revisit
>> > > the process I followed).
>> > >
>> > > I'm happy to help with backporting / stomping storm-kafka-client in
>> > 1.0.x.
>> > > Maybe we can talk offline about it?
>> > >
>> > > - Erik
>> > >
>> > > On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com>
>> wrote:
>> > >
>> > > > UPDATE: Looks like we changed some parts of storm-core while fixing
>> > > > storm-kafka-client issues (especially went in 1.1.0), hence
>> overwriting
>> > > > also incurs changes of storm-core. It doesn't look like a big deal
>> for
>> > > > 1.1.x-branch, but there looks like needed many changes for
>> > 1.0.x-branch.
>> > > >
>> > > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
>> > include
>> > > it
>> > > > into their topology jar, it will also not work with Storm 1.0.x. It
>> > even
>> > > > can't compile.
>> > > >
>> > > > 1.0.x version line was long lived (22 months) even we released Storm
>> > > 1.1.0
>> > > > at 11 months ago. Instead of struggling 1.0.x-branch to up to date,
>> I'd
>> > > > like to suggest that we define 1.0.x-branch as deprecated with
>> guiding
>> > to
>> > > > update to latest 1.1.x version or 1.2.0 (after release), and try to
>> > > resolve
>> > > > storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
>> > > >
>> > > > Makes sense? I'll continue working on 1.1.x-branch and update
>> anyway.
>> > > >
>> > > > -Jungtaek Lim (HeartSaVioR)
>> > > >
>> > > > 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
>> > > >
>> > > > > OK. No more opinion/vote in 5 days. I'll treat consensus was made,
>> > and
>> > > go
>> > > > > ahead making change: overwrite storm-kafka-client 1.2.0 to two
>> > branches
>> > > > > 1.1.x/1.0.x.
>> > > > >
>> > > > > -Jungtaek Lim (HeartSaVioR)
>> > > > >
>> > > > > 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
>> > > > >
>> > > > >> This discussion got 4 +1 (binding) and no -1. Moreover two active
>> > > > >> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
>> > > > >>
>> > > > >> Do we want to hold on for hearing more voices, or treating above
>> > > > opinions
>> > > > >> as consensus and reflect the change?
>> > > > >>
>> > > > >> Btw, I think we need to sort out the sequences between two
>> topics:
>> > > > >> separating storm-kafka-client as independent release cycle, and
>> > this.
>> > > I
>> > > > >> guess some of us agreed former topic doesn't related to current
>> RC,
>> > > but
>> > > > I
>> > > > >> think this topic can be (should be) reflected to current RC
>> ongoing.
>> > > > >>
>> > > > >> -Jungtaek Lim (HeartSaVioR)
>> > > > >>
>> > > > >> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <
>> hlouro@hortonworks.com
>> > > >님이
>> > > > >> 작성:
>> > > > >>
>> > > > >>> +1 to replace storm-kafka-client in 1.0.x branch.
>> > > > >>> Hugo
>> > > > >>>
>> > > > >>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
>> > > > >>> stigdoessing@gmail.com> wrote:
>> > > > >>> >
>> > > > >>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking
>> > semantic
>> > > > >>> > versioning is really nasty, but I think it is the lesser evil
>> in
>> > > this
>> > > > >>> case.
>> > > > >>> >
>> > > > >>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
>> > > > >>> >
>> > > > >>> >> +1 to replace storm-kafka-client in 1.0.x branch
>> > > > >>> >> -Harsha
>> > > > >>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
>> > > > >>> >>> Bump up this thread so that we could reach consensus
>> earlier.
>> > > Given
>> > > > >>> that
>> > > > >>> >> we
>> > > > >>> >>> got concern related to this, I think it is ideal to release
>> > > > >>> 1.1.x/1.0.x
>> > > > >>> >>> with making decision and applying the change if we want.
>> > > > >>> >>>
>> > > > >>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabhwan@gmail.com
>> >님이
>> > 작성:
>> > > > >>> >>>
>> > > > >>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't
>> use
>> > > > Storm
>> > > > >>> >> 1.1.0
>> > > > >>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could
>> > take
>> > > an
>> > > > >>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
>> > > decide
>> > > > to
>> > > > >>> >>>> replace) with Storm 1.0.6, it would be better if we don't
>> > allow
>> > > > >>> leaving
>> > > > >>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
>> > > > >>> >>>>
>> > > > >>> >>>> IMHO, breaking backward compatibility is worse, but leaving
>> > > broken
>> > > > >>> >> thing
>> > > > >>> >>>> is worst. Hence I'm +1 to replace all, with noticing that
>> it
>> > may
>> > > > >>> bring
>> > > > >>> >>>> backward incompatibility in release announce.
>> > > > >>> >>>>
>> > > > >>> >>>> -Jungtaek Lim (HeartSaVioR)
>> > > > >>> >>>>
>> > > > >>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <
>> ptgoetz@gmail.com
>> > >님이
>> > > > 작성:
>> > > > >>> >>>>
>> > > > >>> >>>>> As I mentioned else thread I’m open to this but would
>> defer
>> > to
>> > > > >>> >> community
>> > > > >>> >>>>> consensus.
>> > > > >>> >>>>>
>> > > > >>> >>>>> If there’s concern about doing this for 1.0.x, one option
>> > would
>> > > > be
>> > > > >>> >> skip
>> > > > >>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
>> > > > >>> >>>>>
>> > > > >>> >>>>> -Taylor
>> > > > >>> >>>>>
>> > > > >>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <
>> > kabhwan@gmail.com
>> > > >
>> > > > >>> >> wrote:
>> > > > >>> >>>>>>
>> > > > >>> >>>>>> Hi devs,
>> > > > >>> >>>>>>
>> > > > >>> >>>>>> This is initial post to separate out discussion topic
>> from
>> > > vote
>> > > > >>> >> thread,
>> > > > >>> >>>>> and
>> > > > >>> >>>>>> continue discussing.
>> > > > >>> >>>>>>
>> > > > >>> >>>>>> Background of the topic:
>> > > > >>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
>> > > > >>> >> (relatively)
>> > > > >>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
>> > > > because
>> > > > >>> >>>>> they're
>> > > > >>> >>>>>> diverged too much.
>> > > > >>> >>>>>>
>> > > > >>> >>>>>> Downside:
>> > > > >>> >>>>>> Backward compatibility might be broken for 1.1.x and
>> 1.0.x.
>> > > Not
>> > > > >>> >> sure for
>> > > > >>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
>> > > version
>> > > > is
>> > > > >>> >>>>>> different, and if my memory is right, we already applied
>> > > > backward
>> > > > >>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
>> > > > >>> >>>>>>
>> > > > >>> >>>>>> Please put your opinion regarding topic. You're
>> encouraged
>> > to
>> > > > copy
>> > > > >>> >> your
>> > > > >>> >>>>>> previous post in vote thread which helps to centralize
>> > > opinions
>> > > > in
>> > > > >>> >>>>> current
>> > > > >>> >>>>>> thread.
>> > > > >>> >>>>>>
>> > > > >>> >>>>>> Thanks,
>> > > > >>> >>>>>> Jungtaek Lim (HeartSaVioR)
>> > > > >>> >>>>>
>> > > > >>> >>>>>
>> > > > >>> >>
>> > > > >>>
>> > > > >>>
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks for quick response Erik!
Just filed two issues :
https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
We have another discussion around making storm-kafka-client be experiment
of managing separately (independent of Storm release). So the three
versions which are in release phase might be the last releases of
"battery-included" of storm-kafka-client if our experiment works well.
If we would want to make the change for storm-kafka-client, it might be
better to put the change and release before start experimenting, but that's
just a thought on my own. In opposite way, we could even start experiment
and make change of storm-core of 1.0.x-branch to be compatible with that
version of storm-kafka-client. We could even do it for 1.1.x-branch, but
the change is almost done so it doesn't look like needed to postpone it.
Would like to here everyone's voice on this.
-Jungtaek Lim (HeartSaVioR)
2018년 2월 6일 (화) 오전 10:23, Erik Weathers <ew...@groupon.com.invalid>님이
작성:
> Thanks for the quick response Jungtaek!
>
> Yes, my teammates and myself would like to help on this. Is there an
> existing JIRA for the work you've been doing on the other branches?
>
> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
> quickly when the backport is done, if that is amenable.
> That strategy also might be cleaner since it would avoid other changes in
> 1.0.6 being lumped together with this.
>
> - Erik
>
> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> > UPDATE: I've finished working on overwriting storm-kafka-client
> 1.x-branch
> > to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first
> to
> > trigger Travis CI to see how the build goes well.
> >
> > https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
> > 612e87da8589f5723f05218a
> > https://travis-ci.org/HeartSaVioR/storm/builds/337819430
> >
> > Thanks for the input regarding 1.0.x version, Erik. I guess then we have
> no
> > alternative here: someone has to fix storm-kafka-client as well as
> > storm-core, since including shaded storm-core doesn't make sense for
> > official Storm release.
> >
> > I guess it doesn't take many hour(s), hence may not worth to sync and
> talk
> > offline. I just wanted to judge whether we are OK to make change of
> > storm-core in bugfix version lines, but maybe the judgement itself can be
> > possible after finishing the change, so I'll just go ahead making the
> > change.
> > Since this is blocking release candidate, we should get it ASAP. That's
> why
> > I'm eager to go ahead making the change. If you could spend time now
> > helping with making the change ASAP, please leave short notice (maybe
> with
> > JIRA issue?) and go ahead.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 2월 6일 (화) 오전 9:41, Erik Weathers <ew...@groupon.com.invalid>님이
> > 작성:
> >
> > > hey Jungtaek,
> > >
> > > Thanks for continuing to pursue this!
> > >
> > > The issue for Storm not working on Mesos is due to a fundamental change
> > to
> > > the core scheduling logic in Storm:
> > >
> > > -
> > >
> > > https://issues.apache.org/jira/browse/STORM-2126?focusedComm
> > entId=16136150&page=com.atlassian.jira.plugin.system.
> > issuetabpanels%3Acomment-tabpanel#comment-16136150
> > >
> > > The yet-to-be-ironed-out solution that Bobby was brainstorming about
> > isn't
> > > a short term fix as far as I understand it. I believe it to be many
> many
> > > months (years?) out for it to actually be workable. Per my naive
> > > understanding of the proposal, we'd probably have to completely rewrite
> > the
> > > Storm-on-Mesos framework. So it's probably the right long-term
> solution,
> > > but it isn't anything that should impact this discussion.
> > >
> > > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> > include
> > > it into their topology jar, it will also not work with Storm 1.0.x. It
> > > even can't
> > > compile.
> > >
> > > FWIW, I'm pretty sure that I was able to successfully run
> > > storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after
> shading
> > > in storm-core-1.1.x to the topology uber jar. There was *at least* a
> > > change to some timer-related class in storm-core in 1.1.x (something
> > about
> > > milliseconds IIRC -- it's been 1.5 months since I did it, need to
> revisit
> > > the process I followed).
> > >
> > > I'm happy to help with backporting / stomping storm-kafka-client in
> > 1.0.x.
> > > Maybe we can talk offline about it?
> > >
> > > - Erik
> > >
> > > On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com>
> wrote:
> > >
> > > > UPDATE: Looks like we changed some parts of storm-core while fixing
> > > > storm-kafka-client issues (especially went in 1.1.0), hence
> overwriting
> > > > also incurs changes of storm-core. It doesn't look like a big deal
> for
> > > > 1.1.x-branch, but there looks like needed many changes for
> > 1.0.x-branch.
> > > >
> > > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> > include
> > > it
> > > > into their topology jar, it will also not work with Storm 1.0.x. It
> > even
> > > > can't compile.
> > > >
> > > > 1.0.x version line was long lived (22 months) even we released Storm
> > > 1.1.0
> > > > at 11 months ago. Instead of struggling 1.0.x-branch to up to date,
> I'd
> > > > like to suggest that we define 1.0.x-branch as deprecated with
> guiding
> > to
> > > > update to latest 1.1.x version or 1.2.0 (after release), and try to
> > > resolve
> > > > storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
> > > >
> > > > Makes sense? I'll continue working on 1.1.x-branch and update anyway.
> > > >
> > > > -Jungtaek Lim (HeartSaVioR)
> > > >
> > > > 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > > >
> > > > > OK. No more opinion/vote in 5 days. I'll treat consensus was made,
> > and
> > > go
> > > > > ahead making change: overwrite storm-kafka-client 1.2.0 to two
> > branches
> > > > > 1.1.x/1.0.x.
> > > > >
> > > > > -Jungtaek Lim (HeartSaVioR)
> > > > >
> > > > > 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > > > >
> > > > >> This discussion got 4 +1 (binding) and no -1. Moreover two active
> > > > >> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
> > > > >>
> > > > >> Do we want to hold on for hearing more voices, or treating above
> > > > opinions
> > > > >> as consensus and reflect the change?
> > > > >>
> > > > >> Btw, I think we need to sort out the sequences between two topics:
> > > > >> separating storm-kafka-client as independent release cycle, and
> > this.
> > > I
> > > > >> guess some of us agreed former topic doesn't related to current
> RC,
> > > but
> > > > I
> > > > >> think this topic can be (should be) reflected to current RC
> ongoing.
> > > > >>
> > > > >> -Jungtaek Lim (HeartSaVioR)
> > > > >>
> > > > >> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <
> hlouro@hortonworks.com
> > > >님이
> > > > >> 작성:
> > > > >>
> > > > >>> +1 to replace storm-kafka-client in 1.0.x branch.
> > > > >>> Hugo
> > > > >>>
> > > > >>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
> > > > >>> stigdoessing@gmail.com> wrote:
> > > > >>> >
> > > > >>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking
> > semantic
> > > > >>> > versioning is really nasty, but I think it is the lesser evil
> in
> > > this
> > > > >>> case.
> > > > >>> >
> > > > >>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> > > > >>> >
> > > > >>> >> +1 to replace storm-kafka-client in 1.0.x branch
> > > > >>> >> -Harsha
> > > > >>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> > > > >>> >>> Bump up this thread so that we could reach consensus earlier.
> > > Given
> > > > >>> that
> > > > >>> >> we
> > > > >>> >>> got concern related to this, I think it is ideal to release
> > > > >>> 1.1.x/1.0.x
> > > > >>> >>> with making decision and applying the change if we want.
> > > > >>> >>>
> > > > >>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <ka...@gmail.com>님이
> > 작성:
> > > > >>> >>>
> > > > >>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't
> use
> > > > Storm
> > > > >>> >> 1.1.0
> > > > >>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could
> > take
> > > an
> > > > >>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
> > > decide
> > > > to
> > > > >>> >>>> replace) with Storm 1.0.6, it would be better if we don't
> > allow
> > > > >>> leaving
> > > > >>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
> > > > >>> >>>>
> > > > >>> >>>> IMHO, breaking backward compatibility is worse, but leaving
> > > broken
> > > > >>> >> thing
> > > > >>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it
> > may
> > > > >>> bring
> > > > >>> >>>> backward incompatibility in release announce.
> > > > >>> >>>>
> > > > >>> >>>> -Jungtaek Lim (HeartSaVioR)
> > > > >>> >>>>
> > > > >>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <
> ptgoetz@gmail.com
> > >님이
> > > > 작성:
> > > > >>> >>>>
> > > > >>> >>>>> As I mentioned else thread I’m open to this but would defer
> > to
> > > > >>> >> community
> > > > >>> >>>>> consensus.
> > > > >>> >>>>>
> > > > >>> >>>>> If there’s concern about doing this for 1.0.x, one option
> > would
> > > > be
> > > > >>> >> skip
> > > > >>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> > > > >>> >>>>>
> > > > >>> >>>>> -Taylor
> > > > >>> >>>>>
> > > > >>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <
> > kabhwan@gmail.com
> > > >
> > > > >>> >> wrote:
> > > > >>> >>>>>>
> > > > >>> >>>>>> Hi devs,
> > > > >>> >>>>>>
> > > > >>> >>>>>> This is initial post to separate out discussion topic from
> > > vote
> > > > >>> >> thread,
> > > > >>> >>>>> and
> > > > >>> >>>>>> continue discussing.
> > > > >>> >>>>>>
> > > > >>> >>>>>> Background of the topic:
> > > > >>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> > > > >>> >> (relatively)
> > > > >>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
> > > > because
> > > > >>> >>>>> they're
> > > > >>> >>>>>> diverged too much.
> > > > >>> >>>>>>
> > > > >>> >>>>>> Downside:
> > > > >>> >>>>>> Backward compatibility might be broken for 1.1.x and
> 1.0.x.
> > > Not
> > > > >>> >> sure for
> > > > >>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
> > > version
> > > > is
> > > > >>> >>>>>> different, and if my memory is right, we already applied
> > > > backward
> > > > >>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
> > > > >>> >>>>>>
> > > > >>> >>>>>> Please put your opinion regarding topic. You're encouraged
> > to
> > > > copy
> > > > >>> >> your
> > > > >>> >>>>>> previous post in vote thread which helps to centralize
> > > opinions
> > > > in
> > > > >>> >>>>> current
> > > > >>> >>>>>> thread.
> > > > >>> >>>>>>
> > > > >>> >>>>>> Thanks,
> > > > >>> >>>>>> Jungtaek Lim (HeartSaVioR)
> > > > >>> >>>>>
> > > > >>> >>>>>
> > > > >>> >>
> > > > >>>
> > > > >>>
> > > >
> > >
> >
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Erik Weathers <ew...@groupon.com.INVALID>.
Thanks for the quick response Jungtaek!
Yes, my teammates and myself would like to help on this. Is there an
existing JIRA for the work you've been doing on the other branches?
I propose we don't make this block 1.0.6 -- we can just release 1.0.7
quickly when the backport is done, if that is amenable.
That strategy also might be cleaner since it would avoid other changes in
1.0.6 being lumped together with this.
- Erik
On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> UPDATE: I've finished working on overwriting storm-kafka-client 1.x-branch
> to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first to
> trigger Travis CI to see how the build goes well.
>
> https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
> 612e87da8589f5723f05218a
> https://travis-ci.org/HeartSaVioR/storm/builds/337819430
>
> Thanks for the input regarding 1.0.x version, Erik. I guess then we have no
> alternative here: someone has to fix storm-kafka-client as well as
> storm-core, since including shaded storm-core doesn't make sense for
> official Storm release.
>
> I guess it doesn't take many hour(s), hence may not worth to sync and talk
> offline. I just wanted to judge whether we are OK to make change of
> storm-core in bugfix version lines, but maybe the judgement itself can be
> possible after finishing the change, so I'll just go ahead making the
> change.
> Since this is blocking release candidate, we should get it ASAP. That's why
> I'm eager to go ahead making the change. If you could spend time now
> helping with making the change ASAP, please leave short notice (maybe with
> JIRA issue?) and go ahead.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 6일 (화) 오전 9:41, Erik Weathers <ew...@groupon.com.invalid>님이
> 작성:
>
> > hey Jungtaek,
> >
> > Thanks for continuing to pursue this!
> >
> > The issue for Storm not working on Mesos is due to a fundamental change
> to
> > the core scheduling logic in Storm:
> >
> > -
> >
> > https://issues.apache.org/jira/browse/STORM-2126?focusedComm
> entId=16136150&page=com.atlassian.jira.plugin.system.
> issuetabpanels%3Acomment-tabpanel#comment-16136150
> >
> > The yet-to-be-ironed-out solution that Bobby was brainstorming about
> isn't
> > a short term fix as far as I understand it. I believe it to be many many
> > months (years?) out for it to actually be workable. Per my naive
> > understanding of the proposal, we'd probably have to completely rewrite
> the
> > Storm-on-Mesos framework. So it's probably the right long-term solution,
> > but it isn't anything that should impact this discussion.
> >
> > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> include
> > it into their topology jar, it will also not work with Storm 1.0.x. It
> > even can't
> > compile.
> >
> > FWIW, I'm pretty sure that I was able to successfully run
> > storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after shading
> > in storm-core-1.1.x to the topology uber jar. There was *at least* a
> > change to some timer-related class in storm-core in 1.1.x (something
> about
> > milliseconds IIRC -- it's been 1.5 months since I did it, need to revisit
> > the process I followed).
> >
> > I'm happy to help with backporting / stomping storm-kafka-client in
> 1.0.x.
> > Maybe we can talk offline about it?
> >
> > - Erik
> >
> > On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> >
> > > UPDATE: Looks like we changed some parts of storm-core while fixing
> > > storm-kafka-client issues (especially went in 1.1.0), hence overwriting
> > > also incurs changes of storm-core. It doesn't look like a big deal for
> > > 1.1.x-branch, but there looks like needed many changes for
> 1.0.x-branch.
> > >
> > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> include
> > it
> > > into their topology jar, it will also not work with Storm 1.0.x. It
> even
> > > can't compile.
> > >
> > > 1.0.x version line was long lived (22 months) even we released Storm
> > 1.1.0
> > > at 11 months ago. Instead of struggling 1.0.x-branch to up to date, I'd
> > > like to suggest that we define 1.0.x-branch as deprecated with guiding
> to
> > > update to latest 1.1.x version or 1.2.0 (after release), and try to
> > resolve
> > > storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
> > >
> > > Makes sense? I'll continue working on 1.1.x-branch and update anyway.
> > >
> > > -Jungtaek Lim (HeartSaVioR)
> > >
> > > 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >
> > > > OK. No more opinion/vote in 5 days. I'll treat consensus was made,
> and
> > go
> > > > ahead making change: overwrite storm-kafka-client 1.2.0 to two
> branches
> > > > 1.1.x/1.0.x.
> > > >
> > > > -Jungtaek Lim (HeartSaVioR)
> > > >
> > > > 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > > >
> > > >> This discussion got 4 +1 (binding) and no -1. Moreover two active
> > > >> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
> > > >>
> > > >> Do we want to hold on for hearing more voices, or treating above
> > > opinions
> > > >> as consensus and reflect the change?
> > > >>
> > > >> Btw, I think we need to sort out the sequences between two topics:
> > > >> separating storm-kafka-client as independent release cycle, and
> this.
> > I
> > > >> guess some of us agreed former topic doesn't related to current RC,
> > but
> > > I
> > > >> think this topic can be (should be) reflected to current RC ongoing.
> > > >>
> > > >> -Jungtaek Lim (HeartSaVioR)
> > > >>
> > > >> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hlouro@hortonworks.com
> > >님이
> > > >> 작성:
> > > >>
> > > >>> +1 to replace storm-kafka-client in 1.0.x branch.
> > > >>> Hugo
> > > >>>
> > > >>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
> > > >>> stigdoessing@gmail.com> wrote:
> > > >>> >
> > > >>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking
> semantic
> > > >>> > versioning is really nasty, but I think it is the lesser evil in
> > this
> > > >>> case.
> > > >>> >
> > > >>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> > > >>> >
> > > >>> >> +1 to replace storm-kafka-client in 1.0.x branch
> > > >>> >> -Harsha
> > > >>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> > > >>> >>> Bump up this thread so that we could reach consensus earlier.
> > Given
> > > >>> that
> > > >>> >> we
> > > >>> >>> got concern related to this, I think it is ideal to release
> > > >>> 1.1.x/1.0.x
> > > >>> >>> with making decision and applying the change if we want.
> > > >>> >>>
> > > >>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <ka...@gmail.com>님이
> 작성:
> > > >>> >>>
> > > >>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use
> > > Storm
> > > >>> >> 1.1.0
> > > >>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could
> take
> > an
> > > >>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
> > decide
> > > to
> > > >>> >>>> replace) with Storm 1.0.6, it would be better if we don't
> allow
> > > >>> leaving
> > > >>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
> > > >>> >>>>
> > > >>> >>>> IMHO, breaking backward compatibility is worse, but leaving
> > broken
> > > >>> >> thing
> > > >>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it
> may
> > > >>> bring
> > > >>> >>>> backward incompatibility in release announce.
> > > >>> >>>>
> > > >>> >>>> -Jungtaek Lim (HeartSaVioR)
> > > >>> >>>>
> > > >>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgoetz@gmail.com
> >님이
> > > 작성:
> > > >>> >>>>
> > > >>> >>>>> As I mentioned else thread I’m open to this but would defer
> to
> > > >>> >> community
> > > >>> >>>>> consensus.
> > > >>> >>>>>
> > > >>> >>>>> If there’s concern about doing this for 1.0.x, one option
> would
> > > be
> > > >>> >> skip
> > > >>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> > > >>> >>>>>
> > > >>> >>>>> -Taylor
> > > >>> >>>>>
> > > >>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <
> kabhwan@gmail.com
> > >
> > > >>> >> wrote:
> > > >>> >>>>>>
> > > >>> >>>>>> Hi devs,
> > > >>> >>>>>>
> > > >>> >>>>>> This is initial post to separate out discussion topic from
> > vote
> > > >>> >> thread,
> > > >>> >>>>> and
> > > >>> >>>>>> continue discussing.
> > > >>> >>>>>>
> > > >>> >>>>>> Background of the topic:
> > > >>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> > > >>> >> (relatively)
> > > >>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
> > > because
> > > >>> >>>>> they're
> > > >>> >>>>>> diverged too much.
> > > >>> >>>>>>
> > > >>> >>>>>> Downside:
> > > >>> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x.
> > Not
> > > >>> >> sure for
> > > >>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
> > version
> > > is
> > > >>> >>>>>> different, and if my memory is right, we already applied
> > > backward
> > > >>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
> > > >>> >>>>>>
> > > >>> >>>>>> Please put your opinion regarding topic. You're encouraged
> to
> > > copy
> > > >>> >> your
> > > >>> >>>>>> previous post in vote thread which helps to centralize
> > opinions
> > > in
> > > >>> >>>>> current
> > > >>> >>>>>> thread.
> > > >>> >>>>>>
> > > >>> >>>>>> Thanks,
> > > >>> >>>>>> Jungtaek Lim (HeartSaVioR)
> > > >>> >>>>>
> > > >>> >>>>>
> > > >>> >>
> > > >>>
> > > >>>
> > >
> >
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Jungtaek Lim <ka...@gmail.com>.
UPDATE: I've finished working on overwriting storm-kafka-client 1.x-branch
to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first to
trigger Travis CI to see how the build goes well.
https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66612e87da8589f5723f05218a
https://travis-ci.org/HeartSaVioR/storm/builds/337819430
Thanks for the input regarding 1.0.x version, Erik. I guess then we have no
alternative here: someone has to fix storm-kafka-client as well as
storm-core, since including shaded storm-core doesn't make sense for
official Storm release.
I guess it doesn't take many hour(s), hence may not worth to sync and talk
offline. I just wanted to judge whether we are OK to make change of
storm-core in bugfix version lines, but maybe the judgement itself can be
possible after finishing the change, so I'll just go ahead making the
change.
Since this is blocking release candidate, we should get it ASAP. That's why
I'm eager to go ahead making the change. If you could spend time now
helping with making the change ASAP, please leave short notice (maybe with
JIRA issue?) and go ahead.
Thanks,
Jungtaek Lim (HeartSaVioR)
2018년 2월 6일 (화) 오전 9:41, Erik Weathers <ew...@groupon.com.invalid>님이 작성:
> hey Jungtaek,
>
> Thanks for continuing to pursue this!
>
> The issue for Storm not working on Mesos is due to a fundamental change to
> the core scheduling logic in Storm:
>
> -
>
> https://issues.apache.org/jira/browse/STORM-2126?focusedCommentId=16136150&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16136150
>
> The yet-to-be-ironed-out solution that Bobby was brainstorming about isn't
> a short term fix as far as I understand it. I believe it to be many many
> months (years?) out for it to actually be workable. Per my naive
> understanding of the proposal, we'd probably have to completely rewrite the
> Storm-on-Mesos framework. So it's probably the right long-term solution,
> but it isn't anything that should impact this discussion.
>
> > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include
> it into their topology jar, it will also not work with Storm 1.0.x. It
> even can't
> compile.
>
> FWIW, I'm pretty sure that I was able to successfully run
> storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after shading
> in storm-core-1.1.x to the topology uber jar. There was *at least* a
> change to some timer-related class in storm-core in 1.1.x (something about
> milliseconds IIRC -- it's been 1.5 months since I did it, need to revisit
> the process I followed).
>
> I'm happy to help with backporting / stomping storm-kafka-client in 1.0.x.
> Maybe we can talk offline about it?
>
> - Erik
>
> On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> > UPDATE: Looks like we changed some parts of storm-core while fixing
> > storm-kafka-client issues (especially went in 1.1.0), hence overwriting
> > also incurs changes of storm-core. It doesn't look like a big deal for
> > 1.1.x-branch, but there looks like needed many changes for 1.0.x-branch.
> >
> > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include
> it
> > into their topology jar, it will also not work with Storm 1.0.x. It even
> > can't compile.
> >
> > 1.0.x version line was long lived (22 months) even we released Storm
> 1.1.0
> > at 11 months ago. Instead of struggling 1.0.x-branch to up to date, I'd
> > like to suggest that we define 1.0.x-branch as deprecated with guiding to
> > update to latest 1.1.x version or 1.2.0 (after release), and try to
> resolve
> > storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
> >
> > Makes sense? I'll continue working on 1.1.x-branch and update anyway.
> >
> > -Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> > > OK. No more opinion/vote in 5 days. I'll treat consensus was made, and
> go
> > > ahead making change: overwrite storm-kafka-client 1.2.0 to two branches
> > > 1.1.x/1.0.x.
> > >
> > > -Jungtaek Lim (HeartSaVioR)
> > >
> > > 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >
> > >> This discussion got 4 +1 (binding) and no -1. Moreover two active
> > >> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
> > >>
> > >> Do we want to hold on for hearing more voices, or treating above
> > opinions
> > >> as consensus and reflect the change?
> > >>
> > >> Btw, I think we need to sort out the sequences between two topics:
> > >> separating storm-kafka-client as independent release cycle, and this.
> I
> > >> guess some of us agreed former topic doesn't related to current RC,
> but
> > I
> > >> think this topic can be (should be) reflected to current RC ongoing.
> > >>
> > >> -Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hlouro@hortonworks.com
> >님이
> > >> 작성:
> > >>
> > >>> +1 to replace storm-kafka-client in 1.0.x branch.
> > >>> Hugo
> > >>>
> > >>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
> > >>> stigdoessing@gmail.com> wrote:
> > >>> >
> > >>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
> > >>> > versioning is really nasty, but I think it is the lesser evil in
> this
> > >>> case.
> > >>> >
> > >>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> > >>> >
> > >>> >> +1 to replace storm-kafka-client in 1.0.x branch
> > >>> >> -Harsha
> > >>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> > >>> >>> Bump up this thread so that we could reach consensus earlier.
> Given
> > >>> that
> > >>> >> we
> > >>> >>> got concern related to this, I think it is ideal to release
> > >>> 1.1.x/1.0.x
> > >>> >>> with making decision and applying the change if we want.
> > >>> >>>
> > >>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <ka...@gmail.com>님이 작성:
> > >>> >>>
> > >>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use
> > Storm
> > >>> >> 1.1.0
> > >>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take
> an
> > >>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we
> decide
> > to
> > >>> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
> > >>> leaving
> > >>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
> > >>> >>>>
> > >>> >>>> IMHO, breaking backward compatibility is worse, but leaving
> broken
> > >>> >> thing
> > >>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may
> > >>> bring
> > >>> >>>> backward incompatibility in release announce.
> > >>> >>>>
> > >>> >>>> -Jungtaek Lim (HeartSaVioR)
> > >>> >>>>
> > >>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <pt...@gmail.com>님이
> > 작성:
> > >>> >>>>
> > >>> >>>>> As I mentioned else thread I’m open to this but would defer to
> > >>> >> community
> > >>> >>>>> consensus.
> > >>> >>>>>
> > >>> >>>>> If there’s concern about doing this for 1.0.x, one option would
> > be
> > >>> >> skip
> > >>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> > >>> >>>>>
> > >>> >>>>> -Taylor
> > >>> >>>>>
> > >>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabhwan@gmail.com
> >
> > >>> >> wrote:
> > >>> >>>>>>
> > >>> >>>>>> Hi devs,
> > >>> >>>>>>
> > >>> >>>>>> This is initial post to separate out discussion topic from
> vote
> > >>> >> thread,
> > >>> >>>>> and
> > >>> >>>>>> continue discussing.
> > >>> >>>>>>
> > >>> >>>>>> Background of the topic:
> > >>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> > >>> >> (relatively)
> > >>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
> > because
> > >>> >>>>> they're
> > >>> >>>>>> diverged too much.
> > >>> >>>>>>
> > >>> >>>>>> Downside:
> > >>> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x.
> Not
> > >>> >> sure for
> > >>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client
> version
> > is
> > >>> >>>>>> different, and if my memory is right, we already applied
> > backward
> > >>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
> > >>> >>>>>>
> > >>> >>>>>> Please put your opinion regarding topic. You're encouraged to
> > copy
> > >>> >> your
> > >>> >>>>>> previous post in vote thread which helps to centralize
> opinions
> > in
> > >>> >>>>> current
> > >>> >>>>>> thread.
> > >>> >>>>>>
> > >>> >>>>>> Thanks,
> > >>> >>>>>> Jungtaek Lim (HeartSaVioR)
> > >>> >>>>>
> > >>> >>>>>
> > >>> >>
> > >>>
> > >>>
> >
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Erik Weathers <ew...@groupon.com.INVALID>.
hey Jungtaek,
Thanks for continuing to pursue this!
The issue for Storm not working on Mesos is due to a fundamental change to
the core scheduling logic in Storm:
-
https://issues.apache.org/jira/browse/STORM-2126?focusedCommentId=16136150&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16136150
The yet-to-be-ironed-out solution that Bobby was brainstorming about isn't
a short term fix as far as I understand it. I believe it to be many many
months (years?) out for it to actually be workable. Per my naive
understanding of the proposal, we'd probably have to completely rewrite the
Storm-on-Mesos framework. So it's probably the right long-term solution,
but it isn't anything that should impact this discussion.
> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include
it into their topology jar, it will also not work with Storm 1.0.x. It
even can't
compile.
FWIW, I'm pretty sure that I was able to successfully run
storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after shading
in storm-core-1.1.x to the topology uber jar. There was *at least* a
change to some timer-related class in storm-core in 1.1.x (something about
milliseconds IIRC -- it's been 1.5 months since I did it, need to revisit
the process I followed).
I'm happy to help with backporting / stomping storm-kafka-client in 1.0.x.
Maybe we can talk offline about it?
- Erik
On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> UPDATE: Looks like we changed some parts of storm-core while fixing
> storm-kafka-client issues (especially went in 1.1.0), hence overwriting
> also incurs changes of storm-core. It doesn't look like a big deal for
> 1.1.x-branch, but there looks like needed many changes for 1.0.x-branch.
>
> The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include it
> into their topology jar, it will also not work with Storm 1.0.x. It even
> can't compile.
>
> 1.0.x version line was long lived (22 months) even we released Storm 1.1.0
> at 11 months ago. Instead of struggling 1.0.x-branch to up to date, I'd
> like to suggest that we define 1.0.x-branch as deprecated with guiding to
> update to latest 1.1.x version or 1.2.0 (after release), and try to resolve
> storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
>
> Makes sense? I'll continue working on 1.1.x-branch and update anyway.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> > OK. No more opinion/vote in 5 days. I'll treat consensus was made, and go
> > ahead making change: overwrite storm-kafka-client 1.2.0 to two branches
> > 1.1.x/1.0.x.
> >
> > -Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >
> >> This discussion got 4 +1 (binding) and no -1. Moreover two active
> >> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
> >>
> >> Do we want to hold on for hearing more voices, or treating above
> opinions
> >> as consensus and reflect the change?
> >>
> >> Btw, I think we need to sort out the sequences between two topics:
> >> separating storm-kafka-client as independent release cycle, and this. I
> >> guess some of us agreed former topic doesn't related to current RC, but
> I
> >> think this topic can be (should be) reflected to current RC ongoing.
> >>
> >> -Jungtaek Lim (HeartSaVioR)
> >>
> >> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hl...@hortonworks.com>님이
> >> 작성:
> >>
> >>> +1 to replace storm-kafka-client in 1.0.x branch.
> >>> Hugo
> >>>
> >>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
> >>> stigdoessing@gmail.com> wrote:
> >>> >
> >>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
> >>> > versioning is really nasty, but I think it is the lesser evil in this
> >>> case.
> >>> >
> >>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> >>> >
> >>> >> +1 to replace storm-kafka-client in 1.0.x branch
> >>> >> -Harsha
> >>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> >>> >>> Bump up this thread so that we could reach consensus earlier. Given
> >>> that
> >>> >> we
> >>> >>> got concern related to this, I think it is ideal to release
> >>> 1.1.x/1.0.x
> >>> >>> with making decision and applying the change if we want.
> >>> >>>
> >>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <ka...@gmail.com>님이 작성:
> >>> >>>
> >>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use
> Storm
> >>> >> 1.1.0
> >>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
> >>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide
> to
> >>> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
> >>> leaving
> >>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
> >>> >>>>
> >>> >>>> IMHO, breaking backward compatibility is worse, but leaving broken
> >>> >> thing
> >>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may
> >>> bring
> >>> >>>> backward incompatibility in release announce.
> >>> >>>>
> >>> >>>> -Jungtaek Lim (HeartSaVioR)
> >>> >>>>
> >>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <pt...@gmail.com>님이
> 작성:
> >>> >>>>
> >>> >>>>> As I mentioned else thread I’m open to this but would defer to
> >>> >> community
> >>> >>>>> consensus.
> >>> >>>>>
> >>> >>>>> If there’s concern about doing this for 1.0.x, one option would
> be
> >>> >> skip
> >>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> >>> >>>>>
> >>> >>>>> -Taylor
> >>> >>>>>
> >>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <ka...@gmail.com>
> >>> >> wrote:
> >>> >>>>>>
> >>> >>>>>> Hi devs,
> >>> >>>>>>
> >>> >>>>>> This is initial post to separate out discussion topic from vote
> >>> >> thread,
> >>> >>>>> and
> >>> >>>>>> continue discussing.
> >>> >>>>>>
> >>> >>>>>> Background of the topic:
> >>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> >>> >> (relatively)
> >>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x
> because
> >>> >>>>> they're
> >>> >>>>>> diverged too much.
> >>> >>>>>>
> >>> >>>>>> Downside:
> >>> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x. Not
> >>> >> sure for
> >>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client version
> is
> >>> >>>>>> different, and if my memory is right, we already applied
> backward
> >>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
> >>> >>>>>>
> >>> >>>>>> Please put your opinion regarding topic. You're encouraged to
> copy
> >>> >> your
> >>> >>>>>> previous post in vote thread which helps to centralize opinions
> in
> >>> >>>>> current
> >>> >>>>>> thread.
> >>> >>>>>>
> >>> >>>>>> Thanks,
> >>> >>>>>> Jungtaek Lim (HeartSaVioR)
> >>> >>>>>
> >>> >>>>>
> >>> >>
> >>>
> >>>
>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Jungtaek Lim <ka...@gmail.com>.
UPDATE: Looks like we changed some parts of storm-core while fixing
storm-kafka-client issues (especially went in 1.1.0), hence overwriting
also incurs changes of storm-core. It doesn't look like a big deal for
1.1.x-branch, but there looks like needed many changes for 1.0.x-branch.
The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include it
into their topology jar, it will also not work with Storm 1.0.x. It even
can't compile.
1.0.x version line was long lived (22 months) even we released Storm 1.1.0
at 11 months ago. Instead of struggling 1.0.x-branch to up to date, I'd
like to suggest that we define 1.0.x-branch as deprecated with guiding to
update to latest 1.1.x version or 1.2.0 (after release), and try to resolve
storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
Makes sense? I'll continue working on 1.1.x-branch and update anyway.
-Jungtaek Lim (HeartSaVioR)
2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <ka...@gmail.com>님이 작성:
> OK. No more opinion/vote in 5 days. I'll treat consensus was made, and go
> ahead making change: overwrite storm-kafka-client 1.2.0 to two branches
> 1.1.x/1.0.x.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
>> This discussion got 4 +1 (binding) and no -1. Moreover two active
>> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
>>
>> Do we want to hold on for hearing more voices, or treating above opinions
>> as consensus and reflect the change?
>>
>> Btw, I think we need to sort out the sequences between two topics:
>> separating storm-kafka-client as independent release cycle, and this. I
>> guess some of us agreed former topic doesn't related to current RC, but I
>> think this topic can be (should be) reflected to current RC ongoing.
>>
>> -Jungtaek Lim (HeartSaVioR)
>>
>> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hl...@hortonworks.com>님이
>> 작성:
>>
>>> +1 to replace storm-kafka-client in 1.0.x branch.
>>> Hugo
>>>
>>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
>>> stigdoessing@gmail.com> wrote:
>>> >
>>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
>>> > versioning is really nasty, but I think it is the lesser evil in this
>>> case.
>>> >
>>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
>>> >
>>> >> +1 to replace storm-kafka-client in 1.0.x branch
>>> >> -Harsha
>>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
>>> >>> Bump up this thread so that we could reach consensus earlier. Given
>>> that
>>> >> we
>>> >>> got concern related to this, I think it is ideal to release
>>> 1.1.x/1.0.x
>>> >>> with making decision and applying the change if we want.
>>> >>>
>>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>> >>>
>>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use Storm
>>> >> 1.1.0
>>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
>>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
>>> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
>>> leaving
>>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
>>> >>>>
>>> >>>> IMHO, breaking backward compatibility is worse, but leaving broken
>>> >> thing
>>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may
>>> bring
>>> >>>> backward incompatibility in release announce.
>>> >>>>
>>> >>>> -Jungtaek Lim (HeartSaVioR)
>>> >>>>
>>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>>> >>>>
>>> >>>>> As I mentioned else thread I’m open to this but would defer to
>>> >> community
>>> >>>>> consensus.
>>> >>>>>
>>> >>>>> If there’s concern about doing this for 1.0.x, one option would be
>>> >> skip
>>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
>>> >>>>>
>>> >>>>> -Taylor
>>> >>>>>
>>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <ka...@gmail.com>
>>> >> wrote:
>>> >>>>>>
>>> >>>>>> Hi devs,
>>> >>>>>>
>>> >>>>>> This is initial post to separate out discussion topic from vote
>>> >> thread,
>>> >>>>> and
>>> >>>>>> continue discussing.
>>> >>>>>>
>>> >>>>>> Background of the topic:
>>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
>>> >> (relatively)
>>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x because
>>> >>>>> they're
>>> >>>>>> diverged too much.
>>> >>>>>>
>>> >>>>>> Downside:
>>> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x. Not
>>> >> sure for
>>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client version is
>>> >>>>>> different, and if my memory is right, we already applied backward
>>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
>>> >>>>>>
>>> >>>>>> Please put your opinion regarding topic. You're encouraged to copy
>>> >> your
>>> >>>>>> previous post in vote thread which helps to centralize opinions in
>>> >>>>> current
>>> >>>>>> thread.
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Jungtaek Lim (HeartSaVioR)
>>> >>>>>
>>> >>>>>
>>> >>
>>>
>>>
Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch /
1.0.x-branch with 1.x-branch
Posted by Jungtaek Lim <ka...@gmail.com>.
OK. No more opinion/vote in 5 days. I'll treat consensus was made, and go
ahead making change: overwrite storm-kafka-client 1.2.0 to two branches
1.1.x/1.0.x.
-Jungtaek Lim (HeartSaVioR)
2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <ka...@gmail.com>님이 작성:
> This discussion got 4 +1 (binding) and no -1. Moreover two active
> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
>
> Do we want to hold on for hearing more voices, or treating above opinions
> as consensus and reflect the change?
>
> Btw, I think we need to sort out the sequences between two topics:
> separating storm-kafka-client as independent release cycle, and this. I
> guess some of us agreed former topic doesn't related to current RC, but I
> think this topic can be (should be) reflected to current RC ongoing.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hl...@hortonworks.com>님이 작성:
>
>> +1 to replace storm-kafka-client in 1.0.x branch.
>> Hugo
>>
>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
>> stigdoessing@gmail.com> wrote:
>> >
>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
>> > versioning is really nasty, but I think it is the lesser evil in this
>> case.
>> >
>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
>> >
>> >> +1 to replace storm-kafka-client in 1.0.x branch
>> >> -Harsha
>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
>> >>> Bump up this thread so that we could reach consensus earlier. Given
>> that
>> >> we
>> >>> got concern related to this, I think it is ideal to release
>> 1.1.x/1.0.x
>> >>> with making decision and applying the change if we want.
>> >>>
>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <ka...@gmail.com>님이 작성:
>> >>>
>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use Storm
>> >> 1.1.0
>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
>> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
>> leaving
>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
>> >>>>
>> >>>> IMHO, breaking backward compatibility is worse, but leaving broken
>> >> thing
>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may
>> bring
>> >>>> backward incompatibility in release announce.
>> >>>>
>> >>>> -Jungtaek Lim (HeartSaVioR)
>> >>>>
>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>> >>>>
>> >>>>> As I mentioned else thread I’m open to this but would defer to
>> >> community
>> >>>>> consensus.
>> >>>>>
>> >>>>> If there’s concern about doing this for 1.0.x, one option would be
>> >> skip
>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
>> >>>>>
>> >>>>> -Taylor
>> >>>>>
>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <ka...@gmail.com>
>> >> wrote:
>> >>>>>>
>> >>>>>> Hi devs,
>> >>>>>>
>> >>>>>> This is initial post to separate out discussion topic from vote
>> >> thread,
>> >>>>> and
>> >>>>>> continue discussing.
>> >>>>>>
>> >>>>>> Background of the topic:
>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
>> >> (relatively)
>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x because
>> >>>>> they're
>> >>>>>> diverged too much.
>> >>>>>>
>> >>>>>> Downside:
>> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x. Not
>> >> sure for
>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client version is
>> >>>>>> different, and if my memory is right, we already applied backward
>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
>> >>>>>>
>> >>>>>> Please put your opinion regarding topic. You're encouraged to copy
>> >> your
>> >>>>>> previous post in vote thread which helps to centralize opinions in
>> >>>>> current
>> >>>>>> thread.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Jungtaek Lim (HeartSaVioR)
>> >>>>>
>> >>>>>
>> >>
>>
>>