You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "P. Taylor Goetz" <pt...@gmail.com> on 2017/06/02 18:39:35 UTC

Re: [DISCUSS] Storm 2.0 Roadmap

I’d like to bump this thread and start a discussion around our next release. Here are my thoughts.

There are a number of important fixes in 1.x-branch so I’d like to consider releasing 1.1.1 soon. I’d appreciate input on any open issues that should be resolved for that release.

I’d like us to consider releasing the metrics improvements in STORM-2153 [1] as version 1.2.0. That work is in the metrics_v2 feature branch right now and would need to be reviewed and merged. That work is against the 1.x-branch right now. I would recommend porting it to master *after* the review/merge since there will likely be changes as a result of the review.

> Maybe related to or not, but would we want to create a new branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?



If wee agree to the above, I would say yes. After the 1.1.1 release, we could create a 1.1.x-branch that would be the maintenance/release branch for that version line. 1.x-branch would then become the target for the 1.2.0 release.

There are a few fixes in the 0.10.x branch that probably warrant a release. After that we may want to back away from that version line a bit so we can focus more on newer versions.

In the past, we’ve shied away form doing “beta” releases, but I’m wondering if we might want to revisit that for the 2.0 release — the idea being that it would give early adopter users a chance to kick the tires on what’s coming in 2.0 and provide feedback, find bugs, etc. to help make the final release more solid. I’m on the fence here and could go either way.

I’d appreciate any input others may have.


Thanks,

-Taylor


[1] https://issues.apache.org/jira/browse/STORM-2153



> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> Maybe related to or not, but would we want to create a new branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> 
> I'm not clear we don't release 1.2 for moving toward to 2.0.0, so hence the
> question.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <hl...@hortonworks.com>님이 작성:
> 
>> +1 for finishing the porting to Java ahead of anything else - it will be a
>> significant milestone. I have a JIRA assigned concerning to the porting. I
>> will work on it for the 2.0 release.
>> 
>> it’s a priority to guarantee no performance regressions. As part of this
>> endeavor, explore an automated (or easy) way to run and assert major
>> performance benchmarks. Ideally any contributor should be able to fairly
>> easily test the impact of changes under certain performance test scenarios.
>> 
>> Beam Runner work should take into account the impact of incorporating new
>> JStorm features and Storm Worker Redesign<
>> https://issues.apache.org/jira/browse/STORM-2284>. Not very efficient to
>> start doing it, to  find out that it will have to chance in face of Storm
>> and worker redesign. That is, it should be done after it’s building blocks
>> are stable.
>> 
>> Thanks,
>> Hugo
>> 
>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org<mailto:
>> arunm@apache.org>> wrote:
>> 
>> +1 to release with the porting completed. I think its mainly the UI server
>> and log viewer that’s pending.
>> 
>> We can start doing the regression and performance tests for whatever is
>> already ported.
>> 
>> If anyone is running the master branch in their pre-prod / prod
>> environments, it will be good to know and give us more confidence.
>> 
>> The other features can be added in follow up releases.
>> 
>> Regards,
>> Arun
>> 
>> 
>> On 3/24/17, 11:47 AM, "Satish Duggana" <satish.duggana@gmail.com<mailto:
>> satish.duggana@gmail.com>> wrote:
>> 
>> +1 to have 2.0 with porting and performance(it should be at least as good
>> as 1.x release) issues addressed
>> 
>> We can target other tasks(mentioned by Taylor and Jungtaek) for 2.x-branch.
>> 
>> 
>> Exactly-once support:
>> While thinking through the exactlyonce support design, it is realized
>> better to avoid acking tuples and implement exactly once by snapshotting
>> barriers. It seems JStorm folks followed similar design, they claim it
>> gives better performance. This feature is essential for beam runner and we
>> can decide on respective approaches though.
>> 
>> Beam Runner
>> Lets hold on this for now and keep it in Storm till 2.x. We should avoid
>> having a minimal beam runner in haste. It is better to address STORM-2284,
>> exactly-once and other windowing enhancements to enable beam runner.
>> 
>> JStorm
>> Agree with Jungtaek on looking at the latest JStorm and align/scope with
>> the features for 2.x.
>> 
>> STORM-2284
>> We may want to look at JStorm worker before working on respective
>> components in this epic to pull appropriate enhancements.
>> 
>> YARN/MESOS
>> Supporting Storm on YARN/Mesos for 2.x.
>> 
>> Thanks,
>> Satish.
>> 
>> 
>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:
>> kabhwan@gmail.com>> wrote:
>> 
>> First of all, +1 to complete only port work and do sanity check (including
>> performance regression), and release.
>> 
>> If we can get STORM-2284 within deterministic time frame (say 2~3 months)
>> that should be great, but if not I'd in favor of postponing that to later
>> 2.x release.
>> 
>> JStorm released their new versions after code donation. So there're more
>> things we could get ideas from, or even adopt from.
>> https://github.com/alibaba/jstorm/blob/master/history.md
>> As you noticed from release note link, we also need to update phase 2 since
>> they already changed what we're planning to do in phase 2. For example,
>> they changed backpressure to end-to-end, and changed to use snapshot rather
>> than acker.
>> May be sure, JStorm pulled many features from today's Storm, like Flux,
>> Windowing, more shuffle groupings, log search, log level change, and so on.
>> 
>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> is due to
>> the
>> limitation of Spout lifecycle (all the things are done in single thread),
>> and STORM-1358 <https://issues.apache.org/jira/browse/STORM-1358>(JStorm's
>> multi-thread Spout) can remedy this (despite that Spout implementation may
>> need to guarantee thread-safety later). It's not a just improvement but
>> close to design concern so would like to address sooner than other things
>> in phase 2.
>> 
>> For Storm SQL side, I've lost progress but major work would be adopting
>> group by with windowing. It was not available from Calcite but will be
>> available at next release (1.12.0).
>> I've filed this to STORM-2405
>> <https://issues.apache.org/jira/browse/STORM-2405>, but windowing & micro
>> batch is not intuitive, so I would like to change the underlying API to
>> stream API in SQL. Also filed this to STORM-2406
>> <https://issues.apache.org/jira/browse/STORM-2406>.
>> 
>> Just 2 cents btw, hopefully I would like to see metrics V2 sooner since we
>> lost metrics even when doing normal operation like restarting worker,
>> rebalancing, and so on. Eventually we need to fight with dynamic scaling,
>> and then metrics will be broken often.
>> 
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>> 
>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <st...@harsha.io>님이 작성:
>> 
>> Storm 2.0 migration to java in itself is a big win and would attract
>> wider
>> community and adoption. So my vote would be to resolve the first 3 items
>> to
>> get a release out.
>> All the other featured mentioned are great to have but shouldn't be
>> blockers for 2.0 release.
>> 
>> -Harsha
>> 
>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <pt...@gmail.com>
>> wrote:
>> 
>> With the 1.1.0 release nearing completion, I’d like to turn our
>> attention
>> to 2.0 and develop a plan for what features, etc. to include.
>> 
>> The following 3 are what I feel are the minimum for a 2.0 release.
>> These
>> could likely be resolved relatively quickly:
>> 
>> * Performance — I’ve not benchmarked the master branch vs. 1.0.x or
>> 1.1.x
>> in a while, but I feel it will be important to make sure there are no
>> performance regressions, and would hope that we actually have a
>> performance
>> improvement over previous versions. To that end (e.g. if there is in
>> fact a
>> performance regression), the proposals that Roshan Naik put together
>> for
>> revising the threading and execution model (STORM-2307) and replacing
>> Disruptor with JCTools (STORM-2306) warrant review and consideration.
>> See
>> also STORM-2284 which is the parent JIRA.
>> 
>> * Finish porting Storm UI to java (STORM-1311)
>> 
>> * Finish porting log viewer to java (STORM-1280)
>> 
>> The following are items that are nice to have in 2.0, but I don’t feel
>> are
>> absolutely necessary for an initial 2.0 release:
>> 
>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because it’s
>> relevant) — Initially there seemed to be a lot of interest in this, but
>> that seems to have trailed off. I spoke with some Beam developers and
>> there
>> seems to be interest from that community as well. Do we want to move
>> that
>> effort to the Beam community, or keep it here? Moving it to the Beam
>> community might lead to better collaboration between projects.
>> 
>> * Bounded Spouts (needed for Beam Runner implementation) — Currently
>> spouts are unbounded, there no end to the stream. Beam has the concept
>> of
>> bounded sources (roughly analogous to batch processing). To support
>> that,
>> we would need to implement a similar concept in Storm. One benefit of
>> such
>> a feature would be the ability to handle both bounded and unbounded
>> workflows in Storm.
>> 
>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers behind
>> this
>> effort. What improvements do you envision for 2.0?
>> 
>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been targeting this
>> for 1.2.0, but it’s designed to be easily portable to master/2.0.
>> 
>> * JStorm Migration — Original outline can be found here [1]. Note a lot
>> of
>> the associated JIRAs below are assigned, but there hasn’t been any
>> recent
>> activity or pull requests, we should probably consider them unassigned
>> and
>> up for grabs.:
>> 
>> * Worker Classloader Isolation (STORM-1338) — Lack of this has been the
>> bane of a lot of Storm users almost since day one. We have largely
>> addressed it by shading/relocating dependencies. It would be great to
>> see
>> this addressed once and for all.
>> 
>> * JStorm back pressure implementation (STORM-1324) — The current back
>> pressure implementation leaves a bit to be desired, and the JStorm
>> approach
>> looks promising, though it also depends on the JStorm concept of
>> “topology
>> master” (STORM-1323), which may have some implications regarding
>> security.
>> 
>> * Dynamic Topology Updates (STORM-1335) — This would provide a command
>> to
>> update topology jars and configuration without stopping the topology,
>> and
>> is well suited to leverage the blobstore. The restart command (that can
>> also update the topology configuration) also looks compelling
>> (STORM-1334).
>> 
>> * Additional Scheduler Implementations (STORM-1320)
>> 
>> * Additional Grouping Implementations (STORM-1328)
>> 
>> 
>> As always I’m open to any opinions and suggestions.
>> 
>> -Taylor
>> 
>> [1]
>> 
>> https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=61328109
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
> On Jun 28, 2017, at 10:48 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> +1.
> If the 1.1 and 1.2 lines start to become difficult to maintain we can look at putting them in maintenance mode too once we have a 2.x release.
> I am a little nervous about merging a new feature into 1.x branch without first going to master, but I hope that it will not be too much work to port it to master, and I trust the devs on that branch to do the right thing.

Porting to master should be easy. It was designed from the start to minimize impact to Clojure code to ease porting to master. I wanted to wait to port to master until after it’s reviewed and merged so any changes that come from review don’t have to be separately ported.


> On a related note we have not done much with feature branches before so I am not sure what we want to do about merging in the new metrics API branch to 1.x.  I know for me I have not had time to keep up with the development work going on there.  I would at least like to have a pull request put up for review before we merge it in.  This would fit with our current bylaws that do not mention feature branches.  If all of the changes have already followed the review process then technically I think it is OK to just merge it in, but I still would like to take some time to look at the changes, and especially the new APIs.

The plan for any of the feature branches has always been the same: relax the commit rules for a feature branch, then when ready, issue a pull request from the feature branch to the target branch for thorough review with all the commit rules in effect. We could certainly codify that in our bylaws, but I think we should try the process out a few times before setting things in stone. If I remember correctly this is the first time a feature branch has reached the point of being ready to be reviewed/merged to a target branch.


> - Bobby

-Taylor

> 
> 
> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> That's great news that metrics work is ready!
> 
> I'm +1 to Taylor's proposal, but in order to respect semantic versioning, I
> propose some modifications from Taylor's proposal:
> 
> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back only
> bug fixes to the 1.1.x-branch
> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
> 
> If we also agree above, I would like to volunteer the back-port work.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 28일 (수) 오전 10:09, Harsha <st...@harsha.io>님이 작성:
> 
>> +1 for above stated approach on releasing 1.2.0 with metrics
>> -Harsha
>> 
>> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
>>> The work on metrics is ready for a pull request to 1.x-branch from the
>>> feature branch. I’ve held off because we haven’t reached consensus on a
>>> path forward with the 1.x release lines .
>>> 
>>> I’d like to propose the following for the 1.x line:
>>> 
>>> 1. Create a branch for 1.2 so we have a branch to review the metrics
>>> stuff.
>>> 2. Release 1.1.1
>>> 3. Review/merge metrics work. Port metrics to master.
>>> 4. Release 1.2.0
>>> 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
>>> (we would only support 1.2.x and 1.1.x which are very closely aligned).
>>> 
>>> Dropping support for 1.0.x line would eliminate the need to maintain one
>>> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
>>> very closely aligned. I just up merged metrics_v2 against 1.x-branch
>>> after a while, and there were no conflicts.
>>> 
>>> That would give us a little more bandwidth to focus on 2.0 and needed bug
>>> fixes to the 1.x line like some of the issues raised with
>>> storm-kafka-client. We could even start releasing alpha/beta versions of
>>> 2.0 in parallel to the steps above.
>>> 
>>> Any thoughts on that approach?
>>> 
>>> -Taylor
>>> 
>>> 
>>>> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>>>> 
>>>> Yes I prefer option 1, but it might depend on the progress of metrics
>> V2.
>>>> If it can be done within predictable near future I'm OK to pick option
>> 2,
>>>> but if not, we may be better to focus releasing 2.0.0 and make it
>> really
>>>> happen.
>>>> 
>>>> Whichever we go, I feel it's time to track remaining work on Storm
>> 2.0.0. I
>>>> found some bugs on master branch so filed issues, and we've remaining
>> port
>>>> work (UI and logviewer). We've some other improvements target for
>> 2.0.0:
>>>> worker redesign, beam integration, and so on, and we don't track its
>>>> progress at all. I don't think we should wait for features which
>> progress
>>>> is not transparent (in other words we don't know when it will be
>> finished).
>>>> 
>>>> - Jungtaek Lim (HeartSaVioR)
>>>> 
>>>> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>님이 작성:
>>>> 
>>>>> Bobby/Jungtaek,
>>>>> 
>>>>> Are you saying you want to forego the 1.2 “metrics_v2” release and
>> include
>>>>> it only in 2.0? (I ask because that work is already based on
>> 1.x-branch,
>>>>> and forward-porting it to master is relatively simple.) I’d kind of
>> like
>>>>> that work go out soon.
>>>>> 
>>>>> If we go with option 1, I would want to see a 2.0 release (even if
>> it’s a
>>>>> “beta” or “preview) before putting the 1.x line into maintenance mode.
>>>>> 
>>>>> -Taylor
>>>>> 
>>>>>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID
>>> 
>>>>> wrote:
>>>>>> 
>>>>>> I see 2 ways to address this.
>>>>>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
>> don't
>>>>> backport anything except bug fixes.2) We backport a lot of the
>> backwards
>>>>> compatible changes from 2.x to 1.x.
>>>>>> My personal preference is 1.  It makes it clear the direction we
>> want to
>>>>> go in.  The biggest issue is that we probably would want to do a 2.x
>>>>> release sooner rather then later.  Even if we don't get all of the
>> features
>>>>> that people want, if we just get a release out we can add in new
>> features
>>>>> if they are backwards compatible, or we can create a 3.x line that
>> would
>>>>> have the breaking changes in it.
>>>>>> 
>>>>>> - Bobby
>>>>>> 
>>>>>> 
>>>>>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
>>>>> kabhwan@gmail.com> wrote:
>>>>>> 
>>>>>> I'd like to bump this again instead of initiating new discussion
>> thread.
>>>>>> 
>>>>>> I had having hard time to create and apply pull requests for both
>> master
>>>>>> and 1.x-branch and that's really painful and sometimes blocker for
>> me to
>>>>> do
>>>>>> merge step.
>>>>>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
>> even
>>>>>> IDE can't switch between the branch smoothly. We didn't even address
>>>>>> checkstyle issue yet, but after addressing, it could be "completely"
>>>>>> diverged. JDK version is another major issue, since the pull requests
>>>>>> targeted for master branch are not checked against JDK 7, and some of
>>>>> them
>>>>>> make some issues regarding JDK version while porting back.
>>>>>> 
>>>>>> So personally I really would like to see the plan for 1.x version
>> line
>>>>>> changed - skipping any minor releases including 1.2.0 - and have epic
>>>>> issue
>>>>>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
>>>>>> proposed plan was having 2.0.0 directly after 1.0.0)
>>>>>> 
>>>>>> Would like to hear everyone's opinions. If we have consensus to not
>>>>> having
>>>>>> any minor releases for 1.x version line, I will not port back
>> non-bugfix
>>>>>> pull requests to 1.x-branch, and guide contributors to create pull
>>>>> requests
>>>>>> against master branch, not 1.x version line.
>>>>>> 
>>>>>> Thanks,
>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>> 
>>>>>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
>>>>> avermeerbergen@gmail.com>님이
>>>>>> 작성:
>>>>>> 
>>>>>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
>> system,
>>>>>>> we're very interested anything that can provide better throughput.
>>>>>>> 
>>>>>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
>>>>>>> 
>>>>>>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>>>>>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
>> be
>>>>>>>> delivered sooner and my in-progress implementation suggests that it
>>>>> will
>>>>>>>> yield substantial latency improvements. The 2.0 beta phase would
>> really
>>>>>>>> help kick the tires on the revised messaging system and the
>> performance
>>>>>>>> improvements will also be a good incentive for trying out the 2.0
>> line.
>>>>>>>> 
>>>>>>>> I notice multiple other bottlenecks that are holding back
>> throughput a
>>>>>>>> lot, which can be addressed in a subsequent 2.x minor release.
>>>>>>>> -roshan
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>     I also would love to see metrics V2 code sooner or later too.
>> If we
>>>>>>>> can get
>>>>>>>>     it before releasing 2.0.0 that will be great, and then maybe we
>>>>> could
>>>>>>>> just
>>>>>>>>     move toward to 2.0.0, not adding any improvements to 1.x version
>>>>>>> line.
>>>>>>>>     (And that's what I would want to.)
>>>>>>>> 
>>>>>>>>     If we would really want to have 1.2.0, I suggest that we make
>> the
>>>>>>> 1.1.1
>>>>>>>>     version correct right now rather than after releasing 1.1.1. We
>>>>> also
>>>>>>>> merged
>>>>>>>>     non-bugfix things to 1.x-branch but that's not what users
>> expect. I
>>>>>>>> agree
>>>>>>>>     that work may be painful, but anyway need to do it.
>>>>>>>> 
>>>>>>>>     - Jungtaek Lim (HeartSaVioR)
>>>>>>>> 
>>>>>>>>     2017년 6월 3일 (토) 오전 3:49, Bobby Evans
>> <evans@yahoo-inc.com.invalid
>>>>>> 님이
>>>>>>>> 작성:
>>>>>>>> 
>>>>>>>>> I would love to see the metrics V2 code come out sooner rather
>>>>> than
>>>>>>>>> later.  +1.
>>>>>>>>> My biggest blocker for a 1.x release is
>>>>>>>>> https://github.com/apache/storm/pull/2142  Even though the pull
>>>>>>>> request
>>>>>>>>> says it is minor it showed that we messed up pushing back some
>>>>>>>> changes for
>>>>>>>>> pacemaker to open source (the code does not run at all which for
>>>>> me
>>>>>>>> is a
>>>>>>>>> blocker) and I really want to get that fully fixed/tested before
>>>>>>>> another
>>>>>>>>> release.
>>>>>>>>> As for 2.x I think we are very close to being able to so a 2.x
>>>>>>> alpha
>>>>>>>>> release.  I would like to see metrics V2 merged in simply because
>>>>>>> it
>>>>>>>> is a
>>>>>>>>> big change for user facing APIs.  But after that I would love to
>>>>>>> see
>>>>>>>> us
>>>>>>>>> starting to push forward on getting that out.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> - Bobby
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>>>>>>>>> ptgoetz@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> I’d like to bump this thread and start a discussion around our
>>>>> next
>>>>>>>>> release. Here are my thoughts.
>>>>>>>>> 
>>>>>>>>> There are a number of important fixes in 1.x-branch so I’d like
>>>>> to
>>>>>>>>> consider releasing 1.1.1 soon. I’d appreciate input on any open
>>>>>>>> issues that
>>>>>>>>> should be resolved for that release.
>>>>>>>>> 
>>>>>>>>> I’d like us to consider releasing the metrics improvements in
>>>>>>>> STORM-2153
>>>>>>>>> [1] as version 1.2.0. That work is in the metrics_v2 feature
>>>>> branch
>>>>>>>> right
>>>>>>>>> now and would need to be reviewed and merged. That work is
>>>>> against
>>>>>>>> the
>>>>>>>>> 1.x-branch right now. I would recommend porting it to master
>>>>>>> *after*
>>>>>>>> the
>>>>>>>>> review/merge since there will likely be changes as a result of
>>>>> the
>>>>>>>> review.
>>>>>>>>> 
>>>>>>>>>> Maybe related to or not, but would we want to create a new
>>>>> branch
>>>>>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> If wee agree to the above, I would say yes. After the 1.1.1
>>>>>>> release,
>>>>>>>> we
>>>>>>>>> could create a 1.1.x-branch that would be the maintenance/release
>>>>>>>> branch
>>>>>>>>> for that version line. 1.x-branch would then become the target
>>>>> for
>>>>>>>> the
>>>>>>>>> 1.2.0 release.
>>>>>>>>> 
>>>>>>>>> There are a few fixes in the 0.10.x branch that probably warrant
>>>>> a
>>>>>>>>> release. After that we may want to back away from that version
>>>>> line
>>>>>>>> a bit
>>>>>>>>> so we can focus more on newer versions.
>>>>>>>>> 
>>>>>>>>> In the past, we’ve shied away form doing “beta” releases, but I’m
>>>>>>>>> wondering if we might want to revisit that for the 2.0 release —
>>>>>>> the
>>>>>>>> idea
>>>>>>>>> being that it would give early adopter users a chance to kick the
>>>>>>>> tires on
>>>>>>>>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
>>>>> help
>>>>>>>> make the
>>>>>>>>> final release more solid. I’m on the fence here and could go
>>>>> either
>>>>>>>> way.
>>>>>>>>> 
>>>>>>>>> I’d appreciate any input others may have.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> -Taylor
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> [1] https://issues.apache.org/jira/browse/STORM-2153
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Maybe related to or not, but would we want to create a new
>>>>> branch
>>>>>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>>>>>>>> 
>>>>>>>>>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
>>>>> so
>>>>>>>> hence
>>>>>>>>> the
>>>>>>>>>> question.
>>>>>>>>>> 
>>>>>>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>>>>>> 
>>>>>>>>>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
>>>>>>>> hlouro@hortonworks.com>님이
>>>>>>>>> 작성:
>>>>>>>>>> 
>>>>>>>>>>> +1 for finishing the porting to Java ahead of anything else -
>>>>> it
>>>>>>>> will
>>>>>>>>> be a
>>>>>>>>>>> significant milestone. I have a JIRA assigned concerning to
>>>>> the
>>>>>>>>> porting. I
>>>>>>>>>>> will work on it for the 2.0 release.
>>>>>>>>>>> 
>>>>>>>>>>> it’s a priority to guarantee no performance regressions. As
>>>>> part
>>>>>>>> of this
>>>>>>>>>>> endeavor, explore an automated (or easy) way to run and assert
>>>>>>>> major
>>>>>>>>>>> performance benchmarks. Ideally any contributor should be able
>>>>>>> to
>>>>>>>> fairly
>>>>>>>>>>> easily test the impact of changes under certain performance
>>>>> test
>>>>>>>>> scenarios.
>>>>>>>>>>> 
>>>>>>>>>>> Beam Runner work should take into account the impact of
>>>>>>>> incorporating
>>>>>>>>> new
>>>>>>>>>>> JStorm features and Storm Worker Redesign<
>>>>>>>>>>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
>>>>>>>> efficient
>>>>>>>>> to
>>>>>>>>>>> start doing it, to  find out that it will have to chance in
>>>>> face
>>>>>>>> of
>>>>>>>>> Storm
>>>>>>>>>>> and worker redesign. That is, it should be done after it’s
>>>>>>>> building
>>>>>>>>> blocks
>>>>>>>>>>> are stable.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hugo
>>>>>>>>>>> 
>>>>>>>>>>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
>>>>> arunm@apache.org
>>>>>>>> <mailto:
>>>>>>>>>>> arunm@apache.org>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> +1 to release with the porting completed. I think its mainly
>>>>> the
>>>>>>>> UI
>>>>>>>>> server
>>>>>>>>>>> and log viewer that’s pending.
>>>>>>>>>>> 
>>>>>>>>>>> We can start doing the regression and performance tests for
>>>>>>>> whatever is
>>>>>>>>>>> already ported.
>>>>>>>>>>> 
>>>>>>>>>>> If anyone is running the master branch in their pre-prod /
>>>>> prod
>>>>>>>>>>> environments, it will be good to know and give us more
>>>>>>> confidence.
>>>>>>>>>>> 
>>>>>>>>>>> The other features can be added in follow up releases.
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Arun
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 3/24/17, 11:47 AM, "Satish Duggana" <
>>>>>>> satish.duggana@gmail.com
>>>>>>>>> <mailto:
>>>>>>>>>>> satish.duggana@gmail.com>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> +1 to have 2.0 with porting and performance(it should be at
>>>>>>> least
>>>>>>>> as
>>>>>>>>> good
>>>>>>>>>>> as 1.x release) issues addressed
>>>>>>>>>>> 
>>>>>>>>>>> We can target other tasks(mentioned by Taylor and Jungtaek)
>>>>> for
>>>>>>>>> 2.x-branch.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Exactly-once support:
>>>>>>>>>>> While thinking through the exactlyonce support design, it is
>>>>>>>> realized
>>>>>>>>>>> better to avoid acking tuples and implement exactly once by
>>>>>>>> snapshotting
>>>>>>>>>>> barriers. It seems JStorm folks followed similar design, they
>>>>>>>> claim it
>>>>>>>>>>> gives better performance. This feature is essential for beam
>>>>>>>> runner and
>>>>>>>>> we
>>>>>>>>>>> can decide on respective approaches though.
>>>>>>>>>>> 
>>>>>>>>>>> Beam Runner
>>>>>>>>>>> Lets hold on this for now and keep it in Storm till 2.x. We
>>>>>>>> should avoid
>>>>>>>>>>> having a minimal beam runner in haste. It is better to address
>>>>>>>>> STORM-2284,
>>>>>>>>>>> exactly-once and other windowing enhancements to enable beam
>>>>>>>> runner.
>>>>>>>>>>> 
>>>>>>>>>>> JStorm
>>>>>>>>>>> Agree with Jungtaek on looking at the latest JStorm and
>>>>>>>> align/scope with
>>>>>>>>>>> the features for 2.x.
>>>>>>>>>>> 
>>>>>>>>>>> STORM-2284
>>>>>>>>>>> We may want to look at JStorm worker before working on
>>>>>>> respective
>>>>>>>>>>> components in this epic to pull appropriate enhancements.
>>>>>>>>>>> 
>>>>>>>>>>> YARN/MESOS
>>>>>>>>>>> Supporting Storm on YARN/Mesos for 2.x.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Satish.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
>>>>>>> kabhwan@gmail.com
>>>>>>>>> <mailto:
>>>>>>>>>>> kabhwan@gmail.com>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> First of all, +1 to complete only port work and do sanity
>>>>> check
>>>>>>>>> (including
>>>>>>>>>>> performance regression), and release.
>>>>>>>>>>> 
>>>>>>>>>>> If we can get STORM-2284 within deterministic time frame (say
>>>>>>> 2~3
>>>>>>>>> months)
>>>>>>>>>>> that should be great, but if not I'd in favor of postponing
>>>>> that
>>>>>>>> to
>>>>>>>>> later
>>>>>>>>>>> 2.x release.
>>>>>>>>>>> 
>>>>>>>>>>> JStorm released their new versions after code donation. So
>>>>>>>> there're more
>>>>>>>>>>> things we could get ideas from, or even adopt from.
>>>>>>>>>>> https://github.com/alibaba/jstorm/blob/master/history.md
>>>>>>>>>>> As you noticed from release note link, we also need to update
>>>>>>>> phase 2
>>>>>>>>> since
>>>>>>>>>>> they already changed what we're planning to do in phase 2. For
>>>>>>>> example,
>>>>>>>>>>> they changed backpressure to end-to-end, and changed to use
>>>>>>>> snapshot
>>>>>>>>> rather
>>>>>>>>>>> than acker.
>>>>>>>>>>> May be sure, JStorm pulled many features from today's Storm,
>>>>>>> like
>>>>>>>> Flux,
>>>>>>>>>>> Windowing, more shuffle groupings, log search, log level
>>>>> change,
>>>>>>>> and so
>>>>>>>>> on.
>>>>>>>>>>> 
>>>>>>>>>>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
>>>>>>> is
>>>>>>>> due to
>>>>>>>>>>> the
>>>>>>>>>>> limitation of Spout lifecycle (all the things are done in
>>>>> single
>>>>>>>>> thread),
>>>>>>>>>>> and STORM-1358 <
>>>>>>> https://issues.apache.org/jira/browse/STORM-1358
>>>>>>>>>> (JStorm's
>>>>>>>>>>> multi-thread Spout) can remedy this (despite that Spout
>>>>>>>> implementation
>>>>>>>>> may
>>>>>>>>>>> need to guarantee thread-safety later). It's not a just
>>>>>>>> improvement but
>>>>>>>>>>> close to design concern so would like to address sooner than
>>>>>>> other
>>>>>>>>> things
>>>>>>>>>>> in phase 2.
>>>>>>>>>>> 
>>>>>>>>>>> For Storm SQL side, I've lost progress but major work would be
>>>>>>>> adopting
>>>>>>>>>>> group by with windowing. It was not available from Calcite but
>>>>>>>> will be
>>>>>>>>>>> available at next release (1.12.0).
>>>>>>>>>>> I've filed this to STORM-2405
>>>>>>>>>>> <https://issues.apache.org/jira/browse/STORM-2405>, but
>>>>>>>> windowing &
>>>>>>>>> micro
>>>>>>>>>>> batch is not intuitive, so I would like to change the
>>>>> underlying
>>>>>>>> API to
>>>>>>>>>>> stream API in SQL. Also filed this to STORM-2406
>>>>>>>>>>> <https://issues.apache.org/jira/browse/STORM-2406>.
>>>>>>>>>>> 
>>>>>>>>>>> Just 2 cents btw, hopefully I would like to see metrics V2
>>>>>>> sooner
>>>>>>>> since
>>>>>>>>> we
>>>>>>>>>>> lost metrics even when doing normal operation like restarting
>>>>>>>> worker,
>>>>>>>>>>> rebalancing, and so on. Eventually we need to fight with
>>>>> dynamic
>>>>>>>>> scaling,
>>>>>>>>>>> and then metrics will be broken often.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>>>>> 
>>>>>>>>>>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
>>>>> storm@harsha.io
>>>>>>>> 님이
>>>>>>>> 작성:
>>>>>>>>>>> 
>>>>>>>>>>> Storm 2.0 migration to java in itself is a big win and would
>>>>>>>> attract
>>>>>>>>>>> wider
>>>>>>>>>>> community and adoption. So my vote would be to resolve the
>>>>> first
>>>>>>>> 3 items
>>>>>>>>>>> to
>>>>>>>>>>> get a release out.
>>>>>>>>>>> All the other featured mentioned are great to have but
>>>>> shouldn't
>>>>>>>> be
>>>>>>>>>>> blockers for 2.0 release.
>>>>>>>>>>> 
>>>>>>>>>>> -Harsha
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
>>>>>>>> ptgoetz@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> With the 1.1.0 release nearing completion, I’d like to turn
>>>>> our
>>>>>>>>>>> attention
>>>>>>>>>>> to 2.0 and develop a plan for what features, etc. to include.
>>>>>>>>>>> 
>>>>>>>>>>> The following 3 are what I feel are the minimum for a 2.0
>>>>>>> release.
>>>>>>>>>>> These
>>>>>>>>>>> could likely be resolved relatively quickly:
>>>>>>>>>>> 
>>>>>>>>>>> * Performance — I’ve not benchmarked the master branch vs.
>>>>> 1.0.x
>>>>>>>> or
>>>>>>>>>>> 1.1.x
>>>>>>>>>>> in a while, but I feel it will be important to make sure there
>>>>>>>> are no
>>>>>>>>>>> performance regressions, and would hope that we actually have
>>>>> a
>>>>>>>>>>> performance
>>>>>>>>>>> improvement over previous versions. To that end (e.g. if there
>>>>>>> is
>>>>>>>> in
>>>>>>>>>>> fact a
>>>>>>>>>>> performance regression), the proposals that Roshan Naik put
>>>>>>>> together
>>>>>>>>>>> for
>>>>>>>>>>> revising the threading and execution model (STORM-2307) and
>>>>>>>> replacing
>>>>>>>>>>> Disruptor with JCTools (STORM-2306) warrant review and
>>>>>>>> consideration.
>>>>>>>>>>> See
>>>>>>>>>>> also STORM-2284 which is the parent JIRA.
>>>>>>>>>>> 
>>>>>>>>>>> * Finish porting Storm UI to java (STORM-1311)
>>>>>>>>>>> 
>>>>>>>>>>> * Finish porting log viewer to java (STORM-1280)
>>>>>>>>>>> 
>>>>>>>>>>> The following are items that are nice to have in 2.0, but I
>>>>>>> don’t
>>>>>>>> feel
>>>>>>>>>>> are
>>>>>>>>>>> absolutely necessary for an initial 2.0 release:
>>>>>>>>>>> 
>>>>>>>>>>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
>>>>> because
>>>>>>>> it’s
>>>>>>>>>>> relevant) — Initially there seemed to be a lot of interest in
>>>>>>>> this, but
>>>>>>>>>>> that seems to have trailed off. I spoke with some Beam
>>>>>>> developers
>>>>>>>> and
>>>>>>>>>>> there
>>>>>>>>>>> seems to be interest from that community as well. Do we want
>>>>> to
>>>>>>>> move
>>>>>>>>>>> that
>>>>>>>>>>> effort to the Beam community, or keep it here? Moving it to
>>>>> the
>>>>>>>> Beam
>>>>>>>>>>> community might lead to better collaboration between projects.
>>>>>>>>>>> 
>>>>>>>>>>> * Bounded Spouts (needed for Beam Runner implementation) —
>>>>>>>> Currently
>>>>>>>>>>> spouts are unbounded, there no end to the stream. Beam has the
>>>>>>>> concept
>>>>>>>>>>> of
>>>>>>>>>>> bounded sources (roughly analogous to batch processing). To
>>>>>>>> support
>>>>>>>>>>> that,
>>>>>>>>>>> we would need to implement a similar concept in Storm. One
>>>>>>>> benefit of
>>>>>>>>>>> such
>>>>>>>>>>> a feature would be the ability to handle both bounded and
>>>>>>>> unbounded
>>>>>>>>>>> workflows in Storm.
>>>>>>>>>>> 
>>>>>>>>>>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
>>>>>>>> behind
>>>>>>>>>>> this
>>>>>>>>>>> effort. What improvements do you envision for 2.0?
>>>>>>>>>>> 
>>>>>>>>>>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
>>>>>>>> targeting this
>>>>>>>>>>> for 1.2.0, but it’s designed to be easily portable to
>>>>>>> master/2.0.
>>>>>>>>>>> 
>>>>>>>>>>> * JStorm Migration — Original outline can be found here [1].
>>>>>>> Note
>>>>>>>> a lot
>>>>>>>>>>> of
>>>>>>>>>>> the associated JIRAs below are assigned, but there hasn’t been
>>>>>>> any
>>>>>>>>>>> recent
>>>>>>>>>>> activity or pull requests, we should probably consider them
>>>>>>>> unassigned
>>>>>>>>>>> and
>>>>>>>>>>> up for grabs.:
>>>>>>>>>>> 
>>>>>>>>>>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
>>>>>>>> been the
>>>>>>>>>>> bane of a lot of Storm users almost since day one. We have
>>>>>>> largely
>>>>>>>>>>> addressed it by shading/relocating dependencies. It would be
>>>>>>>> great to
>>>>>>>>>>> see
>>>>>>>>>>> this addressed once and for all.
>>>>>>>>>>> 
>>>>>>>>>>> * JStorm back pressure implementation (STORM-1324) — The
>>>>> current
>>>>>>>> back
>>>>>>>>>>> pressure implementation leaves a bit to be desired, and the
>>>>>>> JStorm
>>>>>>>>>>> approach
>>>>>>>>>>> looks promising, though it also depends on the JStorm concept
>>>>> of
>>>>>>>>>>> “topology
>>>>>>>>>>> master” (STORM-1323), which may have some implications
>>>>> regarding
>>>>>>>>>>> security.
>>>>>>>>>>> 
>>>>>>>>>>> * Dynamic Topology Updates (STORM-1335) — This would provide a
>>>>>>>> command
>>>>>>>>>>> to
>>>>>>>>>>> update topology jars and configuration without stopping the
>>>>>>>> topology,
>>>>>>>>>>> and
>>>>>>>>>>> is well suited to leverage the blobstore. The restart command
>>>>>>>> (that can
>>>>>>>>>>> also update the topology configuration) also looks compelling
>>>>>>>>>>> (STORM-1334).
>>>>>>>>>>> 
>>>>>>>>>>> * Additional Scheduler Implementations (STORM-1320)
>>>>>>>>>>> 
>>>>>>>>>>> * Additional Grouping Implementations (STORM-1328)
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> As always I’m open to any opinions and suggestions.
>>>>>>>>>>> 
>>>>>>>>>>> -Taylor
>>>>>>>>>>> 
>>>>>>>>>>> [1]
>>>>>>>>>>> 
>>>>>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.
>>>>>>>>>>> action?pageId=61328109
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>>> 
>>> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Xin Wang <da...@gmail.com>.
Do we have a clear date to release Storm 2.0 beta version? I saw some users
expecting to use Java8.

BTW. Could somebody help to update the Storm site? The 2.0.0-SNAPSHOT
documents should be updated. Thanks.

 - Xin



2017-06-29 23:27 GMT+08:00 Jungtaek Lim <ka...@gmail.com>:

> FYI I just gave it a try with separating 1.x branch and 1.1.x branch (sure
> experimental in forked repo)
>
> https://github.com/heartsavior/storm/tree/1.1.x-branch-experimental
>
> I've updated CHANGELOG only once in that branch so you can see full of the
> changelog which contains the issues ported back.
>
> https://github.com/HeartSaVioR/storm/commit/81e9d65793abc5defc0ab83c09b26a
> 7dcba7e0eb
>
> Most of the issues are classified to the bug fix, but there're also some
> issues filtered out.
> (For example, wildcard classpath, refactor storm-autocreds, binary
> storm-redis state, and so on)
>
> Please let me know what's your opinion on filtering out non-bugfix issues
> from 1.1.1.
> If there's no objection I'll do the change:
> - rename the branch to 1.1.x-branch and push
> - change the version of 1.x-branch to 1.2.0-SNAPSHOT
> - reflect the version change to JIRA issues
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2017년 6월 29일 (목) 오전 6:41, Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>님이
> 작성:
>
> > Hi Hugo,
> >
> > Thanks for your concerns about our troubles with the new
> > storm-kafka-client.
> >
> > Our "bench" is based on our live production data of our cloud supervision
> > system, collecting  at least 1million metrics/min in our Kafka Brokers
> > cluster (currently based on Kafka 0.10.1.0, with "compatibility flag
> > active").
> >
> > More details are available in the thread in the same dev list entitled
> "Lag
> > issues using Storm 1.1.1 latest build with StormKafkaClient 1.1.1 vs old
> > StormKafka spouts".
> >
> > The latest post on this thread from Stig Døssing is giving me back some
> > hope to see some progress in understanding our issues.
> >
> > My point about writing our own Spout come from our past experience: we've
> > been using Kafka for a very long time in our supervision application. Way
> > before we decided to use Storm, we had our own Java daemons consuming the
> > same topics as today, doing some evaluation and writing them into an
> > in-memory store for later consumption by our web services - see this as
> > a"poor man's streaming system" ;-)  In this legacy code, the part in
> charge
> > of consuming data from Kafka wasn't the most complex which we had: a
> small
> > pool of threads using the old Kafka consumer API... so maybe I'm wrong,
> but
> > for *this* purpose I do not feel like writing a Spout consuming a few
> > topics to be a big effort. But of course, if we do that, then we'll miss
> > the fancy integration in StormUI, flux, and the ability to subscribe to
> > multiple topics based on a wildcard expression.
> >
> > So we're going to carefully dig into Stig's answer, and probably provide
> > more details on our bench before jumping into our home-brewed Kafka
> > spout... then I'll have to make some decision based on how much progress
> we
> > have vs time remaining before our next delivery gate.
> >
> > Hope it clarifies my position with regard to storm-kafka-client
> >
> > Best regards,
> > Alexandre
> >
> >
> >
> >
> >
> >
> > 2017-06-28 22:49 GMT+02:00 Hugo Da Cruz Louro <hl...@hortonworks.com>:
> >
> > > Hi Alexandre,
> > >
> > > In my benchmarks the storm-kafka-client spout improves throughput by
> 70%
> > > and latency by 40% vs the storm-kafka implementation. I am surprised by
> > > your findings substantiating the opposite. Can you share your benchmark
> > > where you compare the performances of both implementations?
> > >
> > > As for you writing your own version of the spout. Why not contribute to
> > > this one instead? Do you think the implementation is that poor? If so,
> > why
> > > do you think it is that poor? Do you expect your first version to be
> much
> > > better than a version that is already in production in several
> customers,
> > > and seems to be working fairly well?
> > >
> > > All the bugs found so far have been addressed. Since it’s a new feature
> > > there may be a few bugs - it is expected. However, I don’t think that
> it
> > is
> > > as bad as you make it sound as there are several people using it in
> > > production for extended periods of time.
> > >
> > > Cheers
> > >
> > > > On Jun 28, 2017, at 12:04 PM, Alexandre Vermeerbergen <
> > > avermeerbergen@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > If that matters, our current experiences with StormKafkaClient
> > > > isdisappointing (see my recent posts "Lag issues using Storm 1.1.1
> > latest
> > > > build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this
> > > mailing
> > > > list).
> > > >
> > > > Our current experience is that the old StormKafka spout always beats
> > the
> > > > new one in term of performance & stability.
> > > >
> > > > Therefore, I am surprised when I see talks about deprecation of the
> old
> > > > StormKafka spout when the new one which just came "General Available"
> > > with
> > > > Storm 1.1.0, is not stable, and it's not better when we try it from
> > > current
> > > > 1.1.x builds to take into account recently closed JIRAs.
> > > >
> > > > We're even considering writing our own Kafka spout with Kafka 0.10.x
> > API
> > > to
> > > > overcome the incompatibility of the old StormKafka spout with Kafka
> > 0.10
> > > > libraries.
> > > >
> > > > Thus, for people which are comfortable with old Kafka spout, I'like
> to
> > > give
> > > > a -1 (non binding) to the proposal of withdrawal of the old
> StormKafka
> > > > spout until the new one converges.
> > > >
> > > > Best regards,
> > > > Alexandre Vermeerbergen
> > > >
> > > >
> > > > 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:
> > > >
> > > >>
> > > >>> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <
> > > hlouro@hortonworks.com>
> > > >> wrote:
> > > >>>
> > > >>> I still need to go over the entire discussion thread in more
> detail,
> > > but
> > > >> one thing I would like to bring up right way is the proposal to
> > > DEPRECATE,
> > > >> and eventually remove, the KafkaSpout with the old Kafka Consumer
> > APIs.
> > > The
> > > >> storm-kafka-client KafkaSpout is getting stabilized, and I think we
> > are
> > > all
> > > >> in agreement that the storm-kafka KafkaSpout has presented
> continuous
> > > >> maintainability problems with some fixes that got in not being
> > backwards
> > > >> compatible.
> > > >>
> > > >> I’m fine with deprecating the old KafkaSpout, but I feel the
> decision
> > to
> > > >> actually remove it needs to take into account the user community.
> The
> > > main
> > > >> sticking point here is compatibility with earlier versions of Kafka.
> > > Like
> > > >> with JDK versions, there are many valid reasons whey users may not
> be
> > > in a
> > > >> position to upgrade to a newer version of Kafka. Outright removal
> > could
> > > >> leave some users in the lurch.
> > > >>
> > > >> Ideally, we could just poll the user community to get an idea of how
> > > much
> > > >> of the user base depends on the old KafkaSpout and use the results
> to
> > > guide
> > > >> our decision. Unfortunately, at least in my past experience, polling
> > the
> > > >> user@ list doesn’t elicit much of a response and the results don’t
> > > >> provide an accurate view of the user community.
> > > >>
> > > >>
> > > >>>
> > > >>> I am pretty confident how things are looking at this point for the
> > > >> KafkaSpout. The Trident Kafka Spout is likely in between alpha and
> > beta,
> > > >> and that should be taken into account. I just recently submitted a
> PR<
> > > >> https://github.com/apache/storm/pull/2174> with some improvements
> to
> > > the
> > > >> Trident Kafka Spout (including the refactoring done to support
> manual
> > > >> partition assignment), and there are some customers using it in
> > > >> pre-production. However, it definitely would benefit from some more
> > > testing.
> > > >>>
> > > >>> Thanks,
> > > >>> Hugo
> > > >>
> > > >> -Taylor
> > > >>
> > > >>
> > > >>>
> > > >>> On Jun 28, 2017, at 7:48 AM, Bobby Evans
> <evans@yahoo-inc.com.INVALID
> > <
> > > >> mailto:evans@yahoo-inc.com.INVALID>> wrote:
> > > >>>
> > > >>> +1.
> > > >>> If the 1.1 and 1.2 lines start to become difficult to maintain we
> can
> > > >> look at putting them in maintenance mode too once we have a 2.x
> > release.
> > > >>> I am a little nervous about merging a new feature into 1.x branch
> > > >> without first going to master, but I hope that it will not be too
> much
> > > work
> > > >> to port it to master, and I trust the devs on that branch to do the
> > > right
> > > >> thing.
> > > >>> On a related note we have not done much with feature branches
> before
> > so
> > > >> I am not sure what we want to do about merging in the new metrics
> API
> > > >> branch to 1.x.  I know for me I have not had time to keep up with
> the
> > > >> development work going on there.  I would at least like to have a
> pull
> > > >> request put up for review before we merge it in.  This would fit
> with
> > > our
> > > >> current bylaws that do not mention feature branches.  If all of the
> > > changes
> > > >> have already followed the review process then technically I think it
> > is
> > > OK
> > > >> to just merge it in, but I still would like to take some time to
> look
> > at
> > > >> the changes, and especially the new APIs.
> > > >>>
> > > >>> - Bobby
> > > >>>
> > > >>>
> > > >>> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <
> > > >> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> > > >>>
> > > >>> That's great news that metrics work is ready!
> > > >>>
> > > >>> I'm +1 to Taylor's proposal, but in order to respect semantic
> > > >> versioning, I
> > > >>> propose some modifications from Taylor's proposal:
> > > >>>
> > > >>> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port
> > back
> > > >> only
> > > >>> bug fixes to the 1.1.x-branch
> > > >>> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
> > > >>>
> > > >>> If we also agree above, I would like to volunteer the back-port
> work.
> > > >>>
> > > >>> Thanks,
> > > >>> Jungtaek Lim (HeartSaVioR)
> > > >>>
> > > >>> 2017년 6월 28일 (수) 오전 10:09, Harsha <storm@harsha.io<mailto:storm@
> > > >> harsha.io>>님이 작성:
> > > >>>
> > > >>> +1 for above stated approach on releasing 1.2.0 with metrics
> > > >>> -Harsha
> > > >>>
> > > >>> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> > > >>> The work on metrics is ready for a pull request to 1.x-branch from
> > the
> > > >>> feature branch. I’ve held off because we haven’t reached consensus
> > on a
> > > >>> path forward with the 1.x release lines .
> > > >>>
> > > >>> I’d like to propose the following for the 1.x line:
> > > >>>
> > > >>> 1. Create a branch for 1.2 so we have a branch to review the
> metrics
> > > >>> stuff.
> > > >>> 2. Release 1.1.1
> > > >>> 3. Review/merge metrics work. Port metrics to master.
> > > >>> 4. Release 1.2.0
> > > >>> 5. Put the entire 1.x line into maintenance mode. Drop support for
> > > 1.0.x.
> > > >>> (we would only support 1.2.x and 1.1.x which are very closely
> > aligned).
> > > >>>
> > > >>> Dropping support for 1.0.x line would eliminate the need to
> maintain
> > > one
> > > >>> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would
> be
> > > >>> very closely aligned. I just up merged metrics_v2 against
> 1.x-branch
> > > >>> after a while, and there were no conflicts.
> > > >>>
> > > >>> That would give us a little more bandwidth to focus on 2.0 and
> needed
> > > bug
> > > >>> fixes to the 1.x line like some of the issues raised with
> > > >>> storm-kafka-client. We could even start releasing alpha/beta
> versions
> > > of
> > > >>> 2.0 in parallel to the steps above.
> > > >>>
> > > >>> Any thoughts on that approach?
> > > >>>
> > > >>> -Taylor
> > > >>>
> > > >>>
> > > >>> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabhwan@gmail.com
> <mailto:
> > > kabh
> > > >> wan@gmail.com>> wrote:
> > > >>>
> > > >>> Yes I prefer option 1, but it might depend on the progress of
> metrics
> > > >>> V2.
> > > >>> If it can be done within predictable near future I'm OK to pick
> > option
> > > >>> 2,
> > > >>> but if not, we may be better to focus releasing 2.0.0 and make it
> > > >>> really
> > > >>> happen.
> > > >>>
> > > >>> Whichever we go, I feel it's time to track remaining work on Storm
> > > >>> 2.0.0. I
> > > >>> found some bugs on master branch so filed issues, and we've
> remaining
> > > >>> port
> > > >>> work (UI and logviewer). We've some other improvements target for
> > > >>> 2.0.0:
> > > >>> worker redesign, beam integration, and so on, and we don't track
> its
> > > >>> progress at all. I don't think we should wait for features which
> > > >>> progress
> > > >>> is not transparent (in other words we don't know when it will be
> > > >>> finished).
> > > >>>
> > > >>> - Jungtaek Lim (HeartSaVioR)
> > > >>>
> > > >>> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgoetz@gmail.com
> <mailto:
> > > ptgo
> > > >> etz@gmail.com>>님이 작성:
> > > >>>
> > > >>> Bobby/Jungtaek,
> > > >>>
> > > >>> Are you saying you want to forego the 1.2 “metrics_v2” release and
> > > >>> include
> > > >>> it only in 2.0? (I ask because that work is already based on
> > > >>> 1.x-branch,
> > > >>> and forward-porting it to master is relatively simple.) I’d kind of
> > > >>> like
> > > >>> that work go out soon.
> > > >>>
> > > >>> If we go with option 1, I would want to see a 2.0 release (even if
> > > >>> it’s a
> > > >>> “beta” or “preview) before putting the 1.x line into maintenance
> > mode.
> > > >>>
> > > >>> -Taylor
> > > >>>
> > > >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans
> <evans@yahoo-inc.com.INVALID
> > <
> > > >> mailto:evans@yahoo-inc.com.INVALID>
> > > >>>
> > > >>> wrote:
> > > >>>
> > > >>> I see 2 ways to address this.
> > > >>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
> > > >>> don't
> > > >>> backport anything except bug fixes.2) We backport a lot of the
> > > >>> backwards
> > > >>> compatible changes from 2.x to 1.x.
> > > >>> My personal preference is 1.  It makes it clear the direction we
> > > >>> want to
> > > >>> go in.  The biggest issue is that we probably would want to do a
> 2.x
> > > >>> release sooner rather then later.  Even if we don't get all of the
> > > >>> features
> > > >>> that people want, if we just get a release out we can add in new
> > > >>> features
> > > >>> if they are backwards compatible, or we can create a 3.x line that
> > > >>> would
> > > >>> have the breaking changes in it.
> > > >>>
> > > >>> - Bobby
> > > >>>
> > > >>>
> > > >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> > > >>> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> > > >>>
> > > >>> I'd like to bump this again instead of initiating new discussion
> > > >>> thread.
> > > >>>
> > > >>> I had having hard time to create and apply pull requests for both
> > > >>> master
> > > >>> and 1.x-branch and that's really painful and sometimes blocker for
> > > >>> me to
> > > >>> do
> > > >>> merge step.
> > > >>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
> > > >>> even
> > > >>> IDE can't switch between the branch smoothly. We didn't even
> address
> > > >>> checkstyle issue yet, but after addressing, it could be
> "completely"
> > > >>> diverged. JDK version is another major issue, since the pull
> requests
> > > >>> targeted for master branch are not checked against JDK 7, and some
> of
> > > >>> them
> > > >>> make some issues regarding JDK version while porting back.
> > > >>>
> > > >>> So personally I really would like to see the plan for 1.x version
> > > >>> line
> > > >>> changed - skipping any minor releases including 1.2.0 - and have
> epic
> > > >>> issue
> > > >>> for 2.0.0 and just go ahead. That was our proposed plan indeed.
> (even
> > > >>> proposed plan was having 2.0.0 directly after 1.0.0)
> > > >>>
> > > >>> Would like to hear everyone's opinions. If we have consensus to not
> > > >>> having
> > > >>> any minor releases for 1.x version line, I will not port back
> > > >>> non-bugfix
> > > >>> pull requests to 1.x-branch, and guide contributors to create pull
> > > >>> requests
> > > >>> against master branch, not 1.x version line.
> > > >>>
> > > >>> Thanks,
> > > >>> Jungtaek Lim (HeartSaVioR)
> > > >>>
> > > >>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> > > >>> avermeerbergen@gmail.com<ma...@gmail.com>>님이
> > > >>> 작성:
> > > >>>
> > > >>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
> > > >>> system,
> > > >>> we're very interested anything that can provide better throughput.
> > > >>>
> > > >>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <roshan@hortonworks.com
> > <mailto:
> > > >> roshan@hortonworks.com>>:
> > > >>>
> > > >>> For 2.0 beta … it would be good to incorporate some of the Worker
> > > >>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> > > >>> be
> > > >>> delivered sooner and my in-progress implementation suggests that it
> > > >>> will
> > > >>> yield substantial latency improvements. The 2.0 beta phase would
> > > >>> really
> > > >>> help kick the tires on the revised messaging system and the
> > > >>> performance
> > > >>> improvements will also be a good incentive for trying out the 2.0
> > > >>> line.
> > > >>>
> > > >>> I notice multiple other bottlenecks that are holding back
> > > >>> throughput a
> > > >>> lot, which can be addressed in a subsequent 2.x minor release.
> > > >>> -roshan
> > > >>>
> > > >>>
> > > >>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabhwan@gmail.com<mailto:kabh
> > > >> wan@gmail.com>> wrote:
> > > >>>
> > > >>>   I also would love to see metrics V2 code sooner or later too.
> > > >>> If we
> > > >>> can get
> > > >>>   it before releasing 2.0.0 that will be great, and then maybe we
> > > >>> could
> > > >>> just
> > > >>>   move toward to 2.0.0, not adding any improvements to 1.x version
> > > >>> line.
> > > >>>   (And that's what I would want to.)
> > > >>>
> > > >>>   If we would really want to have 1.2.0, I suggest that we make
> > > >>> the
> > > >>> 1.1.1
> > > >>>   version correct right now rather than after releasing 1.1.1. We
> > > >>> also
> > > >>> merged
> > > >>>   non-bugfix things to 1.x-branch but that's not what users
> > > >>> expect. I
> > > >>> agree
> > > >>>   that work may be painful, but anyway need to do it.
> > > >>>
> > > >>>   - Jungtaek Lim (HeartSaVioR)
> > > >>>
> > > >>>   2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> > > >>> <ev...@yahoo-inc.com.invalid>
> > > >>> 님이
> > > >>> 작성:
> > > >>>
> > > >>> I would love to see the metrics V2 code come out sooner rather
> > > >>> than
> > > >>> later.  +1.
> > > >>> My biggest blocker for a 1.x release is
> > > >>> https://github.com/apache/storm/pull/2142  Even though the pull
> > > >>> request
> > > >>> says it is minor it showed that we messed up pushing back some
> > > >>> changes for
> > > >>> pacemaker to open source (the code does not run at all which for
> > > >>> me
> > > >>> is a
> > > >>> blocker) and I really want to get that fully fixed/tested before
> > > >>> another
> > > >>> release.
> > > >>> As for 2.x I think we are very close to being able to so a 2.x
> > > >>> alpha
> > > >>> release.  I would like to see metrics V2 merged in simply because
> > > >>> it
> > > >>> is a
> > > >>> big change for user facing APIs.  But after that I would love to
> > > >>> see
> > > >>> us
> > > >>> starting to push forward on getting that out.
> > > >>>
> > > >>>
> > > >>> - Bobby
> > > >>>
> > > >>>
> > > >>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> > > >>> ptgoetz@gmail.com<ma...@gmail.com>> wrote:
> > > >>>
> > > >>> I’d like to bump this thread and start a discussion around our
> > > >>> next
> > > >>> release. Here are my thoughts.
> > > >>>
> > > >>> There are a number of important fixes in 1.x-branch so I’d like
> > > >>> to
> > > >>> consider releasing 1.1.1 soon. I’d appreciate input on any open
> > > >>> issues that
> > > >>> should be resolved for that release.
> > > >>>
> > > >>> I’d like us to consider releasing the metrics improvements in
> > > >>> STORM-2153
> > > >>> [1] as version 1.2.0. That work is in the metrics_v2 feature
> > > >>> branch
> > > >>> right
> > > >>> now and would need to be reviewed and merged. That work is
> > > >>> against
> > > >>> the
> > > >>> 1.x-branch right now. I would recommend porting it to master
> > > >>> *after*
> > > >>> the
> > > >>> review/merge since there will likely be changes as a result of
> > > >>> the
> > > >>> review.
> > > >>>
> > > >>> Maybe related to or not, but would we want to create a new
> > > >>> branch
> > > >>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > > >>>
> > > >>>
> > > >>>
> > > >>> If wee agree to the above, I would say yes. After the 1.1.1
> > > >>> release,
> > > >>> we
> > > >>> could create a 1.1.x-branch that would be the maintenance/release
> > > >>> branch
> > > >>> for that version line. 1.x-branch would then become the target
> > > >>> for
> > > >>> the
> > > >>> 1.2.0 release.
> > > >>>
> > > >>> There are a few fixes in the 0.10.x branch that probably warrant
> > > >>> a
> > > >>> release. After that we may want to back away from that version
> > > >>> line
> > > >>> a bit
> > > >>> so we can focus more on newer versions.
> > > >>>
> > > >>> In the past, we’ve shied away form doing “beta” releases, but I’m
> > > >>> wondering if we might want to revisit that for the 2.0 release —
> > > >>> the
> > > >>> idea
> > > >>> being that it would give early adopter users a chance to kick the
> > > >>> tires on
> > > >>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
> > > >>> help
> > > >>> make the
> > > >>> final release more solid. I’m on the fence here and could go
> > > >>> either
> > > >>> way.
> > > >>>
> > > >>> I’d appreciate any input others may have.
> > > >>>
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> -Taylor
> > > >>>
> > > >>>
> > > >>> [1] https://issues.apache.org/jira/browse/STORM-2153
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabhwan@gmail.com
> <mailto:
> > > kabh
> > > >> wan@gmail.com>>
> > > >>> wrote:
> > > >>>
> > > >>> Maybe related to or not, but would we want to create a new
> > > >>> branch
> > > >>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > > >>>
> > > >>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> > > >>> so
> > > >>> hence
> > > >>> the
> > > >>> question.
> > > >>>
> > > >>> - Jungtaek Lim (HeartSaVioR)
> > > >>>
> > > >>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > > >>> hlouro@hortonworks.com<ma...@hortonworks.com>>님이
> > > >>> 작성:
> > > >>>
> > > >>> +1 for finishing the porting to Java ahead of anything else -
> > > >>> it
> > > >>> will
> > > >>> be a
> > > >>> significant milestone. I have a JIRA assigned concerning to
> > > >>> the
> > > >>> porting. I
> > > >>> will work on it for the 2.0 release.
> > > >>>
> > > >>> it’s a priority to guarantee no performance regressions. As
> > > >>> part
> > > >>> of this
> > > >>> endeavor, explore an automated (or easy) way to run and assert
> > > >>> major
> > > >>> performance benchmarks. Ideally any contributor should be able
> > > >>> to
> > > >>> fairly
> > > >>> easily test the impact of changes under certain performance
> > > >>> test
> > > >>> scenarios.
> > > >>>
> > > >>> Beam Runner work should take into account the impact of
> > > >>> incorporating
> > > >>> new
> > > >>> JStorm features and Storm Worker Redesign<
> > > >>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > > >>> efficient
> > > >>> to
> > > >>> start doing it, to  find out that it will have to chance in
> > > >>> face
> > > >>> of
> > > >>> Storm
> > > >>> and worker redesign. That is, it should be done after it’s
> > > >>> building
> > > >>> blocks
> > > >>> are stable.
> > > >>>
> > > >>> Thanks,
> > > >>> Hugo
> > > >>>
> > > >>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> > > >>> arunm@apache.org<ma...@apache.org>
> > > >>> <mailto:
> > > >>> arunm@apache.org>> wrote:
> > > >>>
> > > >>> +1 to release with the porting completed. I think its mainly
> > > >>> the
> > > >>> UI
> > > >>> server
> > > >>> and log viewer that’s pending.
> > > >>>
> > > >>> We can start doing the regression and performance tests for
> > > >>> whatever is
> > > >>> already ported.
> > > >>>
> > > >>> If anyone is running the master branch in their pre-prod /
> > > >>> prod
> > > >>> environments, it will be good to know and give us more
> > > >>> confidence.
> > > >>>
> > > >>> The other features can be added in follow up releases.
> > > >>>
> > > >>> Regards,
> > > >>> Arun
> > > >>>
> > > >>>
> > > >>> On 3/24/17, 11:47 AM, "Satish Duggana" <
> > > >>> satish.duggana@gmail.com
> > > >>> <mailto:
> > > >>> satish.duggana@gmail.com>> wrote:
> > > >>>
> > > >>> +1 to have 2.0 with porting and performance(it should be at
> > > >>> least
> > > >>> as
> > > >>> good
> > > >>> as 1.x release) issues addressed
> > > >>>
> > > >>> We can target other tasks(mentioned by Taylor and Jungtaek)
> > > >>> for
> > > >>> 2.x-branch.
> > > >>>
> > > >>>
> > > >>> Exactly-once support:
> > > >>> While thinking through the exactlyonce support design, it is
> > > >>> realized
> > > >>> better to avoid acking tuples and implement exactly once by
> > > >>> snapshotting
> > > >>> barriers. It seems JStorm folks followed similar design, they
> > > >>> claim it
> > > >>> gives better performance. This feature is essential for beam
> > > >>> runner and
> > > >>> we
> > > >>> can decide on respective approaches though.
> > > >>>
> > > >>> Beam Runner
> > > >>> Lets hold on this for now and keep it in Storm till 2.x. We
> > > >>> should avoid
> > > >>> having a minimal beam runner in haste. It is better to address
> > > >>> STORM-2284,
> > > >>> exactly-once and other windowing enhancements to enable beam
> > > >>> runner.
> > > >>>
> > > >>> JStorm
> > > >>> Agree with Jungtaek on looking at the latest JStorm and
> > > >>> align/scope with
> > > >>> the features for 2.x.
> > > >>>
> > > >>> STORM-2284
> > > >>> We may want to look at JStorm worker before working on
> > > >>> respective
> > > >>> components in this epic to pull appropriate enhancements.
> > > >>>
> > > >>> YARN/MESOS
> > > >>> Supporting Storm on YARN/Mesos for 2.x.
> > > >>>
> > > >>> Thanks,
> > > >>> Satish.
> > > >>>
> > > >>>
> > > >>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> > > >>> kabhwan@gmail.com
> > > >>> <mailto:
> > > >>> kabhwan@gmail.com>> wrote:
> > > >>>
> > > >>> First of all, +1 to complete only port work and do sanity
> > > >>> check
> > > >>> (including
> > > >>> performance regression), and release.
> > > >>>
> > > >>> If we can get STORM-2284 within deterministic time frame (say
> > > >>> 2~3
> > > >>> months)
> > > >>> that should be great, but if not I'd in favor of postponing
> > > >>> that
> > > >>> to
> > > >>> later
> > > >>> 2.x release.
> > > >>>
> > > >>> JStorm released their new versions after code donation. So
> > > >>> there're more
> > > >>> things we could get ideas from, or even adopt from.
> > > >>> https://github.com/alibaba/jstorm/blob/master/history.md
> > > >>> As you noticed from release note link, we also need to update
> > > >>> phase 2
> > > >>> since
> > > >>> they already changed what we're planning to do in phase 2. For
> > > >>> example,
> > > >>> they changed backpressure to end-to-end, and changed to use
> > > >>> snapshot
> > > >>> rather
> > > >>> than acker.
> > > >>> May be sure, JStorm pulled many features from today's Storm,
> > > >>> like
> > > >>> Flux,
> > > >>> Windowing, more shuffle groupings, log search, log level
> > > >>> change,
> > > >>> and so
> > > >>> on.
> > > >>>
> > > >>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> > > >>> is
> > > >>> due to
> > > >>> the
> > > >>> limitation of Spout lifecycle (all the things are done in
> > > >>> single
> > > >>> thread),
> > > >>> and STORM-1358 <
> > > >>> https://issues.apache.org/jira/browse/STORM-1358
> > > >>> (JStorm's
> > > >>> multi-thread Spout) can remedy this (despite that Spout
> > > >>> implementation
> > > >>> may
> > > >>> need to guarantee thread-safety later). It's not a just
> > > >>> improvement but
> > > >>> close to design concern so would like to address sooner than
> > > >>> other
> > > >>> things
> > > >>> in phase 2.
> > > >>>
> > > >>> For Storm SQL side, I've lost progress but major work would be
> > > >>> adopting
> > > >>> group by with windowing. It was not available from Calcite but
> > > >>> will be
> > > >>> available at next release (1.12.0).
> > > >>> I've filed this to STORM-2405
> > > >>> <https://issues.apache.org/jira/browse/STORM-2405>, but
> > > >>> windowing &
> > > >>> micro
> > > >>> batch is not intuitive, so I would like to change the
> > > >>> underlying
> > > >>> API to
> > > >>> stream API in SQL. Also filed this to STORM-2406
> > > >>> <https://issues.apache.org/jira/browse/STORM-2406>.
> > > >>>
> > > >>> Just 2 cents btw, hopefully I would like to see metrics V2
> > > >>> sooner
> > > >>> since
> > > >>> we
> > > >>> lost metrics even when doing normal operation like restarting
> > > >>> worker,
> > > >>> rebalancing, and so on. Eventually we need to fight with
> > > >>> dynamic
> > > >>> scaling,
> > > >>> and then metrics will be broken often.
> > > >>>
> > > >>> Thanks,
> > > >>> Jungtaek Lim (HeartSaVioR)
> > > >>>
> > > >>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> > > >>> storm@harsha.io
> > > >>> 님이
> > > >>> 작성:
> > > >>>
> > > >>> Storm 2.0 migration to java in itself is a big win and would
> > > >>> attract
> > > >>> wider
> > > >>> community and adoption. So my vote would be to resolve the
> > > >>> first
> > > >>> 3 items
> > > >>> to
> > > >>> get a release out.
> > > >>> All the other featured mentioned are great to have but
> > > >>> shouldn't
> > > >>> be
> > > >>> blockers for 2.0 release.
> > > >>>
> > > >>> -Harsha
> > > >>>
> > > >>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > > >>> ptgoetz@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> With the 1.1.0 release nearing completion, I’d like to turn
> > > >>> our
> > > >>> attention
> > > >>> to 2.0 and develop a plan for what features, etc. to include.
> > > >>>
> > > >>> The following 3 are what I feel are the minimum for a 2.0
> > > >>> release.
> > > >>> These
> > > >>> could likely be resolved relatively quickly:
> > > >>>
> > > >>> * Performance — I’ve not benchmarked the master branch vs.
> > > >>> 1.0.x
> > > >>> or
> > > >>> 1.1.x
> > > >>> in a while, but I feel it will be important to make sure there
> > > >>> are no
> > > >>> performance regressions, and would hope that we actually have
> > > >>> a
> > > >>> performance
> > > >>> improvement over previous versions. To that end (e.g. if there
> > > >>> is
> > > >>> in
> > > >>> fact a
> > > >>> performance regression), the proposals that Roshan Naik put
> > > >>> together
> > > >>> for
> > > >>> revising the threading and execution model (STORM-2307) and
> > > >>> replacing
> > > >>> Disruptor with JCTools (STORM-2306) warrant review and
> > > >>> consideration.
> > > >>> See
> > > >>> also STORM-2284 which is the parent JIRA.
> > > >>>
> > > >>> * Finish porting Storm UI to java (STORM-1311)
> > > >>>
> > > >>> * Finish porting log viewer to java (STORM-1280)
> > > >>>
> > > >>> The following are items that are nice to have in 2.0, but I
> > > >>> don’t
> > > >>> feel
> > > >>> are
> > > >>> absolutely necessary for an initial 2.0 release:
> > > >>>
> > > >>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> > > >>> because
> > > >>> it’s
> > > >>> relevant) — Initially there seemed to be a lot of interest in
> > > >>> this, but
> > > >>> that seems to have trailed off. I spoke with some Beam
> > > >>> developers
> > > >>> and
> > > >>> there
> > > >>> seems to be interest from that community as well. Do we want
> > > >>> to
> > > >>> move
> > > >>> that
> > > >>> effort to the Beam community, or keep it here? Moving it to
> > > >>> the
> > > >>> Beam
> > > >>> community might lead to better collaboration between projects.
> > > >>>
> > > >>> * Bounded Spouts (needed for Beam Runner implementation) —
> > > >>> Currently
> > > >>> spouts are unbounded, there no end to the stream. Beam has the
> > > >>> concept
> > > >>> of
> > > >>> bounded sources (roughly analogous to batch processing). To
> > > >>> support
> > > >>> that,
> > > >>> we would need to implement a similar concept in Storm. One
> > > >>> benefit of
> > > >>> such
> > > >>> a feature would be the ability to handle both bounded and
> > > >>> unbounded
> > > >>> workflows in Storm.
> > > >>>
> > > >>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > > >>> behind
> > > >>> this
> > > >>> effort. What improvements do you envision for 2.0?
> > > >>>
> > > >>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > > >>> targeting this
> > > >>> for 1.2.0, but it’s designed to be easily portable to
> > > >>> master/2.0.
> > > >>>
> > > >>> * JStorm Migration — Original outline can be found here [1].
> > > >>> Note
> > > >>> a lot
> > > >>> of
> > > >>> the associated JIRAs below are assigned, but there hasn’t been
> > > >>> any
> > > >>> recent
> > > >>> activity or pull requests, we should probably consider them
> > > >>> unassigned
> > > >>> and
> > > >>> up for grabs.:
> > > >>>
> > > >>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > > >>> been the
> > > >>> bane of a lot of Storm users almost since day one. We have
> > > >>> largely
> > > >>> addressed it by shading/relocating dependencies. It would be
> > > >>> great to
> > > >>> see
> > > >>> this addressed once and for all.
> > > >>>
> > > >>> * JStorm back pressure implementation (STORM-1324) — The
> > > >>> current
> > > >>> back
> > > >>> pressure implementation leaves a bit to be desired, and the
> > > >>> JStorm
> > > >>> approach
> > > >>> looks promising, though it also depends on the JStorm concept
> > > >>> of
> > > >>> “topology
> > > >>> master” (STORM-1323), which may have some implications
> > > >>> regarding
> > > >>> security.
> > > >>>
> > > >>> * Dynamic Topology Updates (STORM-1335) — This would provide a
> > > >>> command
> > > >>> to
> > > >>> update topology jars and configuration without stopping the
> > > >>> topology,
> > > >>> and
> > > >>> is well suited to leverage the blobstore. The restart command
> > > >>> (that can
> > > >>> also update the topology configuration) also looks compelling
> > > >>> (STORM-1334).
> > > >>>
> > > >>> * Additional Scheduler Implementations (STORM-1320)
> > > >>>
> > > >>> * Additional Grouping Implementations (STORM-1328)
> > > >>>
> > > >>>
> > > >>> As always I’m open to any opinions and suggestions.
> > > >>>
> > > >>> -Taylor
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>> https://cwiki.apache.org/confluence/pages/viewpage.
> > > >>> action?pageId=61328109
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Jungtaek Lim <ka...@gmail.com>.
FYI I just gave it a try with separating 1.x branch and 1.1.x branch (sure
experimental in forked repo)

https://github.com/heartsavior/storm/tree/1.1.x-branch-experimental

I've updated CHANGELOG only once in that branch so you can see full of the
changelog which contains the issues ported back.

https://github.com/HeartSaVioR/storm/commit/81e9d65793abc5defc0ab83c09b26a7dcba7e0eb

Most of the issues are classified to the bug fix, but there're also some
issues filtered out.
(For example, wildcard classpath, refactor storm-autocreds, binary
storm-redis state, and so on)

Please let me know what's your opinion on filtering out non-bugfix issues
from 1.1.1.
If there's no objection I'll do the change:
- rename the branch to 1.1.x-branch and push
- change the version of 1.x-branch to 1.2.0-SNAPSHOT
- reflect the version change to JIRA issues

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 29일 (목) 오전 6:41, Alexandre Vermeerbergen <av...@gmail.com>님이
작성:

> Hi Hugo,
>
> Thanks for your concerns about our troubles with the new
> storm-kafka-client.
>
> Our "bench" is based on our live production data of our cloud supervision
> system, collecting  at least 1million metrics/min in our Kafka Brokers
> cluster (currently based on Kafka 0.10.1.0, with "compatibility flag
> active").
>
> More details are available in the thread in the same dev list entitled "Lag
> issues using Storm 1.1.1 latest build with StormKafkaClient 1.1.1 vs old
> StormKafka spouts".
>
> The latest post on this thread from Stig Døssing is giving me back some
> hope to see some progress in understanding our issues.
>
> My point about writing our own Spout come from our past experience: we've
> been using Kafka for a very long time in our supervision application. Way
> before we decided to use Storm, we had our own Java daemons consuming the
> same topics as today, doing some evaluation and writing them into an
> in-memory store for later consumption by our web services - see this as
> a"poor man's streaming system" ;-)  In this legacy code, the part in charge
> of consuming data from Kafka wasn't the most complex which we had: a small
> pool of threads using the old Kafka consumer API... so maybe I'm wrong, but
> for *this* purpose I do not feel like writing a Spout consuming a few
> topics to be a big effort. But of course, if we do that, then we'll miss
> the fancy integration in StormUI, flux, and the ability to subscribe to
> multiple topics based on a wildcard expression.
>
> So we're going to carefully dig into Stig's answer, and probably provide
> more details on our bench before jumping into our home-brewed Kafka
> spout... then I'll have to make some decision based on how much progress we
> have vs time remaining before our next delivery gate.
>
> Hope it clarifies my position with regard to storm-kafka-client
>
> Best regards,
> Alexandre
>
>
>
>
>
>
> 2017-06-28 22:49 GMT+02:00 Hugo Da Cruz Louro <hl...@hortonworks.com>:
>
> > Hi Alexandre,
> >
> > In my benchmarks the storm-kafka-client spout improves throughput by 70%
> > and latency by 40% vs the storm-kafka implementation. I am surprised by
> > your findings substantiating the opposite. Can you share your benchmark
> > where you compare the performances of both implementations?
> >
> > As for you writing your own version of the spout. Why not contribute to
> > this one instead? Do you think the implementation is that poor? If so,
> why
> > do you think it is that poor? Do you expect your first version to be much
> > better than a version that is already in production in several customers,
> > and seems to be working fairly well?
> >
> > All the bugs found so far have been addressed. Since it’s a new feature
> > there may be a few bugs - it is expected. However, I don’t think that it
> is
> > as bad as you make it sound as there are several people using it in
> > production for extended periods of time.
> >
> > Cheers
> >
> > > On Jun 28, 2017, at 12:04 PM, Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > If that matters, our current experiences with StormKafkaClient
> > > isdisappointing (see my recent posts "Lag issues using Storm 1.1.1
> latest
> > > build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this
> > mailing
> > > list).
> > >
> > > Our current experience is that the old StormKafka spout always beats
> the
> > > new one in term of performance & stability.
> > >
> > > Therefore, I am surprised when I see talks about deprecation of the old
> > > StormKafka spout when the new one which just came "General Available"
> > with
> > > Storm 1.1.0, is not stable, and it's not better when we try it from
> > current
> > > 1.1.x builds to take into account recently closed JIRAs.
> > >
> > > We're even considering writing our own Kafka spout with Kafka 0.10.x
> API
> > to
> > > overcome the incompatibility of the old StormKafka spout with Kafka
> 0.10
> > > libraries.
> > >
> > > Thus, for people which are comfortable with old Kafka spout, I'like to
> > give
> > > a -1 (non binding) to the proposal of withdrawal of the old StormKafka
> > > spout until the new one converges.
> > >
> > > Best regards,
> > > Alexandre Vermeerbergen
> > >
> > >
> > > 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:
> > >
> > >>
> > >>> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <
> > hlouro@hortonworks.com>
> > >> wrote:
> > >>>
> > >>> I still need to go over the entire discussion thread in more detail,
> > but
> > >> one thing I would like to bring up right way is the proposal to
> > DEPRECATE,
> > >> and eventually remove, the KafkaSpout with the old Kafka Consumer
> APIs.
> > The
> > >> storm-kafka-client KafkaSpout is getting stabilized, and I think we
> are
> > all
> > >> in agreement that the storm-kafka KafkaSpout has presented continuous
> > >> maintainability problems with some fixes that got in not being
> backwards
> > >> compatible.
> > >>
> > >> I’m fine with deprecating the old KafkaSpout, but I feel the decision
> to
> > >> actually remove it needs to take into account the user community. The
> > main
> > >> sticking point here is compatibility with earlier versions of Kafka.
> > Like
> > >> with JDK versions, there are many valid reasons whey users may not be
> > in a
> > >> position to upgrade to a newer version of Kafka. Outright removal
> could
> > >> leave some users in the lurch.
> > >>
> > >> Ideally, we could just poll the user community to get an idea of how
> > much
> > >> of the user base depends on the old KafkaSpout and use the results to
> > guide
> > >> our decision. Unfortunately, at least in my past experience, polling
> the
> > >> user@ list doesn’t elicit much of a response and the results don’t
> > >> provide an accurate view of the user community.
> > >>
> > >>
> > >>>
> > >>> I am pretty confident how things are looking at this point for the
> > >> KafkaSpout. The Trident Kafka Spout is likely in between alpha and
> beta,
> > >> and that should be taken into account. I just recently submitted a PR<
> > >> https://github.com/apache/storm/pull/2174> with some improvements to
> > the
> > >> Trident Kafka Spout (including the refactoring done to support manual
> > >> partition assignment), and there are some customers using it in
> > >> pre-production. However, it definitely would benefit from some more
> > testing.
> > >>>
> > >>> Thanks,
> > >>> Hugo
> > >>
> > >> -Taylor
> > >>
> > >>
> > >>>
> > >>> On Jun 28, 2017, at 7:48 AM, Bobby Evans <evans@yahoo-inc.com.INVALID
> <
> > >> mailto:evans@yahoo-inc.com.INVALID>> wrote:
> > >>>
> > >>> +1.
> > >>> If the 1.1 and 1.2 lines start to become difficult to maintain we can
> > >> look at putting them in maintenance mode too once we have a 2.x
> release.
> > >>> I am a little nervous about merging a new feature into 1.x branch
> > >> without first going to master, but I hope that it will not be too much
> > work
> > >> to port it to master, and I trust the devs on that branch to do the
> > right
> > >> thing.
> > >>> On a related note we have not done much with feature branches before
> so
> > >> I am not sure what we want to do about merging in the new metrics API
> > >> branch to 1.x.  I know for me I have not had time to keep up with the
> > >> development work going on there.  I would at least like to have a pull
> > >> request put up for review before we merge it in.  This would fit with
> > our
> > >> current bylaws that do not mention feature branches.  If all of the
> > changes
> > >> have already followed the review process then technically I think it
> is
> > OK
> > >> to just merge it in, but I still would like to take some time to look
> at
> > >> the changes, and especially the new APIs.
> > >>>
> > >>> - Bobby
> > >>>
> > >>>
> > >>> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <
> > >> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> > >>>
> > >>> That's great news that metrics work is ready!
> > >>>
> > >>> I'm +1 to Taylor's proposal, but in order to respect semantic
> > >> versioning, I
> > >>> propose some modifications from Taylor's proposal:
> > >>>
> > >>> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port
> back
> > >> only
> > >>> bug fixes to the 1.1.x-branch
> > >>> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
> > >>>
> > >>> If we also agree above, I would like to volunteer the back-port work.
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 6월 28일 (수) 오전 10:09, Harsha <storm@harsha.io<mailto:storm@
> > >> harsha.io>>님이 작성:
> > >>>
> > >>> +1 for above stated approach on releasing 1.2.0 with metrics
> > >>> -Harsha
> > >>>
> > >>> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> > >>> The work on metrics is ready for a pull request to 1.x-branch from
> the
> > >>> feature branch. I’ve held off because we haven’t reached consensus
> on a
> > >>> path forward with the 1.x release lines .
> > >>>
> > >>> I’d like to propose the following for the 1.x line:
> > >>>
> > >>> 1. Create a branch for 1.2 so we have a branch to review the metrics
> > >>> stuff.
> > >>> 2. Release 1.1.1
> > >>> 3. Review/merge metrics work. Port metrics to master.
> > >>> 4. Release 1.2.0
> > >>> 5. Put the entire 1.x line into maintenance mode. Drop support for
> > 1.0.x.
> > >>> (we would only support 1.2.x and 1.1.x which are very closely
> aligned).
> > >>>
> > >>> Dropping support for 1.0.x line would eliminate the need to maintain
> > one
> > >>> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> > >>> very closely aligned. I just up merged metrics_v2 against 1.x-branch
> > >>> after a while, and there were no conflicts.
> > >>>
> > >>> That would give us a little more bandwidth to focus on 2.0 and needed
> > bug
> > >>> fixes to the 1.x line like some of the issues raised with
> > >>> storm-kafka-client. We could even start releasing alpha/beta versions
> > of
> > >>> 2.0 in parallel to the steps above.
> > >>>
> > >>> Any thoughts on that approach?
> > >>>
> > >>> -Taylor
> > >>>
> > >>>
> > >>> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:
> > kabh
> > >> wan@gmail.com>> wrote:
> > >>>
> > >>> Yes I prefer option 1, but it might depend on the progress of metrics
> > >>> V2.
> > >>> If it can be done within predictable near future I'm OK to pick
> option
> > >>> 2,
> > >>> but if not, we may be better to focus releasing 2.0.0 and make it
> > >>> really
> > >>> happen.
> > >>>
> > >>> Whichever we go, I feel it's time to track remaining work on Storm
> > >>> 2.0.0. I
> > >>> found some bugs on master branch so filed issues, and we've remaining
> > >>> port
> > >>> work (UI and logviewer). We've some other improvements target for
> > >>> 2.0.0:
> > >>> worker redesign, beam integration, and so on, and we don't track its
> > >>> progress at all. I don't think we should wait for features which
> > >>> progress
> > >>> is not transparent (in other words we don't know when it will be
> > >>> finished).
> > >>>
> > >>> - Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
> > ptgo
> > >> etz@gmail.com>>님이 작성:
> > >>>
> > >>> Bobby/Jungtaek,
> > >>>
> > >>> Are you saying you want to forego the 1.2 “metrics_v2” release and
> > >>> include
> > >>> it only in 2.0? (I ask because that work is already based on
> > >>> 1.x-branch,
> > >>> and forward-porting it to master is relatively simple.) I’d kind of
> > >>> like
> > >>> that work go out soon.
> > >>>
> > >>> If we go with option 1, I would want to see a 2.0 release (even if
> > >>> it’s a
> > >>> “beta” or “preview) before putting the 1.x line into maintenance
> mode.
> > >>>
> > >>> -Taylor
> > >>>
> > >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID
> <
> > >> mailto:evans@yahoo-inc.com.INVALID>
> > >>>
> > >>> wrote:
> > >>>
> > >>> I see 2 ways to address this.
> > >>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
> > >>> don't
> > >>> backport anything except bug fixes.2) We backport a lot of the
> > >>> backwards
> > >>> compatible changes from 2.x to 1.x.
> > >>> My personal preference is 1.  It makes it clear the direction we
> > >>> want to
> > >>> go in.  The biggest issue is that we probably would want to do a 2.x
> > >>> release sooner rather then later.  Even if we don't get all of the
> > >>> features
> > >>> that people want, if we just get a release out we can add in new
> > >>> features
> > >>> if they are backwards compatible, or we can create a 3.x line that
> > >>> would
> > >>> have the breaking changes in it.
> > >>>
> > >>> - Bobby
> > >>>
> > >>>
> > >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> > >>> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> > >>>
> > >>> I'd like to bump this again instead of initiating new discussion
> > >>> thread.
> > >>>
> > >>> I had having hard time to create and apply pull requests for both
> > >>> master
> > >>> and 1.x-branch and that's really painful and sometimes blocker for
> > >>> me to
> > >>> do
> > >>> merge step.
> > >>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
> > >>> even
> > >>> IDE can't switch between the branch smoothly. We didn't even address
> > >>> checkstyle issue yet, but after addressing, it could be "completely"
> > >>> diverged. JDK version is another major issue, since the pull requests
> > >>> targeted for master branch are not checked against JDK 7, and some of
> > >>> them
> > >>> make some issues regarding JDK version while porting back.
> > >>>
> > >>> So personally I really would like to see the plan for 1.x version
> > >>> line
> > >>> changed - skipping any minor releases including 1.2.0 - and have epic
> > >>> issue
> > >>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> > >>> proposed plan was having 2.0.0 directly after 1.0.0)
> > >>>
> > >>> Would like to hear everyone's opinions. If we have consensus to not
> > >>> having
> > >>> any minor releases for 1.x version line, I will not port back
> > >>> non-bugfix
> > >>> pull requests to 1.x-branch, and guide contributors to create pull
> > >>> requests
> > >>> against master branch, not 1.x version line.
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> > >>> avermeerbergen@gmail.com<ma...@gmail.com>>님이
> > >>> 작성:
> > >>>
> > >>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
> > >>> system,
> > >>> we're very interested anything that can provide better throughput.
> > >>>
> > >>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <roshan@hortonworks.com
> <mailto:
> > >> roshan@hortonworks.com>>:
> > >>>
> > >>> For 2.0 beta … it would be good to incorporate some of the Worker
> > >>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> > >>> be
> > >>> delivered sooner and my in-progress implementation suggests that it
> > >>> will
> > >>> yield substantial latency improvements. The 2.0 beta phase would
> > >>> really
> > >>> help kick the tires on the revised messaging system and the
> > >>> performance
> > >>> improvements will also be a good incentive for trying out the 2.0
> > >>> line.
> > >>>
> > >>> I notice multiple other bottlenecks that are holding back
> > >>> throughput a
> > >>> lot, which can be addressed in a subsequent 2.x minor release.
> > >>> -roshan
> > >>>
> > >>>
> > >>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabhwan@gmail.com<mailto:kabh
> > >> wan@gmail.com>> wrote:
> > >>>
> > >>>   I also would love to see metrics V2 code sooner or later too.
> > >>> If we
> > >>> can get
> > >>>   it before releasing 2.0.0 that will be great, and then maybe we
> > >>> could
> > >>> just
> > >>>   move toward to 2.0.0, not adding any improvements to 1.x version
> > >>> line.
> > >>>   (And that's what I would want to.)
> > >>>
> > >>>   If we would really want to have 1.2.0, I suggest that we make
> > >>> the
> > >>> 1.1.1
> > >>>   version correct right now rather than after releasing 1.1.1. We
> > >>> also
> > >>> merged
> > >>>   non-bugfix things to 1.x-branch but that's not what users
> > >>> expect. I
> > >>> agree
> > >>>   that work may be painful, but anyway need to do it.
> > >>>
> > >>>   - Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>>   2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> > >>> <ev...@yahoo-inc.com.invalid>
> > >>> 님이
> > >>> 작성:
> > >>>
> > >>> I would love to see the metrics V2 code come out sooner rather
> > >>> than
> > >>> later.  +1.
> > >>> My biggest blocker for a 1.x release is
> > >>> https://github.com/apache/storm/pull/2142  Even though the pull
> > >>> request
> > >>> says it is minor it showed that we messed up pushing back some
> > >>> changes for
> > >>> pacemaker to open source (the code does not run at all which for
> > >>> me
> > >>> is a
> > >>> blocker) and I really want to get that fully fixed/tested before
> > >>> another
> > >>> release.
> > >>> As for 2.x I think we are very close to being able to so a 2.x
> > >>> alpha
> > >>> release.  I would like to see metrics V2 merged in simply because
> > >>> it
> > >>> is a
> > >>> big change for user facing APIs.  But after that I would love to
> > >>> see
> > >>> us
> > >>> starting to push forward on getting that out.
> > >>>
> > >>>
> > >>> - Bobby
> > >>>
> > >>>
> > >>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> > >>> ptgoetz@gmail.com<ma...@gmail.com>> wrote:
> > >>>
> > >>> I’d like to bump this thread and start a discussion around our
> > >>> next
> > >>> release. Here are my thoughts.
> > >>>
> > >>> There are a number of important fixes in 1.x-branch so I’d like
> > >>> to
> > >>> consider releasing 1.1.1 soon. I’d appreciate input on any open
> > >>> issues that
> > >>> should be resolved for that release.
> > >>>
> > >>> I’d like us to consider releasing the metrics improvements in
> > >>> STORM-2153
> > >>> [1] as version 1.2.0. That work is in the metrics_v2 feature
> > >>> branch
> > >>> right
> > >>> now and would need to be reviewed and merged. That work is
> > >>> against
> > >>> the
> > >>> 1.x-branch right now. I would recommend porting it to master
> > >>> *after*
> > >>> the
> > >>> review/merge since there will likely be changes as a result of
> > >>> the
> > >>> review.
> > >>>
> > >>> Maybe related to or not, but would we want to create a new
> > >>> branch
> > >>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >>>
> > >>>
> > >>>
> > >>> If wee agree to the above, I would say yes. After the 1.1.1
> > >>> release,
> > >>> we
> > >>> could create a 1.1.x-branch that would be the maintenance/release
> > >>> branch
> > >>> for that version line. 1.x-branch would then become the target
> > >>> for
> > >>> the
> > >>> 1.2.0 release.
> > >>>
> > >>> There are a few fixes in the 0.10.x branch that probably warrant
> > >>> a
> > >>> release. After that we may want to back away from that version
> > >>> line
> > >>> a bit
> > >>> so we can focus more on newer versions.
> > >>>
> > >>> In the past, we’ve shied away form doing “beta” releases, but I’m
> > >>> wondering if we might want to revisit that for the 2.0 release —
> > >>> the
> > >>> idea
> > >>> being that it would give early adopter users a chance to kick the
> > >>> tires on
> > >>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
> > >>> help
> > >>> make the
> > >>> final release more solid. I’m on the fence here and could go
> > >>> either
> > >>> way.
> > >>>
> > >>> I’d appreciate any input others may have.
> > >>>
> > >>>
> > >>> Thanks,
> > >>>
> > >>> -Taylor
> > >>>
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/STORM-2153
> > >>>
> > >>>
> > >>>
> > >>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabhwan@gmail.com<mailto:
> > kabh
> > >> wan@gmail.com>>
> > >>> wrote:
> > >>>
> > >>> Maybe related to or not, but would we want to create a new
> > >>> branch
> > >>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >>>
> > >>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> > >>> so
> > >>> hence
> > >>> the
> > >>> question.
> > >>>
> > >>> - Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > >>> hlouro@hortonworks.com<ma...@hortonworks.com>>님이
> > >>> 작성:
> > >>>
> > >>> +1 for finishing the porting to Java ahead of anything else -
> > >>> it
> > >>> will
> > >>> be a
> > >>> significant milestone. I have a JIRA assigned concerning to
> > >>> the
> > >>> porting. I
> > >>> will work on it for the 2.0 release.
> > >>>
> > >>> it’s a priority to guarantee no performance regressions. As
> > >>> part
> > >>> of this
> > >>> endeavor, explore an automated (or easy) way to run and assert
> > >>> major
> > >>> performance benchmarks. Ideally any contributor should be able
> > >>> to
> > >>> fairly
> > >>> easily test the impact of changes under certain performance
> > >>> test
> > >>> scenarios.
> > >>>
> > >>> Beam Runner work should take into account the impact of
> > >>> incorporating
> > >>> new
> > >>> JStorm features and Storm Worker Redesign<
> > >>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > >>> efficient
> > >>> to
> > >>> start doing it, to  find out that it will have to chance in
> > >>> face
> > >>> of
> > >>> Storm
> > >>> and worker redesign. That is, it should be done after it’s
> > >>> building
> > >>> blocks
> > >>> are stable.
> > >>>
> > >>> Thanks,
> > >>> Hugo
> > >>>
> > >>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> > >>> arunm@apache.org<ma...@apache.org>
> > >>> <mailto:
> > >>> arunm@apache.org>> wrote:
> > >>>
> > >>> +1 to release with the porting completed. I think its mainly
> > >>> the
> > >>> UI
> > >>> server
> > >>> and log viewer that’s pending.
> > >>>
> > >>> We can start doing the regression and performance tests for
> > >>> whatever is
> > >>> already ported.
> > >>>
> > >>> If anyone is running the master branch in their pre-prod /
> > >>> prod
> > >>> environments, it will be good to know and give us more
> > >>> confidence.
> > >>>
> > >>> The other features can be added in follow up releases.
> > >>>
> > >>> Regards,
> > >>> Arun
> > >>>
> > >>>
> > >>> On 3/24/17, 11:47 AM, "Satish Duggana" <
> > >>> satish.duggana@gmail.com
> > >>> <mailto:
> > >>> satish.duggana@gmail.com>> wrote:
> > >>>
> > >>> +1 to have 2.0 with porting and performance(it should be at
> > >>> least
> > >>> as
> > >>> good
> > >>> as 1.x release) issues addressed
> > >>>
> > >>> We can target other tasks(mentioned by Taylor and Jungtaek)
> > >>> for
> > >>> 2.x-branch.
> > >>>
> > >>>
> > >>> Exactly-once support:
> > >>> While thinking through the exactlyonce support design, it is
> > >>> realized
> > >>> better to avoid acking tuples and implement exactly once by
> > >>> snapshotting
> > >>> barriers. It seems JStorm folks followed similar design, they
> > >>> claim it
> > >>> gives better performance. This feature is essential for beam
> > >>> runner and
> > >>> we
> > >>> can decide on respective approaches though.
> > >>>
> > >>> Beam Runner
> > >>> Lets hold on this for now and keep it in Storm till 2.x. We
> > >>> should avoid
> > >>> having a minimal beam runner in haste. It is better to address
> > >>> STORM-2284,
> > >>> exactly-once and other windowing enhancements to enable beam
> > >>> runner.
> > >>>
> > >>> JStorm
> > >>> Agree with Jungtaek on looking at the latest JStorm and
> > >>> align/scope with
> > >>> the features for 2.x.
> > >>>
> > >>> STORM-2284
> > >>> We may want to look at JStorm worker before working on
> > >>> respective
> > >>> components in this epic to pull appropriate enhancements.
> > >>>
> > >>> YARN/MESOS
> > >>> Supporting Storm on YARN/Mesos for 2.x.
> > >>>
> > >>> Thanks,
> > >>> Satish.
> > >>>
> > >>>
> > >>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> > >>> kabhwan@gmail.com
> > >>> <mailto:
> > >>> kabhwan@gmail.com>> wrote:
> > >>>
> > >>> First of all, +1 to complete only port work and do sanity
> > >>> check
> > >>> (including
> > >>> performance regression), and release.
> > >>>
> > >>> If we can get STORM-2284 within deterministic time frame (say
> > >>> 2~3
> > >>> months)
> > >>> that should be great, but if not I'd in favor of postponing
> > >>> that
> > >>> to
> > >>> later
> > >>> 2.x release.
> > >>>
> > >>> JStorm released their new versions after code donation. So
> > >>> there're more
> > >>> things we could get ideas from, or even adopt from.
> > >>> https://github.com/alibaba/jstorm/blob/master/history.md
> > >>> As you noticed from release note link, we also need to update
> > >>> phase 2
> > >>> since
> > >>> they already changed what we're planning to do in phase 2. For
> > >>> example,
> > >>> they changed backpressure to end-to-end, and changed to use
> > >>> snapshot
> > >>> rather
> > >>> than acker.
> > >>> May be sure, JStorm pulled many features from today's Storm,
> > >>> like
> > >>> Flux,
> > >>> Windowing, more shuffle groupings, log search, log level
> > >>> change,
> > >>> and so
> > >>> on.
> > >>>
> > >>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> > >>> is
> > >>> due to
> > >>> the
> > >>> limitation of Spout lifecycle (all the things are done in
> > >>> single
> > >>> thread),
> > >>> and STORM-1358 <
> > >>> https://issues.apache.org/jira/browse/STORM-1358
> > >>> (JStorm's
> > >>> multi-thread Spout) can remedy this (despite that Spout
> > >>> implementation
> > >>> may
> > >>> need to guarantee thread-safety later). It's not a just
> > >>> improvement but
> > >>> close to design concern so would like to address sooner than
> > >>> other
> > >>> things
> > >>> in phase 2.
> > >>>
> > >>> For Storm SQL side, I've lost progress but major work would be
> > >>> adopting
> > >>> group by with windowing. It was not available from Calcite but
> > >>> will be
> > >>> available at next release (1.12.0).
> > >>> I've filed this to STORM-2405
> > >>> <https://issues.apache.org/jira/browse/STORM-2405>, but
> > >>> windowing &
> > >>> micro
> > >>> batch is not intuitive, so I would like to change the
> > >>> underlying
> > >>> API to
> > >>> stream API in SQL. Also filed this to STORM-2406
> > >>> <https://issues.apache.org/jira/browse/STORM-2406>.
> > >>>
> > >>> Just 2 cents btw, hopefully I would like to see metrics V2
> > >>> sooner
> > >>> since
> > >>> we
> > >>> lost metrics even when doing normal operation like restarting
> > >>> worker,
> > >>> rebalancing, and so on. Eventually we need to fight with
> > >>> dynamic
> > >>> scaling,
> > >>> and then metrics will be broken often.
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> > >>> storm@harsha.io
> > >>> 님이
> > >>> 작성:
> > >>>
> > >>> Storm 2.0 migration to java in itself is a big win and would
> > >>> attract
> > >>> wider
> > >>> community and adoption. So my vote would be to resolve the
> > >>> first
> > >>> 3 items
> > >>> to
> > >>> get a release out.
> > >>> All the other featured mentioned are great to have but
> > >>> shouldn't
> > >>> be
> > >>> blockers for 2.0 release.
> > >>>
> > >>> -Harsha
> > >>>
> > >>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > >>> ptgoetz@gmail.com>
> > >>> wrote:
> > >>>
> > >>> With the 1.1.0 release nearing completion, I’d like to turn
> > >>> our
> > >>> attention
> > >>> to 2.0 and develop a plan for what features, etc. to include.
> > >>>
> > >>> The following 3 are what I feel are the minimum for a 2.0
> > >>> release.
> > >>> These
> > >>> could likely be resolved relatively quickly:
> > >>>
> > >>> * Performance — I’ve not benchmarked the master branch vs.
> > >>> 1.0.x
> > >>> or
> > >>> 1.1.x
> > >>> in a while, but I feel it will be important to make sure there
> > >>> are no
> > >>> performance regressions, and would hope that we actually have
> > >>> a
> > >>> performance
> > >>> improvement over previous versions. To that end (e.g. if there
> > >>> is
> > >>> in
> > >>> fact a
> > >>> performance regression), the proposals that Roshan Naik put
> > >>> together
> > >>> for
> > >>> revising the threading and execution model (STORM-2307) and
> > >>> replacing
> > >>> Disruptor with JCTools (STORM-2306) warrant review and
> > >>> consideration.
> > >>> See
> > >>> also STORM-2284 which is the parent JIRA.
> > >>>
> > >>> * Finish porting Storm UI to java (STORM-1311)
> > >>>
> > >>> * Finish porting log viewer to java (STORM-1280)
> > >>>
> > >>> The following are items that are nice to have in 2.0, but I
> > >>> don’t
> > >>> feel
> > >>> are
> > >>> absolutely necessary for an initial 2.0 release:
> > >>>
> > >>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> > >>> because
> > >>> it’s
> > >>> relevant) — Initially there seemed to be a lot of interest in
> > >>> this, but
> > >>> that seems to have trailed off. I spoke with some Beam
> > >>> developers
> > >>> and
> > >>> there
> > >>> seems to be interest from that community as well. Do we want
> > >>> to
> > >>> move
> > >>> that
> > >>> effort to the Beam community, or keep it here? Moving it to
> > >>> the
> > >>> Beam
> > >>> community might lead to better collaboration between projects.
> > >>>
> > >>> * Bounded Spouts (needed for Beam Runner implementation) —
> > >>> Currently
> > >>> spouts are unbounded, there no end to the stream. Beam has the
> > >>> concept
> > >>> of
> > >>> bounded sources (roughly analogous to batch processing). To
> > >>> support
> > >>> that,
> > >>> we would need to implement a similar concept in Storm. One
> > >>> benefit of
> > >>> such
> > >>> a feature would be the ability to handle both bounded and
> > >>> unbounded
> > >>> workflows in Storm.
> > >>>
> > >>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > >>> behind
> > >>> this
> > >>> effort. What improvements do you envision for 2.0?
> > >>>
> > >>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > >>> targeting this
> > >>> for 1.2.0, but it’s designed to be easily portable to
> > >>> master/2.0.
> > >>>
> > >>> * JStorm Migration — Original outline can be found here [1].
> > >>> Note
> > >>> a lot
> > >>> of
> > >>> the associated JIRAs below are assigned, but there hasn’t been
> > >>> any
> > >>> recent
> > >>> activity or pull requests, we should probably consider them
> > >>> unassigned
> > >>> and
> > >>> up for grabs.:
> > >>>
> > >>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > >>> been the
> > >>> bane of a lot of Storm users almost since day one. We have
> > >>> largely
> > >>> addressed it by shading/relocating dependencies. It would be
> > >>> great to
> > >>> see
> > >>> this addressed once and for all.
> > >>>
> > >>> * JStorm back pressure implementation (STORM-1324) — The
> > >>> current
> > >>> back
> > >>> pressure implementation leaves a bit to be desired, and the
> > >>> JStorm
> > >>> approach
> > >>> looks promising, though it also depends on the JStorm concept
> > >>> of
> > >>> “topology
> > >>> master” (STORM-1323), which may have some implications
> > >>> regarding
> > >>> security.
> > >>>
> > >>> * Dynamic Topology Updates (STORM-1335) — This would provide a
> > >>> command
> > >>> to
> > >>> update topology jars and configuration without stopping the
> > >>> topology,
> > >>> and
> > >>> is well suited to leverage the blobstore. The restart command
> > >>> (that can
> > >>> also update the topology configuration) also looks compelling
> > >>> (STORM-1334).
> > >>>
> > >>> * Additional Scheduler Implementations (STORM-1320)
> > >>>
> > >>> * Additional Grouping Implementations (STORM-1328)
> > >>>
> > >>>
> > >>> As always I’m open to any opinions and suggestions.
> > >>>
> > >>> -Taylor
> > >>>
> > >>> [1]
> > >>>
> > >>> https://cwiki.apache.org/confluence/pages/viewpage.
> > >>> action?pageId=61328109
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hi Hugo,

Thanks for your concerns about our troubles with the new storm-kafka-client.

Our "bench" is based on our live production data of our cloud supervision
system, collecting  at least 1million metrics/min in our Kafka Brokers
cluster (currently based on Kafka 0.10.1.0, with "compatibility flag
active").

More details are available in the thread in the same dev list entitled "Lag
issues using Storm 1.1.1 latest build with StormKafkaClient 1.1.1 vs old
StormKafka spouts".

The latest post on this thread from Stig Døssing is giving me back some
hope to see some progress in understanding our issues.

My point about writing our own Spout come from our past experience: we've
been using Kafka for a very long time in our supervision application. Way
before we decided to use Storm, we had our own Java daemons consuming the
same topics as today, doing some evaluation and writing them into an
in-memory store for later consumption by our web services - see this as
a"poor man's streaming system" ;-)  In this legacy code, the part in charge
of consuming data from Kafka wasn't the most complex which we had: a small
pool of threads using the old Kafka consumer API... so maybe I'm wrong, but
for *this* purpose I do not feel like writing a Spout consuming a few
topics to be a big effort. But of course, if we do that, then we'll miss
the fancy integration in StormUI, flux, and the ability to subscribe to
multiple topics based on a wildcard expression.

So we're going to carefully dig into Stig's answer, and probably provide
more details on our bench before jumping into our home-brewed Kafka
spout... then I'll have to make some decision based on how much progress we
have vs time remaining before our next delivery gate.

Hope it clarifies my position with regard to storm-kafka-client

Best regards,
Alexandre






2017-06-28 22:49 GMT+02:00 Hugo Da Cruz Louro <hl...@hortonworks.com>:

> Hi Alexandre,
>
> In my benchmarks the storm-kafka-client spout improves throughput by 70%
> and latency by 40% vs the storm-kafka implementation. I am surprised by
> your findings substantiating the opposite. Can you share your benchmark
> where you compare the performances of both implementations?
>
> As for you writing your own version of the spout. Why not contribute to
> this one instead? Do you think the implementation is that poor? If so, why
> do you think it is that poor? Do you expect your first version to be much
> better than a version that is already in production in several customers,
> and seems to be working fairly well?
>
> All the bugs found so far have been addressed. Since it’s a new feature
> there may be a few bugs - it is expected. However, I don’t think that it is
> as bad as you make it sound as there are several people using it in
> production for extended periods of time.
>
> Cheers
>
> > On Jun 28, 2017, at 12:04 PM, Alexandre Vermeerbergen <
> avermeerbergen@gmail.com> wrote:
> >
> > Hello,
> >
> > If that matters, our current experiences with StormKafkaClient
> > isdisappointing (see my recent posts "Lag issues using Storm 1.1.1 latest
> > build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this
> mailing
> > list).
> >
> > Our current experience is that the old StormKafka spout always beats the
> > new one in term of performance & stability.
> >
> > Therefore, I am surprised when I see talks about deprecation of the old
> > StormKafka spout when the new one which just came "General Available"
> with
> > Storm 1.1.0, is not stable, and it's not better when we try it from
> current
> > 1.1.x builds to take into account recently closed JIRAs.
> >
> > We're even considering writing our own Kafka spout with Kafka 0.10.x API
> to
> > overcome the incompatibility of the old StormKafka spout with Kafka 0.10
> > libraries.
> >
> > Thus, for people which are comfortable with old Kafka spout, I'like to
> give
> > a -1 (non binding) to the proposal of withdrawal of the old StormKafka
> > spout until the new one converges.
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
> >
> > 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:
> >
> >>
> >>> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <
> hlouro@hortonworks.com>
> >> wrote:
> >>>
> >>> I still need to go over the entire discussion thread in more detail,
> but
> >> one thing I would like to bring up right way is the proposal to
> DEPRECATE,
> >> and eventually remove, the KafkaSpout with the old Kafka Consumer APIs.
> The
> >> storm-kafka-client KafkaSpout is getting stabilized, and I think we are
> all
> >> in agreement that the storm-kafka KafkaSpout has presented continuous
> >> maintainability problems with some fixes that got in not being backwards
> >> compatible.
> >>
> >> I’m fine with deprecating the old KafkaSpout, but I feel the decision to
> >> actually remove it needs to take into account the user community. The
> main
> >> sticking point here is compatibility with earlier versions of Kafka.
> Like
> >> with JDK versions, there are many valid reasons whey users may not be
> in a
> >> position to upgrade to a newer version of Kafka. Outright removal could
> >> leave some users in the lurch.
> >>
> >> Ideally, we could just poll the user community to get an idea of how
> much
> >> of the user base depends on the old KafkaSpout and use the results to
> guide
> >> our decision. Unfortunately, at least in my past experience, polling the
> >> user@ list doesn’t elicit much of a response and the results don’t
> >> provide an accurate view of the user community.
> >>
> >>
> >>>
> >>> I am pretty confident how things are looking at this point for the
> >> KafkaSpout. The Trident Kafka Spout is likely in between alpha and beta,
> >> and that should be taken into account. I just recently submitted a PR<
> >> https://github.com/apache/storm/pull/2174> with some improvements to
> the
> >> Trident Kafka Spout (including the refactoring done to support manual
> >> partition assignment), and there are some customers using it in
> >> pre-production. However, it definitely would benefit from some more
> testing.
> >>>
> >>> Thanks,
> >>> Hugo
> >>
> >> -Taylor
> >>
> >>
> >>>
> >>> On Jun 28, 2017, at 7:48 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
> >> mailto:evans@yahoo-inc.com.INVALID>> wrote:
> >>>
> >>> +1.
> >>> If the 1.1 and 1.2 lines start to become difficult to maintain we can
> >> look at putting them in maintenance mode too once we have a 2.x release.
> >>> I am a little nervous about merging a new feature into 1.x branch
> >> without first going to master, but I hope that it will not be too much
> work
> >> to port it to master, and I trust the devs on that branch to do the
> right
> >> thing.
> >>> On a related note we have not done much with feature branches before so
> >> I am not sure what we want to do about merging in the new metrics API
> >> branch to 1.x.  I know for me I have not had time to keep up with the
> >> development work going on there.  I would at least like to have a pull
> >> request put up for review before we merge it in.  This would fit with
> our
> >> current bylaws that do not mention feature branches.  If all of the
> changes
> >> have already followed the review process then technically I think it is
> OK
> >> to just merge it in, but I still would like to take some time to look at
> >> the changes, and especially the new APIs.
> >>>
> >>> - Bobby
> >>>
> >>>
> >>> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <
> >> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> >>>
> >>> That's great news that metrics work is ready!
> >>>
> >>> I'm +1 to Taylor's proposal, but in order to respect semantic
> >> versioning, I
> >>> propose some modifications from Taylor's proposal:
> >>>
> >>> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back
> >> only
> >>> bug fixes to the 1.1.x-branch
> >>> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
> >>>
> >>> If we also agree above, I would like to volunteer the back-port work.
> >>>
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>>
> >>> 2017년 6월 28일 (수) 오전 10:09, Harsha <storm@harsha.io<mailto:storm@
> >> harsha.io>>님이 작성:
> >>>
> >>> +1 for above stated approach on releasing 1.2.0 with metrics
> >>> -Harsha
> >>>
> >>> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> >>> The work on metrics is ready for a pull request to 1.x-branch from the
> >>> feature branch. I’ve held off because we haven’t reached consensus on a
> >>> path forward with the 1.x release lines .
> >>>
> >>> I’d like to propose the following for the 1.x line:
> >>>
> >>> 1. Create a branch for 1.2 so we have a branch to review the metrics
> >>> stuff.
> >>> 2. Release 1.1.1
> >>> 3. Review/merge metrics work. Port metrics to master.
> >>> 4. Release 1.2.0
> >>> 5. Put the entire 1.x line into maintenance mode. Drop support for
> 1.0.x.
> >>> (we would only support 1.2.x and 1.1.x which are very closely aligned).
> >>>
> >>> Dropping support for 1.0.x line would eliminate the need to maintain
> one
> >>> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> >>> very closely aligned. I just up merged metrics_v2 against 1.x-branch
> >>> after a while, and there were no conflicts.
> >>>
> >>> That would give us a little more bandwidth to focus on 2.0 and needed
> bug
> >>> fixes to the 1.x line like some of the issues raised with
> >>> storm-kafka-client. We could even start releasing alpha/beta versions
> of
> >>> 2.0 in parallel to the steps above.
> >>>
> >>> Any thoughts on that approach?
> >>>
> >>> -Taylor
> >>>
> >>>
> >>> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:
> kabh
> >> wan@gmail.com>> wrote:
> >>>
> >>> Yes I prefer option 1, but it might depend on the progress of metrics
> >>> V2.
> >>> If it can be done within predictable near future I'm OK to pick option
> >>> 2,
> >>> but if not, we may be better to focus releasing 2.0.0 and make it
> >>> really
> >>> happen.
> >>>
> >>> Whichever we go, I feel it's time to track remaining work on Storm
> >>> 2.0.0. I
> >>> found some bugs on master branch so filed issues, and we've remaining
> >>> port
> >>> work (UI and logviewer). We've some other improvements target for
> >>> 2.0.0:
> >>> worker redesign, beam integration, and so on, and we don't track its
> >>> progress at all. I don't think we should wait for features which
> >>> progress
> >>> is not transparent (in other words we don't know when it will be
> >>> finished).
> >>>
> >>> - Jungtaek Lim (HeartSaVioR)
> >>>
> >>> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
> ptgo
> >> etz@gmail.com>>님이 작성:
> >>>
> >>> Bobby/Jungtaek,
> >>>
> >>> Are you saying you want to forego the 1.2 “metrics_v2” release and
> >>> include
> >>> it only in 2.0? (I ask because that work is already based on
> >>> 1.x-branch,
> >>> and forward-porting it to master is relatively simple.) I’d kind of
> >>> like
> >>> that work go out soon.
> >>>
> >>> If we go with option 1, I would want to see a 2.0 release (even if
> >>> it’s a
> >>> “beta” or “preview) before putting the 1.x line into maintenance mode.
> >>>
> >>> -Taylor
> >>>
> >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
> >> mailto:evans@yahoo-inc.com.INVALID>
> >>>
> >>> wrote:
> >>>
> >>> I see 2 ways to address this.
> >>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
> >>> don't
> >>> backport anything except bug fixes.2) We backport a lot of the
> >>> backwards
> >>> compatible changes from 2.x to 1.x.
> >>> My personal preference is 1.  It makes it clear the direction we
> >>> want to
> >>> go in.  The biggest issue is that we probably would want to do a 2.x
> >>> release sooner rather then later.  Even if we don't get all of the
> >>> features
> >>> that people want, if we just get a release out we can add in new
> >>> features
> >>> if they are backwards compatible, or we can create a 3.x line that
> >>> would
> >>> have the breaking changes in it.
> >>>
> >>> - Bobby
> >>>
> >>>
> >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> >>> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> >>>
> >>> I'd like to bump this again instead of initiating new discussion
> >>> thread.
> >>>
> >>> I had having hard time to create and apply pull requests for both
> >>> master
> >>> and 1.x-branch and that's really painful and sometimes blocker for
> >>> me to
> >>> do
> >>> merge step.
> >>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
> >>> even
> >>> IDE can't switch between the branch smoothly. We didn't even address
> >>> checkstyle issue yet, but after addressing, it could be "completely"
> >>> diverged. JDK version is another major issue, since the pull requests
> >>> targeted for master branch are not checked against JDK 7, and some of
> >>> them
> >>> make some issues regarding JDK version while porting back.
> >>>
> >>> So personally I really would like to see the plan for 1.x version
> >>> line
> >>> changed - skipping any minor releases including 1.2.0 - and have epic
> >>> issue
> >>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> >>> proposed plan was having 2.0.0 directly after 1.0.0)
> >>>
> >>> Would like to hear everyone's opinions. If we have consensus to not
> >>> having
> >>> any minor releases for 1.x version line, I will not port back
> >>> non-bugfix
> >>> pull requests to 1.x-branch, and guide contributors to create pull
> >>> requests
> >>> against master branch, not 1.x version line.
> >>>
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>>
> >>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> >>> avermeerbergen@gmail.com<ma...@gmail.com>>님이
> >>> 작성:
> >>>
> >>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
> >>> system,
> >>> we're very interested anything that can provide better throughput.
> >>>
> >>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <roshan@hortonworks.com<mailto:
> >> roshan@hortonworks.com>>:
> >>>
> >>> For 2.0 beta … it would be good to incorporate some of the Worker
> >>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> >>> be
> >>> delivered sooner and my in-progress implementation suggests that it
> >>> will
> >>> yield substantial latency improvements. The 2.0 beta phase would
> >>> really
> >>> help kick the tires on the revised messaging system and the
> >>> performance
> >>> improvements will also be a good incentive for trying out the 2.0
> >>> line.
> >>>
> >>> I notice multiple other bottlenecks that are holding back
> >>> throughput a
> >>> lot, which can be addressed in a subsequent 2.x minor release.
> >>> -roshan
> >>>
> >>>
> >>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabhwan@gmail.com<mailto:kabh
> >> wan@gmail.com>> wrote:
> >>>
> >>>   I also would love to see metrics V2 code sooner or later too.
> >>> If we
> >>> can get
> >>>   it before releasing 2.0.0 that will be great, and then maybe we
> >>> could
> >>> just
> >>>   move toward to 2.0.0, not adding any improvements to 1.x version
> >>> line.
> >>>   (And that's what I would want to.)
> >>>
> >>>   If we would really want to have 1.2.0, I suggest that we make
> >>> the
> >>> 1.1.1
> >>>   version correct right now rather than after releasing 1.1.1. We
> >>> also
> >>> merged
> >>>   non-bugfix things to 1.x-branch but that's not what users
> >>> expect. I
> >>> agree
> >>>   that work may be painful, but anyway need to do it.
> >>>
> >>>   - Jungtaek Lim (HeartSaVioR)
> >>>
> >>>   2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> >>> <ev...@yahoo-inc.com.invalid>
> >>> 님이
> >>> 작성:
> >>>
> >>> I would love to see the metrics V2 code come out sooner rather
> >>> than
> >>> later.  +1.
> >>> My biggest blocker for a 1.x release is
> >>> https://github.com/apache/storm/pull/2142  Even though the pull
> >>> request
> >>> says it is minor it showed that we messed up pushing back some
> >>> changes for
> >>> pacemaker to open source (the code does not run at all which for
> >>> me
> >>> is a
> >>> blocker) and I really want to get that fully fixed/tested before
> >>> another
> >>> release.
> >>> As for 2.x I think we are very close to being able to so a 2.x
> >>> alpha
> >>> release.  I would like to see metrics V2 merged in simply because
> >>> it
> >>> is a
> >>> big change for user facing APIs.  But after that I would love to
> >>> see
> >>> us
> >>> starting to push forward on getting that out.
> >>>
> >>>
> >>> - Bobby
> >>>
> >>>
> >>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> >>> ptgoetz@gmail.com<ma...@gmail.com>> wrote:
> >>>
> >>> I’d like to bump this thread and start a discussion around our
> >>> next
> >>> release. Here are my thoughts.
> >>>
> >>> There are a number of important fixes in 1.x-branch so I’d like
> >>> to
> >>> consider releasing 1.1.1 soon. I’d appreciate input on any open
> >>> issues that
> >>> should be resolved for that release.
> >>>
> >>> I’d like us to consider releasing the metrics improvements in
> >>> STORM-2153
> >>> [1] as version 1.2.0. That work is in the metrics_v2 feature
> >>> branch
> >>> right
> >>> now and would need to be reviewed and merged. That work is
> >>> against
> >>> the
> >>> 1.x-branch right now. I would recommend porting it to master
> >>> *after*
> >>> the
> >>> review/merge since there will likely be changes as a result of
> >>> the
> >>> review.
> >>>
> >>> Maybe related to or not, but would we want to create a new
> >>> branch
> >>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >>>
> >>>
> >>>
> >>> If wee agree to the above, I would say yes. After the 1.1.1
> >>> release,
> >>> we
> >>> could create a 1.1.x-branch that would be the maintenance/release
> >>> branch
> >>> for that version line. 1.x-branch would then become the target
> >>> for
> >>> the
> >>> 1.2.0 release.
> >>>
> >>> There are a few fixes in the 0.10.x branch that probably warrant
> >>> a
> >>> release. After that we may want to back away from that version
> >>> line
> >>> a bit
> >>> so we can focus more on newer versions.
> >>>
> >>> In the past, we’ve shied away form doing “beta” releases, but I’m
> >>> wondering if we might want to revisit that for the 2.0 release —
> >>> the
> >>> idea
> >>> being that it would give early adopter users a chance to kick the
> >>> tires on
> >>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
> >>> help
> >>> make the
> >>> final release more solid. I’m on the fence here and could go
> >>> either
> >>> way.
> >>>
> >>> I’d appreciate any input others may have.
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> -Taylor
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/STORM-2153
> >>>
> >>>
> >>>
> >>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabhwan@gmail.com<mailto:
> kabh
> >> wan@gmail.com>>
> >>> wrote:
> >>>
> >>> Maybe related to or not, but would we want to create a new
> >>> branch
> >>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >>>
> >>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> >>> so
> >>> hence
> >>> the
> >>> question.
> >>>
> >>> - Jungtaek Lim (HeartSaVioR)
> >>>
> >>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> >>> hlouro@hortonworks.com<ma...@hortonworks.com>>님이
> >>> 작성:
> >>>
> >>> +1 for finishing the porting to Java ahead of anything else -
> >>> it
> >>> will
> >>> be a
> >>> significant milestone. I have a JIRA assigned concerning to
> >>> the
> >>> porting. I
> >>> will work on it for the 2.0 release.
> >>>
> >>> it’s a priority to guarantee no performance regressions. As
> >>> part
> >>> of this
> >>> endeavor, explore an automated (or easy) way to run and assert
> >>> major
> >>> performance benchmarks. Ideally any contributor should be able
> >>> to
> >>> fairly
> >>> easily test the impact of changes under certain performance
> >>> test
> >>> scenarios.
> >>>
> >>> Beam Runner work should take into account the impact of
> >>> incorporating
> >>> new
> >>> JStorm features and Storm Worker Redesign<
> >>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> >>> efficient
> >>> to
> >>> start doing it, to  find out that it will have to chance in
> >>> face
> >>> of
> >>> Storm
> >>> and worker redesign. That is, it should be done after it’s
> >>> building
> >>> blocks
> >>> are stable.
> >>>
> >>> Thanks,
> >>> Hugo
> >>>
> >>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> >>> arunm@apache.org<ma...@apache.org>
> >>> <mailto:
> >>> arunm@apache.org>> wrote:
> >>>
> >>> +1 to release with the porting completed. I think its mainly
> >>> the
> >>> UI
> >>> server
> >>> and log viewer that’s pending.
> >>>
> >>> We can start doing the regression and performance tests for
> >>> whatever is
> >>> already ported.
> >>>
> >>> If anyone is running the master branch in their pre-prod /
> >>> prod
> >>> environments, it will be good to know and give us more
> >>> confidence.
> >>>
> >>> The other features can be added in follow up releases.
> >>>
> >>> Regards,
> >>> Arun
> >>>
> >>>
> >>> On 3/24/17, 11:47 AM, "Satish Duggana" <
> >>> satish.duggana@gmail.com
> >>> <mailto:
> >>> satish.duggana@gmail.com>> wrote:
> >>>
> >>> +1 to have 2.0 with porting and performance(it should be at
> >>> least
> >>> as
> >>> good
> >>> as 1.x release) issues addressed
> >>>
> >>> We can target other tasks(mentioned by Taylor and Jungtaek)
> >>> for
> >>> 2.x-branch.
> >>>
> >>>
> >>> Exactly-once support:
> >>> While thinking through the exactlyonce support design, it is
> >>> realized
> >>> better to avoid acking tuples and implement exactly once by
> >>> snapshotting
> >>> barriers. It seems JStorm folks followed similar design, they
> >>> claim it
> >>> gives better performance. This feature is essential for beam
> >>> runner and
> >>> we
> >>> can decide on respective approaches though.
> >>>
> >>> Beam Runner
> >>> Lets hold on this for now and keep it in Storm till 2.x. We
> >>> should avoid
> >>> having a minimal beam runner in haste. It is better to address
> >>> STORM-2284,
> >>> exactly-once and other windowing enhancements to enable beam
> >>> runner.
> >>>
> >>> JStorm
> >>> Agree with Jungtaek on looking at the latest JStorm and
> >>> align/scope with
> >>> the features for 2.x.
> >>>
> >>> STORM-2284
> >>> We may want to look at JStorm worker before working on
> >>> respective
> >>> components in this epic to pull appropriate enhancements.
> >>>
> >>> YARN/MESOS
> >>> Supporting Storm on YARN/Mesos for 2.x.
> >>>
> >>> Thanks,
> >>> Satish.
> >>>
> >>>
> >>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> >>> kabhwan@gmail.com
> >>> <mailto:
> >>> kabhwan@gmail.com>> wrote:
> >>>
> >>> First of all, +1 to complete only port work and do sanity
> >>> check
> >>> (including
> >>> performance regression), and release.
> >>>
> >>> If we can get STORM-2284 within deterministic time frame (say
> >>> 2~3
> >>> months)
> >>> that should be great, but if not I'd in favor of postponing
> >>> that
> >>> to
> >>> later
> >>> 2.x release.
> >>>
> >>> JStorm released their new versions after code donation. So
> >>> there're more
> >>> things we could get ideas from, or even adopt from.
> >>> https://github.com/alibaba/jstorm/blob/master/history.md
> >>> As you noticed from release note link, we also need to update
> >>> phase 2
> >>> since
> >>> they already changed what we're planning to do in phase 2. For
> >>> example,
> >>> they changed backpressure to end-to-end, and changed to use
> >>> snapshot
> >>> rather
> >>> than acker.
> >>> May be sure, JStorm pulled many features from today's Storm,
> >>> like
> >>> Flux,
> >>> Windowing, more shuffle groupings, log search, log level
> >>> change,
> >>> and so
> >>> on.
> >>>
> >>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> >>> is
> >>> due to
> >>> the
> >>> limitation of Spout lifecycle (all the things are done in
> >>> single
> >>> thread),
> >>> and STORM-1358 <
> >>> https://issues.apache.org/jira/browse/STORM-1358
> >>> (JStorm's
> >>> multi-thread Spout) can remedy this (despite that Spout
> >>> implementation
> >>> may
> >>> need to guarantee thread-safety later). It's not a just
> >>> improvement but
> >>> close to design concern so would like to address sooner than
> >>> other
> >>> things
> >>> in phase 2.
> >>>
> >>> For Storm SQL side, I've lost progress but major work would be
> >>> adopting
> >>> group by with windowing. It was not available from Calcite but
> >>> will be
> >>> available at next release (1.12.0).
> >>> I've filed this to STORM-2405
> >>> <https://issues.apache.org/jira/browse/STORM-2405>, but
> >>> windowing &
> >>> micro
> >>> batch is not intuitive, so I would like to change the
> >>> underlying
> >>> API to
> >>> stream API in SQL. Also filed this to STORM-2406
> >>> <https://issues.apache.org/jira/browse/STORM-2406>.
> >>>
> >>> Just 2 cents btw, hopefully I would like to see metrics V2
> >>> sooner
> >>> since
> >>> we
> >>> lost metrics even when doing normal operation like restarting
> >>> worker,
> >>> rebalancing, and so on. Eventually we need to fight with
> >>> dynamic
> >>> scaling,
> >>> and then metrics will be broken often.
> >>>
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>>
> >>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> >>> storm@harsha.io
> >>> 님이
> >>> 작성:
> >>>
> >>> Storm 2.0 migration to java in itself is a big win and would
> >>> attract
> >>> wider
> >>> community and adoption. So my vote would be to resolve the
> >>> first
> >>> 3 items
> >>> to
> >>> get a release out.
> >>> All the other featured mentioned are great to have but
> >>> shouldn't
> >>> be
> >>> blockers for 2.0 release.
> >>>
> >>> -Harsha
> >>>
> >>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> >>> ptgoetz@gmail.com>
> >>> wrote:
> >>>
> >>> With the 1.1.0 release nearing completion, I’d like to turn
> >>> our
> >>> attention
> >>> to 2.0 and develop a plan for what features, etc. to include.
> >>>
> >>> The following 3 are what I feel are the minimum for a 2.0
> >>> release.
> >>> These
> >>> could likely be resolved relatively quickly:
> >>>
> >>> * Performance — I’ve not benchmarked the master branch vs.
> >>> 1.0.x
> >>> or
> >>> 1.1.x
> >>> in a while, but I feel it will be important to make sure there
> >>> are no
> >>> performance regressions, and would hope that we actually have
> >>> a
> >>> performance
> >>> improvement over previous versions. To that end (e.g. if there
> >>> is
> >>> in
> >>> fact a
> >>> performance regression), the proposals that Roshan Naik put
> >>> together
> >>> for
> >>> revising the threading and execution model (STORM-2307) and
> >>> replacing
> >>> Disruptor with JCTools (STORM-2306) warrant review and
> >>> consideration.
> >>> See
> >>> also STORM-2284 which is the parent JIRA.
> >>>
> >>> * Finish porting Storm UI to java (STORM-1311)
> >>>
> >>> * Finish porting log viewer to java (STORM-1280)
> >>>
> >>> The following are items that are nice to have in 2.0, but I
> >>> don’t
> >>> feel
> >>> are
> >>> absolutely necessary for an initial 2.0 release:
> >>>
> >>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> >>> because
> >>> it’s
> >>> relevant) — Initially there seemed to be a lot of interest in
> >>> this, but
> >>> that seems to have trailed off. I spoke with some Beam
> >>> developers
> >>> and
> >>> there
> >>> seems to be interest from that community as well. Do we want
> >>> to
> >>> move
> >>> that
> >>> effort to the Beam community, or keep it here? Moving it to
> >>> the
> >>> Beam
> >>> community might lead to better collaboration between projects.
> >>>
> >>> * Bounded Spouts (needed for Beam Runner implementation) —
> >>> Currently
> >>> spouts are unbounded, there no end to the stream. Beam has the
> >>> concept
> >>> of
> >>> bounded sources (roughly analogous to batch processing). To
> >>> support
> >>> that,
> >>> we would need to implement a similar concept in Storm. One
> >>> benefit of
> >>> such
> >>> a feature would be the ability to handle both bounded and
> >>> unbounded
> >>> workflows in Storm.
> >>>
> >>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> >>> behind
> >>> this
> >>> effort. What improvements do you envision for 2.0?
> >>>
> >>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> >>> targeting this
> >>> for 1.2.0, but it’s designed to be easily portable to
> >>> master/2.0.
> >>>
> >>> * JStorm Migration — Original outline can be found here [1].
> >>> Note
> >>> a lot
> >>> of
> >>> the associated JIRAs below are assigned, but there hasn’t been
> >>> any
> >>> recent
> >>> activity or pull requests, we should probably consider them
> >>> unassigned
> >>> and
> >>> up for grabs.:
> >>>
> >>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> >>> been the
> >>> bane of a lot of Storm users almost since day one. We have
> >>> largely
> >>> addressed it by shading/relocating dependencies. It would be
> >>> great to
> >>> see
> >>> this addressed once and for all.
> >>>
> >>> * JStorm back pressure implementation (STORM-1324) — The
> >>> current
> >>> back
> >>> pressure implementation leaves a bit to be desired, and the
> >>> JStorm
> >>> approach
> >>> looks promising, though it also depends on the JStorm concept
> >>> of
> >>> “topology
> >>> master” (STORM-1323), which may have some implications
> >>> regarding
> >>> security.
> >>>
> >>> * Dynamic Topology Updates (STORM-1335) — This would provide a
> >>> command
> >>> to
> >>> update topology jars and configuration without stopping the
> >>> topology,
> >>> and
> >>> is well suited to leverage the blobstore. The restart command
> >>> (that can
> >>> also update the topology configuration) also looks compelling
> >>> (STORM-1334).
> >>>
> >>> * Additional Scheduler Implementations (STORM-1320)
> >>>
> >>> * Additional Grouping Implementations (STORM-1328)
> >>>
> >>>
> >>> As always I’m open to any opinions and suggestions.
> >>>
> >>> -Taylor
> >>>
> >>> [1]
> >>>
> >>> https://cwiki.apache.org/confluence/pages/viewpage.
> >>> action?pageId=61328109
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
>
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Hugo Da Cruz Louro <hl...@hortonworks.com>.
Hi Alexandre,

In my benchmarks the storm-kafka-client spout improves throughput by 70% and latency by 40% vs the storm-kafka implementation. I am surprised by your findings substantiating the opposite. Can you share your benchmark where you compare the performances of both implementations?

As for you writing your own version of the spout. Why not contribute to this one instead? Do you think the implementation is that poor? If so, why do you think it is that poor? Do you expect your first version to be much better than a version that is already in production in several customers, and seems to be working fairly well?

All the bugs found so far have been addressed. Since it’s a new feature there may be a few bugs - it is expected. However, I don’t think that it is as bad as you make it sound as there are several people using it in production for extended periods of time.

Cheers

> On Jun 28, 2017, at 12:04 PM, Alexandre Vermeerbergen <av...@gmail.com> wrote:
> 
> Hello,
> 
> If that matters, our current experiences with StormKafkaClient
> isdisappointing (see my recent posts "Lag issues using Storm 1.1.1 latest
> build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this mailing
> list).
> 
> Our current experience is that the old StormKafka spout always beats the
> new one in term of performance & stability.
> 
> Therefore, I am surprised when I see talks about deprecation of the old
> StormKafka spout when the new one which just came "General Available" with
> Storm 1.1.0, is not stable, and it's not better when we try it from current
> 1.1.x builds to take into account recently closed JIRAs.
> 
> We're even considering writing our own Kafka spout with Kafka 0.10.x API to
> overcome the incompatibility of the old StormKafka spout with Kafka 0.10
> libraries.
> 
> Thus, for people which are comfortable with old Kafka spout, I'like to give
> a -1 (non binding) to the proposal of withdrawal of the old StormKafka
> spout until the new one converges.
> 
> Best regards,
> Alexandre Vermeerbergen
> 
> 
> 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:
> 
>> 
>>> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <hl...@hortonworks.com>
>> wrote:
>>> 
>>> I still need to go over the entire discussion thread in more detail, but
>> one thing I would like to bring up right way is the proposal to DEPRECATE,
>> and eventually remove, the KafkaSpout with the old Kafka Consumer APIs. The
>> storm-kafka-client KafkaSpout is getting stabilized, and I think we are all
>> in agreement that the storm-kafka KafkaSpout has presented continuous
>> maintainability problems with some fixes that got in not being backwards
>> compatible.
>> 
>> I’m fine with deprecating the old KafkaSpout, but I feel the decision to
>> actually remove it needs to take into account the user community. The main
>> sticking point here is compatibility with earlier versions of Kafka. Like
>> with JDK versions, there are many valid reasons whey users may not be in a
>> position to upgrade to a newer version of Kafka. Outright removal could
>> leave some users in the lurch.
>> 
>> Ideally, we could just poll the user community to get an idea of how much
>> of the user base depends on the old KafkaSpout and use the results to guide
>> our decision. Unfortunately, at least in my past experience, polling the
>> user@ list doesn’t elicit much of a response and the results don’t
>> provide an accurate view of the user community.
>> 
>> 
>>> 
>>> I am pretty confident how things are looking at this point for the
>> KafkaSpout. The Trident Kafka Spout is likely in between alpha and beta,
>> and that should be taken into account. I just recently submitted a PR<
>> https://github.com/apache/storm/pull/2174> with some improvements to the
>> Trident Kafka Spout (including the refactoring done to support manual
>> partition assignment), and there are some customers using it in
>> pre-production. However, it definitely would benefit from some more testing.
>>> 
>>> Thanks,
>>> Hugo
>> 
>> -Taylor
>> 
>> 
>>> 
>>> On Jun 28, 2017, at 7:48 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
>> mailto:evans@yahoo-inc.com.INVALID>> wrote:
>>> 
>>> +1.
>>> If the 1.1 and 1.2 lines start to become difficult to maintain we can
>> look at putting them in maintenance mode too once we have a 2.x release.
>>> I am a little nervous about merging a new feature into 1.x branch
>> without first going to master, but I hope that it will not be too much work
>> to port it to master, and I trust the devs on that branch to do the right
>> thing.
>>> On a related note we have not done much with feature branches before so
>> I am not sure what we want to do about merging in the new metrics API
>> branch to 1.x.  I know for me I have not had time to keep up with the
>> development work going on there.  I would at least like to have a pull
>> request put up for review before we merge it in.  This would fit with our
>> current bylaws that do not mention feature branches.  If all of the changes
>> have already followed the review process then technically I think it is OK
>> to just merge it in, but I still would like to take some time to look at
>> the changes, and especially the new APIs.
>>> 
>>> - Bobby
>>> 
>>> 
>>> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <
>> kabhwan@gmail.com<ma...@gmail.com>> wrote:
>>> 
>>> That's great news that metrics work is ready!
>>> 
>>> I'm +1 to Taylor's proposal, but in order to respect semantic
>> versioning, I
>>> propose some modifications from Taylor's proposal:
>>> 
>>> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back
>> only
>>> bug fixes to the 1.1.x-branch
>>> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
>>> 
>>> If we also agree above, I would like to volunteer the back-port work.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 6월 28일 (수) 오전 10:09, Harsha <storm@harsha.io<mailto:storm@
>> harsha.io>>님이 작성:
>>> 
>>> +1 for above stated approach on releasing 1.2.0 with metrics
>>> -Harsha
>>> 
>>> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
>>> The work on metrics is ready for a pull request to 1.x-branch from the
>>> feature branch. I’ve held off because we haven’t reached consensus on a
>>> path forward with the 1.x release lines .
>>> 
>>> I’d like to propose the following for the 1.x line:
>>> 
>>> 1. Create a branch for 1.2 so we have a branch to review the metrics
>>> stuff.
>>> 2. Release 1.1.1
>>> 3. Review/merge metrics work. Port metrics to master.
>>> 4. Release 1.2.0
>>> 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
>>> (we would only support 1.2.x and 1.1.x which are very closely aligned).
>>> 
>>> Dropping support for 1.0.x line would eliminate the need to maintain one
>>> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
>>> very closely aligned. I just up merged metrics_v2 against 1.x-branch
>>> after a while, and there were no conflicts.
>>> 
>>> That would give us a little more bandwidth to focus on 2.0 and needed bug
>>> fixes to the 1.x line like some of the issues raised with
>>> storm-kafka-client. We could even start releasing alpha/beta versions of
>>> 2.0 in parallel to the steps above.
>>> 
>>> Any thoughts on that approach?
>>> 
>>> -Taylor
>>> 
>>> 
>>> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:kabh
>> wan@gmail.com>> wrote:
>>> 
>>> Yes I prefer option 1, but it might depend on the progress of metrics
>>> V2.
>>> If it can be done within predictable near future I'm OK to pick option
>>> 2,
>>> but if not, we may be better to focus releasing 2.0.0 and make it
>>> really
>>> happen.
>>> 
>>> Whichever we go, I feel it's time to track remaining work on Storm
>>> 2.0.0. I
>>> found some bugs on master branch so filed issues, and we've remaining
>>> port
>>> work (UI and logviewer). We've some other improvements target for
>>> 2.0.0:
>>> worker redesign, beam integration, and so on, and we don't track its
>>> progress at all. I don't think we should wait for features which
>>> progress
>>> is not transparent (in other words we don't know when it will be
>>> finished).
>>> 
>>> - Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgoetz@gmail.com<mailto:ptgo
>> etz@gmail.com>>님이 작성:
>>> 
>>> Bobby/Jungtaek,
>>> 
>>> Are you saying you want to forego the 1.2 “metrics_v2” release and
>>> include
>>> it only in 2.0? (I ask because that work is already based on
>>> 1.x-branch,
>>> and forward-porting it to master is relatively simple.) I’d kind of
>>> like
>>> that work go out soon.
>>> 
>>> If we go with option 1, I would want to see a 2.0 release (even if
>>> it’s a
>>> “beta” or “preview) before putting the 1.x line into maintenance mode.
>>> 
>>> -Taylor
>>> 
>>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
>> mailto:evans@yahoo-inc.com.INVALID>
>>> 
>>> wrote:
>>> 
>>> I see 2 ways to address this.
>>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
>>> don't
>>> backport anything except bug fixes.2) We backport a lot of the
>>> backwards
>>> compatible changes from 2.x to 1.x.
>>> My personal preference is 1.  It makes it clear the direction we
>>> want to
>>> go in.  The biggest issue is that we probably would want to do a 2.x
>>> release sooner rather then later.  Even if we don't get all of the
>>> features
>>> that people want, if we just get a release out we can add in new
>>> features
>>> if they are backwards compatible, or we can create a 3.x line that
>>> would
>>> have the breaking changes in it.
>>> 
>>> - Bobby
>>> 
>>> 
>>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
>>> kabhwan@gmail.com<ma...@gmail.com>> wrote:
>>> 
>>> I'd like to bump this again instead of initiating new discussion
>>> thread.
>>> 
>>> I had having hard time to create and apply pull requests for both
>>> master
>>> and 1.x-branch and that's really painful and sometimes blocker for
>>> me to
>>> do
>>> merge step.
>>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
>>> even
>>> IDE can't switch between the branch smoothly. We didn't even address
>>> checkstyle issue yet, but after addressing, it could be "completely"
>>> diverged. JDK version is another major issue, since the pull requests
>>> targeted for master branch are not checked against JDK 7, and some of
>>> them
>>> make some issues regarding JDK version while porting back.
>>> 
>>> So personally I really would like to see the plan for 1.x version
>>> line
>>> changed - skipping any minor releases including 1.2.0 - and have epic
>>> issue
>>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
>>> proposed plan was having 2.0.0 directly after 1.0.0)
>>> 
>>> Would like to hear everyone's opinions. If we have consensus to not
>>> having
>>> any minor releases for 1.x version line, I will not port back
>>> non-bugfix
>>> pull requests to 1.x-branch, and guide contributors to create pull
>>> requests
>>> against master branch, not 1.x version line.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
>>> avermeerbergen@gmail.com<ma...@gmail.com>>님이
>>> 작성:
>>> 
>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
>>> system,
>>> we're very interested anything that can provide better throughput.
>>> 
>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <roshan@hortonworks.com<mailto:
>> roshan@hortonworks.com>>:
>>> 
>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
>>> be
>>> delivered sooner and my in-progress implementation suggests that it
>>> will
>>> yield substantial latency improvements. The 2.0 beta phase would
>>> really
>>> help kick the tires on the revised messaging system and the
>>> performance
>>> improvements will also be a good incentive for trying out the 2.0
>>> line.
>>> 
>>> I notice multiple other bottlenecks that are holding back
>>> throughput a
>>> lot, which can be addressed in a subsequent 2.x minor release.
>>> -roshan
>>> 
>>> 
>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabhwan@gmail.com<mailto:kabh
>> wan@gmail.com>> wrote:
>>> 
>>>   I also would love to see metrics V2 code sooner or later too.
>>> If we
>>> can get
>>>   it before releasing 2.0.0 that will be great, and then maybe we
>>> could
>>> just
>>>   move toward to 2.0.0, not adding any improvements to 1.x version
>>> line.
>>>   (And that's what I would want to.)
>>> 
>>>   If we would really want to have 1.2.0, I suggest that we make
>>> the
>>> 1.1.1
>>>   version correct right now rather than after releasing 1.1.1. We
>>> also
>>> merged
>>>   non-bugfix things to 1.x-branch but that's not what users
>>> expect. I
>>> agree
>>>   that work may be painful, but anyway need to do it.
>>> 
>>>   - Jungtaek Lim (HeartSaVioR)
>>> 
>>>   2017년 6월 3일 (토) 오전 3:49, Bobby Evans
>>> <ev...@yahoo-inc.com.invalid>
>>> 님이
>>> 작성:
>>> 
>>> I would love to see the metrics V2 code come out sooner rather
>>> than
>>> later.  +1.
>>> My biggest blocker for a 1.x release is
>>> https://github.com/apache/storm/pull/2142  Even though the pull
>>> request
>>> says it is minor it showed that we messed up pushing back some
>>> changes for
>>> pacemaker to open source (the code does not run at all which for
>>> me
>>> is a
>>> blocker) and I really want to get that fully fixed/tested before
>>> another
>>> release.
>>> As for 2.x I think we are very close to being able to so a 2.x
>>> alpha
>>> release.  I would like to see metrics V2 merged in simply because
>>> it
>>> is a
>>> big change for user facing APIs.  But after that I would love to
>>> see
>>> us
>>> starting to push forward on getting that out.
>>> 
>>> 
>>> - Bobby
>>> 
>>> 
>>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>>> ptgoetz@gmail.com<ma...@gmail.com>> wrote:
>>> 
>>> I’d like to bump this thread and start a discussion around our
>>> next
>>> release. Here are my thoughts.
>>> 
>>> There are a number of important fixes in 1.x-branch so I’d like
>>> to
>>> consider releasing 1.1.1 soon. I’d appreciate input on any open
>>> issues that
>>> should be resolved for that release.
>>> 
>>> I’d like us to consider releasing the metrics improvements in
>>> STORM-2153
>>> [1] as version 1.2.0. That work is in the metrics_v2 feature
>>> branch
>>> right
>>> now and would need to be reviewed and merged. That work is
>>> against
>>> the
>>> 1.x-branch right now. I would recommend porting it to master
>>> *after*
>>> the
>>> review/merge since there will likely be changes as a result of
>>> the
>>> review.
>>> 
>>> Maybe related to or not, but would we want to create a new
>>> branch
>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>> 
>>> 
>>> 
>>> If wee agree to the above, I would say yes. After the 1.1.1
>>> release,
>>> we
>>> could create a 1.1.x-branch that would be the maintenance/release
>>> branch
>>> for that version line. 1.x-branch would then become the target
>>> for
>>> the
>>> 1.2.0 release.
>>> 
>>> There are a few fixes in the 0.10.x branch that probably warrant
>>> a
>>> release. After that we may want to back away from that version
>>> line
>>> a bit
>>> so we can focus more on newer versions.
>>> 
>>> In the past, we’ve shied away form doing “beta” releases, but I’m
>>> wondering if we might want to revisit that for the 2.0 release —
>>> the
>>> idea
>>> being that it would give early adopter users a chance to kick the
>>> tires on
>>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
>>> help
>>> make the
>>> final release more solid. I’m on the fence here and could go
>>> either
>>> way.
>>> 
>>> I’d appreciate any input others may have.
>>> 
>>> 
>>> Thanks,
>>> 
>>> -Taylor
>>> 
>>> 
>>> [1] https://issues.apache.org/jira/browse/STORM-2153
>>> 
>>> 
>>> 
>>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabhwan@gmail.com<mailto:kabh
>> wan@gmail.com>>
>>> wrote:
>>> 
>>> Maybe related to or not, but would we want to create a new
>>> branch
>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>> 
>>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
>>> so
>>> hence
>>> the
>>> question.
>>> 
>>> - Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
>>> hlouro@hortonworks.com<ma...@hortonworks.com>>님이
>>> 작성:
>>> 
>>> +1 for finishing the porting to Java ahead of anything else -
>>> it
>>> will
>>> be a
>>> significant milestone. I have a JIRA assigned concerning to
>>> the
>>> porting. I
>>> will work on it for the 2.0 release.
>>> 
>>> it’s a priority to guarantee no performance regressions. As
>>> part
>>> of this
>>> endeavor, explore an automated (or easy) way to run and assert
>>> major
>>> performance benchmarks. Ideally any contributor should be able
>>> to
>>> fairly
>>> easily test the impact of changes under certain performance
>>> test
>>> scenarios.
>>> 
>>> Beam Runner work should take into account the impact of
>>> incorporating
>>> new
>>> JStorm features and Storm Worker Redesign<
>>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
>>> efficient
>>> to
>>> start doing it, to  find out that it will have to chance in
>>> face
>>> of
>>> Storm
>>> and worker redesign. That is, it should be done after it’s
>>> building
>>> blocks
>>> are stable.
>>> 
>>> Thanks,
>>> Hugo
>>> 
>>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
>>> arunm@apache.org<ma...@apache.org>
>>> <mailto:
>>> arunm@apache.org>> wrote:
>>> 
>>> +1 to release with the porting completed. I think its mainly
>>> the
>>> UI
>>> server
>>> and log viewer that’s pending.
>>> 
>>> We can start doing the regression and performance tests for
>>> whatever is
>>> already ported.
>>> 
>>> If anyone is running the master branch in their pre-prod /
>>> prod
>>> environments, it will be good to know and give us more
>>> confidence.
>>> 
>>> The other features can be added in follow up releases.
>>> 
>>> Regards,
>>> Arun
>>> 
>>> 
>>> On 3/24/17, 11:47 AM, "Satish Duggana" <
>>> satish.duggana@gmail.com
>>> <mailto:
>>> satish.duggana@gmail.com>> wrote:
>>> 
>>> +1 to have 2.0 with porting and performance(it should be at
>>> least
>>> as
>>> good
>>> as 1.x release) issues addressed
>>> 
>>> We can target other tasks(mentioned by Taylor and Jungtaek)
>>> for
>>> 2.x-branch.
>>> 
>>> 
>>> Exactly-once support:
>>> While thinking through the exactlyonce support design, it is
>>> realized
>>> better to avoid acking tuples and implement exactly once by
>>> snapshotting
>>> barriers. It seems JStorm folks followed similar design, they
>>> claim it
>>> gives better performance. This feature is essential for beam
>>> runner and
>>> we
>>> can decide on respective approaches though.
>>> 
>>> Beam Runner
>>> Lets hold on this for now and keep it in Storm till 2.x. We
>>> should avoid
>>> having a minimal beam runner in haste. It is better to address
>>> STORM-2284,
>>> exactly-once and other windowing enhancements to enable beam
>>> runner.
>>> 
>>> JStorm
>>> Agree with Jungtaek on looking at the latest JStorm and
>>> align/scope with
>>> the features for 2.x.
>>> 
>>> STORM-2284
>>> We may want to look at JStorm worker before working on
>>> respective
>>> components in this epic to pull appropriate enhancements.
>>> 
>>> YARN/MESOS
>>> Supporting Storm on YARN/Mesos for 2.x.
>>> 
>>> Thanks,
>>> Satish.
>>> 
>>> 
>>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
>>> kabhwan@gmail.com
>>> <mailto:
>>> kabhwan@gmail.com>> wrote:
>>> 
>>> First of all, +1 to complete only port work and do sanity
>>> check
>>> (including
>>> performance regression), and release.
>>> 
>>> If we can get STORM-2284 within deterministic time frame (say
>>> 2~3
>>> months)
>>> that should be great, but if not I'd in favor of postponing
>>> that
>>> to
>>> later
>>> 2.x release.
>>> 
>>> JStorm released their new versions after code donation. So
>>> there're more
>>> things we could get ideas from, or even adopt from.
>>> https://github.com/alibaba/jstorm/blob/master/history.md
>>> As you noticed from release note link, we also need to update
>>> phase 2
>>> since
>>> they already changed what we're planning to do in phase 2. For
>>> example,
>>> they changed backpressure to end-to-end, and changed to use
>>> snapshot
>>> rather
>>> than acker.
>>> May be sure, JStorm pulled many features from today's Storm,
>>> like
>>> Flux,
>>> Windowing, more shuffle groupings, log search, log level
>>> change,
>>> and so
>>> on.
>>> 
>>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
>>> is
>>> due to
>>> the
>>> limitation of Spout lifecycle (all the things are done in
>>> single
>>> thread),
>>> and STORM-1358 <
>>> https://issues.apache.org/jira/browse/STORM-1358
>>> (JStorm's
>>> multi-thread Spout) can remedy this (despite that Spout
>>> implementation
>>> may
>>> need to guarantee thread-safety later). It's not a just
>>> improvement but
>>> close to design concern so would like to address sooner than
>>> other
>>> things
>>> in phase 2.
>>> 
>>> For Storm SQL side, I've lost progress but major work would be
>>> adopting
>>> group by with windowing. It was not available from Calcite but
>>> will be
>>> available at next release (1.12.0).
>>> I've filed this to STORM-2405
>>> <https://issues.apache.org/jira/browse/STORM-2405>, but
>>> windowing &
>>> micro
>>> batch is not intuitive, so I would like to change the
>>> underlying
>>> API to
>>> stream API in SQL. Also filed this to STORM-2406
>>> <https://issues.apache.org/jira/browse/STORM-2406>.
>>> 
>>> Just 2 cents btw, hopefully I would like to see metrics V2
>>> sooner
>>> since
>>> we
>>> lost metrics even when doing normal operation like restarting
>>> worker,
>>> rebalancing, and so on. Eventually we need to fight with
>>> dynamic
>>> scaling,
>>> and then metrics will be broken often.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
>>> storm@harsha.io
>>> 님이
>>> 작성:
>>> 
>>> Storm 2.0 migration to java in itself is a big win and would
>>> attract
>>> wider
>>> community and adoption. So my vote would be to resolve the
>>> first
>>> 3 items
>>> to
>>> get a release out.
>>> All the other featured mentioned are great to have but
>>> shouldn't
>>> be
>>> blockers for 2.0 release.
>>> 
>>> -Harsha
>>> 
>>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
>>> ptgoetz@gmail.com>
>>> wrote:
>>> 
>>> With the 1.1.0 release nearing completion, I’d like to turn
>>> our
>>> attention
>>> to 2.0 and develop a plan for what features, etc. to include.
>>> 
>>> The following 3 are what I feel are the minimum for a 2.0
>>> release.
>>> These
>>> could likely be resolved relatively quickly:
>>> 
>>> * Performance — I’ve not benchmarked the master branch vs.
>>> 1.0.x
>>> or
>>> 1.1.x
>>> in a while, but I feel it will be important to make sure there
>>> are no
>>> performance regressions, and would hope that we actually have
>>> a
>>> performance
>>> improvement over previous versions. To that end (e.g. if there
>>> is
>>> in
>>> fact a
>>> performance regression), the proposals that Roshan Naik put
>>> together
>>> for
>>> revising the threading and execution model (STORM-2307) and
>>> replacing
>>> Disruptor with JCTools (STORM-2306) warrant review and
>>> consideration.
>>> See
>>> also STORM-2284 which is the parent JIRA.
>>> 
>>> * Finish porting Storm UI to java (STORM-1311)
>>> 
>>> * Finish porting log viewer to java (STORM-1280)
>>> 
>>> The following are items that are nice to have in 2.0, but I
>>> don’t
>>> feel
>>> are
>>> absolutely necessary for an initial 2.0 release:
>>> 
>>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
>>> because
>>> it’s
>>> relevant) — Initially there seemed to be a lot of interest in
>>> this, but
>>> that seems to have trailed off. I spoke with some Beam
>>> developers
>>> and
>>> there
>>> seems to be interest from that community as well. Do we want
>>> to
>>> move
>>> that
>>> effort to the Beam community, or keep it here? Moving it to
>>> the
>>> Beam
>>> community might lead to better collaboration between projects.
>>> 
>>> * Bounded Spouts (needed for Beam Runner implementation) —
>>> Currently
>>> spouts are unbounded, there no end to the stream. Beam has the
>>> concept
>>> of
>>> bounded sources (roughly analogous to batch processing). To
>>> support
>>> that,
>>> we would need to implement a similar concept in Storm. One
>>> benefit of
>>> such
>>> a feature would be the ability to handle both bounded and
>>> unbounded
>>> workflows in Storm.
>>> 
>>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
>>> behind
>>> this
>>> effort. What improvements do you envision for 2.0?
>>> 
>>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
>>> targeting this
>>> for 1.2.0, but it’s designed to be easily portable to
>>> master/2.0.
>>> 
>>> * JStorm Migration — Original outline can be found here [1].
>>> Note
>>> a lot
>>> of
>>> the associated JIRAs below are assigned, but there hasn’t been
>>> any
>>> recent
>>> activity or pull requests, we should probably consider them
>>> unassigned
>>> and
>>> up for grabs.:
>>> 
>>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
>>> been the
>>> bane of a lot of Storm users almost since day one. We have
>>> largely
>>> addressed it by shading/relocating dependencies. It would be
>>> great to
>>> see
>>> this addressed once and for all.
>>> 
>>> * JStorm back pressure implementation (STORM-1324) — The
>>> current
>>> back
>>> pressure implementation leaves a bit to be desired, and the
>>> JStorm
>>> approach
>>> looks promising, though it also depends on the JStorm concept
>>> of
>>> “topology
>>> master” (STORM-1323), which may have some implications
>>> regarding
>>> security.
>>> 
>>> * Dynamic Topology Updates (STORM-1335) — This would provide a
>>> command
>>> to
>>> update topology jars and configuration without stopping the
>>> topology,
>>> and
>>> is well suited to leverage the blobstore. The restart command
>>> (that can
>>> also update the topology configuration) also looks compelling
>>> (STORM-1334).
>>> 
>>> * Additional Scheduler Implementations (STORM-1320)
>>> 
>>> * Additional Grouping Implementations (STORM-1328)
>>> 
>>> 
>>> As always I’m open to any opinions and suggestions.
>>> 
>>> -Taylor
>>> 
>>> [1]
>>> 
>>> https://cwiki.apache.org/confluence/pages/viewpage.
>>> action?pageId=61328109
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
> On Jun 28, 2017, at 4:01 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> If my memory is right, when releasing 1.1.0 we postponed resolving some
> critical issues for storm-kafka-client, and seems like it still haven't
> been sorted out. I even think these issues can be effectively blocker for
> 1.1.1 if we want to treat this module as stable.
> 
> So in fact and ideally, it should've been marked as experimental or beta
> until it has no more critical issue, and I feel it's still in beta phase.


+1

IIRC, I think we had one issue opened during the voting phase when we released 1.1.0, and it has since snowballed somewhat.

For a 1.1.1 release I think we need to make sure it is solid, and if not, retroactively give it a beta/experimental/etc. label in the release announcement.

-Taylor

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Jungtaek Lim <ka...@gmail.com>.
-1 to deprecate the old storm-kafka.

I don't feel storm-kafka-client is stable given that it has some critical
issues, and the module doesn't have enough volunteer committers to be
stabilized faster as it should be.
If my memory is right, when releasing 1.1.0 we postponed resolving some
critical issues for storm-kafka-client, and seems like it still haven't
been sorted out. I even think these issues can be effectively blocker for
1.1.1 if we want to treat this module as stable.

So in fact and ideally, it should've been marked as experimental or beta
until it has no more critical issue, and I feel it's still in beta phase.

- Jungtaek Lim (HeartSaVioR)
On Thu, 29 Jun 2017 at 4:04 AM Alexandre Vermeerbergen <
avermeerbergen@gmail.com> wrote:

> Hello,
>
> If that matters, our current experiences with StormKafkaClient
> isdisappointing (see my recent posts "Lag issues using Storm 1.1.1 latest
> build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this mailing
> list).
>
> Our current experience is that the old StormKafka spout always beats the
> new one in term of performance & stability.
>
> Therefore, I am surprised when I see talks about deprecation of the old
> StormKafka spout when the new one which just came "General Available" with
> Storm 1.1.0, is not stable, and it's not better when we try it from current
> 1.1.x builds to take into account recently closed JIRAs.
>
> We're even considering writing our own Kafka spout with Kafka 0.10.x API to
> overcome the incompatibility of the old StormKafka spout with Kafka 0.10
> libraries.
>
> Thus, for people which are comfortable with old Kafka spout, I'like to give
> a -1 (non binding) to the proposal of withdrawal of the old StormKafka
> spout until the new one converges.
>
> Best regards,
> Alexandre Vermeerbergen
>
>
> 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:
>
> >
> > > On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <
> hlouro@hortonworks.com>
> > wrote:
> > >
> > > I still need to go over the entire discussion thread in more detail,
> but
> > one thing I would like to bring up right way is the proposal to
> DEPRECATE,
> > and eventually remove, the KafkaSpout with the old Kafka Consumer APIs.
> The
> > storm-kafka-client KafkaSpout is getting stabilized, and I think we are
> all
> > in agreement that the storm-kafka KafkaSpout has presented continuous
> > maintainability problems with some fixes that got in not being backwards
> > compatible.
> >
> > I’m fine with deprecating the old KafkaSpout, but I feel the decision to
> > actually remove it needs to take into account the user community. The
> main
> > sticking point here is compatibility with earlier versions of Kafka. Like
> > with JDK versions, there are many valid reasons whey users may not be in
> a
> > position to upgrade to a newer version of Kafka. Outright removal could
> > leave some users in the lurch.
> >
> > Ideally, we could just poll the user community to get an idea of how much
> > of the user base depends on the old KafkaSpout and use the results to
> guide
> > our decision. Unfortunately, at least in my past experience, polling the
> > user@ list doesn’t elicit much of a response and the results don’t
> > provide an accurate view of the user community.
> >
> >
> > >
> > > I am pretty confident how things are looking at this point for the
> > KafkaSpout. The Trident Kafka Spout is likely in between alpha and beta,
> > and that should be taken into account. I just recently submitted a PR<
> > https://github.com/apache/storm/pull/2174> with some improvements to the
> > Trident Kafka Spout (including the refactoring done to support manual
> > partition assignment), and there are some customers using it in
> > pre-production. However, it definitely would benefit from some more
> testing.
> > >
> > > Thanks,
> > > Hugo
> >
> > -Taylor
> >
> >
> > >
> > > On Jun 28, 2017, at 7:48 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
> > mailto:evans@yahoo-inc.com.INVALID>> wrote:
> > >
> > > +1.
> > > If the 1.1 and 1.2 lines start to become difficult to maintain we can
> > look at putting them in maintenance mode too once we have a 2.x release.
> > > I am a little nervous about merging a new feature into 1.x branch
> > without first going to master, but I hope that it will not be too much
> work
> > to port it to master, and I trust the devs on that branch to do the right
> > thing.
> > > On a related note we have not done much with feature branches before so
> > I am not sure what we want to do about merging in the new metrics API
> > branch to 1.x.  I know for me I have not had time to keep up with the
> > development work going on there.  I would at least like to have a pull
> > request put up for review before we merge it in.  This would fit with our
> > current bylaws that do not mention feature branches.  If all of the
> changes
> > have already followed the review process then technically I think it is
> OK
> > to just merge it in, but I still would like to take some time to look at
> > the changes, and especially the new APIs.
> > >
> > > - Bobby
> > >
> > >
> > > On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <
> > kabhwan@gmail.com<ma...@gmail.com>> wrote:
> > >
> > > That's great news that metrics work is ready!
> > >
> > > I'm +1 to Taylor's proposal, but in order to respect semantic
> > versioning, I
> > > propose some modifications from Taylor's proposal:
> > >
> > > - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back
> > only
> > > bug fixes to the 1.1.x-branch
> > > - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
> > >
> > > If we also agree above, I would like to volunteer the back-port work.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 6월 28일 (수) 오전 10:09, Harsha <storm@harsha.io<mailto:storm@
> > harsha.io>>님이 작성:
> > >
> > > +1 for above stated approach on releasing 1.2.0 with metrics
> > > -Harsha
> > >
> > > On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> > > The work on metrics is ready for a pull request to 1.x-branch from the
> > > feature branch. I’ve held off because we haven’t reached consensus on a
> > > path forward with the 1.x release lines .
> > >
> > > I’d like to propose the following for the 1.x line:
> > >
> > > 1. Create a branch for 1.2 so we have a branch to review the metrics
> > > stuff.
> > > 2. Release 1.1.1
> > > 3. Review/merge metrics work. Port metrics to master.
> > > 4. Release 1.2.0
> > > 5. Put the entire 1.x line into maintenance mode. Drop support for
> 1.0.x.
> > > (we would only support 1.2.x and 1.1.x which are very closely aligned).
> > >
> > > Dropping support for 1.0.x line would eliminate the need to maintain
> one
> > > of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> > > very closely aligned. I just up merged metrics_v2 against 1.x-branch
> > > after a while, and there were no conflicts.
> > >
> > > That would give us a little more bandwidth to focus on 2.0 and needed
> bug
> > > fixes to the 1.x line like some of the issues raised with
> > > storm-kafka-client. We could even start releasing alpha/beta versions
> of
> > > 2.0 in parallel to the steps above.
> > >
> > > Any thoughts on that approach?
> > >
> > > -Taylor
> > >
> > >
> > > On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:
> kabh
> > wan@gmail.com>> wrote:
> > >
> > > Yes I prefer option 1, but it might depend on the progress of metrics
> > > V2.
> > > If it can be done within predictable near future I'm OK to pick option
> > > 2,
> > > but if not, we may be better to focus releasing 2.0.0 and make it
> > > really
> > > happen.
> > >
> > > Whichever we go, I feel it's time to track remaining work on Storm
> > > 2.0.0. I
> > > found some bugs on master branch so filed issues, and we've remaining
> > > port
> > > work (UI and logviewer). We've some other improvements target for
> > > 2.0.0:
> > > worker redesign, beam integration, and so on, and we don't track its
> > > progress at all. I don't think we should wait for features which
> > > progress
> > > is not transparent (in other words we don't know when it will be
> > > finished).
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
> ptgo
> > etz@gmail.com>>님이 작성:
> > >
> > > Bobby/Jungtaek,
> > >
> > > Are you saying you want to forego the 1.2 “metrics_v2” release and
> > > include
> > > it only in 2.0? (I ask because that work is already based on
> > > 1.x-branch,
> > > and forward-porting it to master is relatively simple.) I’d kind of
> > > like
> > > that work go out soon.
> > >
> > > If we go with option 1, I would want to see a 2.0 release (even if
> > > it’s a
> > > “beta” or “preview) before putting the 1.x line into maintenance mode.
> > >
> > > -Taylor
> > >
> > > On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
> > mailto:evans@yahoo-inc.com.INVALID>
> > >
> > > wrote:
> > >
> > > I see 2 ways to address this.
> > > 1) We put the 1.x line into maintenance mode like with 0.10.  We
> > > don't
> > > backport anything except bug fixes.2) We backport a lot of the
> > > backwards
> > > compatible changes from 2.x to 1.x.
> > > My personal preference is 1.  It makes it clear the direction we
> > > want to
> > > go in.  The biggest issue is that we probably would want to do a 2.x
> > > release sooner rather then later.  Even if we don't get all of the
> > > features
> > > that people want, if we just get a release out we can add in new
> > > features
> > > if they are backwards compatible, or we can create a 3.x line that
> > > would
> > > have the breaking changes in it.
> > >
> > > - Bobby
> > >
> > >
> > > On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> > > kabhwan@gmail.com<ma...@gmail.com>> wrote:
> > >
> > > I'd like to bump this again instead of initiating new discussion
> > > thread.
> > >
> > > I had having hard time to create and apply pull requests for both
> > > master
> > > and 1.x-branch and that's really painful and sometimes blocker for
> > > me to
> > > do
> > > merge step.
> > > Two branches are heavily diverged more than between 0.10 and 1.0.0,
> > > even
> > > IDE can't switch between the branch smoothly. We didn't even address
> > > checkstyle issue yet, but after addressing, it could be "completely"
> > > diverged. JDK version is another major issue, since the pull requests
> > > targeted for master branch are not checked against JDK 7, and some of
> > > them
> > > make some issues regarding JDK version while porting back.
> > >
> > > So personally I really would like to see the plan for 1.x version
> > > line
> > > changed - skipping any minor releases including 1.2.0 - and have epic
> > > issue
> > > for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> > > proposed plan was having 2.0.0 directly after 1.0.0)
> > >
> > > Would like to hear everyone's opinions. If we have consensus to not
> > > having
> > > any minor releases for 1.x version line, I will not port back
> > > non-bugfix
> > > pull requests to 1.x-branch, and guide contributors to create pull
> > > requests
> > > against master branch, not 1.x version line.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> > > avermeerbergen@gmail.com<ma...@gmail.com>>님이
> > > 작성:
> > >
> > > +1 for Roshan's suggestion : in our Storm 1.x based supervision
> > > system,
> > > we're very interested anything that can provide better throughput.
> > >
> > > 2017-06-03 18:12 GMT+02:00 Roshan Naik <roshan@hortonworks.com<mailto:
> > roshan@hortonworks.com>>:
> > >
> > > For 2.0 beta … it would be good to incorporate some of the Worker
> > > improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> > > be
> > > delivered sooner and my in-progress implementation suggests that it
> > > will
> > > yield substantial latency improvements. The 2.0 beta phase would
> > > really
> > > help kick the tires on the revised messaging system and the
> > > performance
> > > improvements will also be a good incentive for trying out the 2.0
> > > line.
> > >
> > > I notice multiple other bottlenecks that are holding back
> > > throughput a
> > > lot, which can be addressed in a subsequent 2.x minor release.
> > > -roshan
> > >
> > >
> > > On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabhwan@gmail.com<mailto:kabh
> > wan@gmail.com>> wrote:
> > >
> > >    I also would love to see metrics V2 code sooner or later too.
> > > If we
> > > can get
> > >    it before releasing 2.0.0 that will be great, and then maybe we
> > > could
> > > just
> > >    move toward to 2.0.0, not adding any improvements to 1.x version
> > > line.
> > >    (And that's what I would want to.)
> > >
> > >    If we would really want to have 1.2.0, I suggest that we make
> > > the
> > > 1.1.1
> > >    version correct right now rather than after releasing 1.1.1. We
> > > also
> > > merged
> > >    non-bugfix things to 1.x-branch but that's not what users
> > > expect. I
> > > agree
> > >    that work may be painful, but anyway need to do it.
> > >
> > >    - Jungtaek Lim (HeartSaVioR)
> > >
> > >    2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> > > <ev...@yahoo-inc.com.invalid>
> > > 님이
> > > 작성:
> > >
> > > I would love to see the metrics V2 code come out sooner rather
> > > than
> > > later.  +1.
> > > My biggest blocker for a 1.x release is
> > > https://github.com/apache/storm/pull/2142  Even though the pull
> > > request
> > > says it is minor it showed that we messed up pushing back some
> > > changes for
> > > pacemaker to open source (the code does not run at all which for
> > > me
> > > is a
> > > blocker) and I really want to get that fully fixed/tested before
> > > another
> > > release.
> > > As for 2.x I think we are very close to being able to so a 2.x
> > > alpha
> > > release.  I would like to see metrics V2 merged in simply because
> > > it
> > > is a
> > > big change for user facing APIs.  But after that I would love to
> > > see
> > > us
> > > starting to push forward on getting that out.
> > >
> > >
> > > - Bobby
> > >
> > >
> > > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> > > ptgoetz@gmail.com<ma...@gmail.com>> wrote:
> > >
> > > I’d like to bump this thread and start a discussion around our
> > > next
> > > release. Here are my thoughts.
> > >
> > > There are a number of important fixes in 1.x-branch so I’d like
> > > to
> > > consider releasing 1.1.1 soon. I’d appreciate input on any open
> > > issues that
> > > should be resolved for that release.
> > >
> > > I’d like us to consider releasing the metrics improvements in
> > > STORM-2153
> > > [1] as version 1.2.0. That work is in the metrics_v2 feature
> > > branch
> > > right
> > > now and would need to be reviewed and merged. That work is
> > > against
> > > the
> > > 1.x-branch right now. I would recommend porting it to master
> > > *after*
> > > the
> > > review/merge since there will likely be changes as a result of
> > > the
> > > review.
> > >
> > > Maybe related to or not, but would we want to create a new
> > > branch
> > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >
> > >
> > >
> > > If wee agree to the above, I would say yes. After the 1.1.1
> > > release,
> > > we
> > > could create a 1.1.x-branch that would be the maintenance/release
> > > branch
> > > for that version line. 1.x-branch would then become the target
> > > for
> > > the
> > > 1.2.0 release.
> > >
> > > There are a few fixes in the 0.10.x branch that probably warrant
> > > a
> > > release. After that we may want to back away from that version
> > > line
> > > a bit
> > > so we can focus more on newer versions.
> > >
> > > In the past, we’ve shied away form doing “beta” releases, but I’m
> > > wondering if we might want to revisit that for the 2.0 release —
> > > the
> > > idea
> > > being that it would give early adopter users a chance to kick the
> > > tires on
> > > what’s coming in 2.0 and provide feedback, find bugs, etc. to
> > > help
> > > make the
> > > final release more solid. I’m on the fence here and could go
> > > either
> > > way.
> > >
> > > I’d appreciate any input others may have.
> > >
> > >
> > > Thanks,
> > >
> > > -Taylor
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/STORM-2153
> > >
> > >
> > >
> > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabhwan@gmail.com<mailto:
> kabh
> > wan@gmail.com>>
> > > wrote:
> > >
> > > Maybe related to or not, but would we want to create a new
> > > branch
> > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >
> > > I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> > > so
> > > hence
> > > the
> > > question.
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > > hlouro@hortonworks.com<ma...@hortonworks.com>>님이
> > > 작성:
> > >
> > > +1 for finishing the porting to Java ahead of anything else -
> > > it
> > > will
> > > be a
> > > significant milestone. I have a JIRA assigned concerning to
> > > the
> > > porting. I
> > > will work on it for the 2.0 release.
> > >
> > > it’s a priority to guarantee no performance regressions. As
> > > part
> > > of this
> > > endeavor, explore an automated (or easy) way to run and assert
> > > major
> > > performance benchmarks. Ideally any contributor should be able
> > > to
> > > fairly
> > > easily test the impact of changes under certain performance
> > > test
> > > scenarios.
> > >
> > > Beam Runner work should take into account the impact of
> > > incorporating
> > > new
> > > JStorm features and Storm Worker Redesign<
> > > https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > > efficient
> > > to
> > > start doing it, to  find out that it will have to chance in
> > > face
> > > of
> > > Storm
> > > and worker redesign. That is, it should be done after it’s
> > > building
> > > blocks
> > > are stable.
> > >
> > > Thanks,
> > > Hugo
> > >
> > > On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> > > arunm@apache.org<ma...@apache.org>
> > > <mailto:
> > > arunm@apache.org>> wrote:
> > >
> > > +1 to release with the porting completed. I think its mainly
> > > the
> > > UI
> > > server
> > > and log viewer that’s pending.
> > >
> > > We can start doing the regression and performance tests for
> > > whatever is
> > > already ported.
> > >
> > > If anyone is running the master branch in their pre-prod /
> > > prod
> > > environments, it will be good to know and give us more
> > > confidence.
> > >
> > > The other features can be added in follow up releases.
> > >
> > > Regards,
> > > Arun
> > >
> > >
> > > On 3/24/17, 11:47 AM, "Satish Duggana" <
> > > satish.duggana@gmail.com
> > > <mailto:
> > > satish.duggana@gmail.com>> wrote:
> > >
> > > +1 to have 2.0 with porting and performance(it should be at
> > > least
> > > as
> > > good
> > > as 1.x release) issues addressed
> > >
> > > We can target other tasks(mentioned by Taylor and Jungtaek)
> > > for
> > > 2.x-branch.
> > >
> > >
> > > Exactly-once support:
> > > While thinking through the exactlyonce support design, it is
> > > realized
> > > better to avoid acking tuples and implement exactly once by
> > > snapshotting
> > > barriers. It seems JStorm folks followed similar design, they
> > > claim it
> > > gives better performance. This feature is essential for beam
> > > runner and
> > > we
> > > can decide on respective approaches though.
> > >
> > > Beam Runner
> > > Lets hold on this for now and keep it in Storm till 2.x. We
> > > should avoid
> > > having a minimal beam runner in haste. It is better to address
> > > STORM-2284,
> > > exactly-once and other windowing enhancements to enable beam
> > > runner.
> > >
> > > JStorm
> > > Agree with Jungtaek on looking at the latest JStorm and
> > > align/scope with
> > > the features for 2.x.
> > >
> > > STORM-2284
> > > We may want to look at JStorm worker before working on
> > > respective
> > > components in this epic to pull appropriate enhancements.
> > >
> > > YARN/MESOS
> > > Supporting Storm on YARN/Mesos for 2.x.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> > > kabhwan@gmail.com
> > > <mailto:
> > > kabhwan@gmail.com>> wrote:
> > >
> > > First of all, +1 to complete only port work and do sanity
> > > check
> > > (including
> > > performance regression), and release.
> > >
> > > If we can get STORM-2284 within deterministic time frame (say
> > > 2~3
> > > months)
> > > that should be great, but if not I'd in favor of postponing
> > > that
> > > to
> > > later
> > > 2.x release.
> > >
> > > JStorm released their new versions after code donation. So
> > > there're more
> > > things we could get ideas from, or even adopt from.
> > > https://github.com/alibaba/jstorm/blob/master/history.md
> > > As you noticed from release note link, we also need to update
> > > phase 2
> > > since
> > > they already changed what we're planning to do in phase 2. For
> > > example,
> > > they changed backpressure to end-to-end, and changed to use
> > > snapshot
> > > rather
> > > than acker.
> > > May be sure, JStorm pulled many features from today's Storm,
> > > like
> > > Flux,
> > > Windowing, more shuffle groupings, log search, log level
> > > change,
> > > and so
> > > on.
> > >
> > > STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> > > is
> > > due to
> > > the
> > > limitation of Spout lifecycle (all the things are done in
> > > single
> > > thread),
> > > and STORM-1358 <
> > > https://issues.apache.org/jira/browse/STORM-1358
> > > (JStorm's
> > > multi-thread Spout) can remedy this (despite that Spout
> > > implementation
> > > may
> > > need to guarantee thread-safety later). It's not a just
> > > improvement but
> > > close to design concern so would like to address sooner than
> > > other
> > > things
> > > in phase 2.
> > >
> > > For Storm SQL side, I've lost progress but major work would be
> > > adopting
> > > group by with windowing. It was not available from Calcite but
> > > will be
> > > available at next release (1.12.0).
> > > I've filed this to STORM-2405
> > > <https://issues.apache.org/jira/browse/STORM-2405>, but
> > > windowing &
> > > micro
> > > batch is not intuitive, so I would like to change the
> > > underlying
> > > API to
> > > stream API in SQL. Also filed this to STORM-2406
> > > <https://issues.apache.org/jira/browse/STORM-2406>.
> > >
> > > Just 2 cents btw, hopefully I would like to see metrics V2
> > > sooner
> > > since
> > > we
> > > lost metrics even when doing normal operation like restarting
> > > worker,
> > > rebalancing, and so on. Eventually we need to fight with
> > > dynamic
> > > scaling,
> > > and then metrics will be broken often.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> > > storm@harsha.io
> > > 님이
> > > 작성:
> > >
> > > Storm 2.0 migration to java in itself is a big win and would
> > > attract
> > > wider
> > > community and adoption. So my vote would be to resolve the
> > > first
> > > 3 items
> > > to
> > > get a release out.
> > > All the other featured mentioned are great to have but
> > > shouldn't
> > > be
> > > blockers for 2.0 release.
> > >
> > > -Harsha
> > >
> > > On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > > ptgoetz@gmail.com>
> > > wrote:
> > >
> > > With the 1.1.0 release nearing completion, I’d like to turn
> > > our
> > > attention
> > > to 2.0 and develop a plan for what features, etc. to include.
> > >
> > > The following 3 are what I feel are the minimum for a 2.0
> > > release.
> > > These
> > > could likely be resolved relatively quickly:
> > >
> > > * Performance — I’ve not benchmarked the master branch vs.
> > > 1.0.x
> > > or
> > > 1.1.x
> > > in a while, but I feel it will be important to make sure there
> > > are no
> > > performance regressions, and would hope that we actually have
> > > a
> > > performance
> > > improvement over previous versions. To that end (e.g. if there
> > > is
> > > in
> > > fact a
> > > performance regression), the proposals that Roshan Naik put
> > > together
> > > for
> > > revising the threading and execution model (STORM-2307) and
> > > replacing
> > > Disruptor with JCTools (STORM-2306) warrant review and
> > > consideration.
> > > See
> > > also STORM-2284 which is the parent JIRA.
> > >
> > > * Finish porting Storm UI to java (STORM-1311)
> > >
> > > * Finish porting log viewer to java (STORM-1280)
> > >
> > > The following are items that are nice to have in 2.0, but I
> > > don’t
> > > feel
> > > are
> > > absolutely necessary for an initial 2.0 release:
> > >
> > > * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> > > because
> > > it’s
> > > relevant) — Initially there seemed to be a lot of interest in
> > > this, but
> > > that seems to have trailed off. I spoke with some Beam
> > > developers
> > > and
> > > there
> > > seems to be interest from that community as well. Do we want
> > > to
> > > move
> > > that
> > > effort to the Beam community, or keep it here? Moving it to
> > > the
> > > Beam
> > > community might lead to better collaboration between projects.
> > >
> > > * Bounded Spouts (needed for Beam Runner implementation) —
> > > Currently
> > > spouts are unbounded, there no end to the stream. Beam has the
> > > concept
> > > of
> > > bounded sources (roughly analogous to batch processing). To
> > > support
> > > that,
> > > we would need to implement a similar concept in Storm. One
> > > benefit of
> > > such
> > > a feature would be the ability to handle both bounded and
> > > unbounded
> > > workflows in Storm.
> > >
> > > * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > > behind
> > > this
> > > effort. What improvements do you envision for 2.0?
> > >
> > > * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > > targeting this
> > > for 1.2.0, but it’s designed to be easily portable to
> > > master/2.0.
> > >
> > > * JStorm Migration — Original outline can be found here [1].
> > > Note
> > > a lot
> > > of
> > > the associated JIRAs below are assigned, but there hasn’t been
> > > any
> > > recent
> > > activity or pull requests, we should probably consider them
> > > unassigned
> > > and
> > > up for grabs.:
> > >
> > > * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > > been the
> > > bane of a lot of Storm users almost since day one. We have
> > > largely
> > > addressed it by shading/relocating dependencies. It would be
> > > great to
> > > see
> > > this addressed once and for all.
> > >
> > > * JStorm back pressure implementation (STORM-1324) — The
> > > current
> > > back
> > > pressure implementation leaves a bit to be desired, and the
> > > JStorm
> > > approach
> > > looks promising, though it also depends on the JStorm concept
> > > of
> > > “topology
> > > master” (STORM-1323), which may have some implications
> > > regarding
> > > security.
> > >
> > > * Dynamic Topology Updates (STORM-1335) — This would provide a
> > > command
> > > to
> > > update topology jars and configuration without stopping the
> > > topology,
> > > and
> > > is well suited to leverage the blobstore. The restart command
> > > (that can
> > > also update the topology configuration) also looks compelling
> > > (STORM-1334).
> > >
> > > * Additional Scheduler Implementations (STORM-1320)
> > >
> > > * Additional Grouping Implementations (STORM-1328)
> > >
> > >
> > > As always I’m open to any opinions and suggestions.
> > >
> > > -Taylor
> > >
> > > [1]
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=61328109
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Alexandre,

Thanks for your input.

I think we’re very much on the same page. The new Kafka spout needs to on par with the old one in terms of performance and stability before we even think about deprecation, let alone removal. I’ve heard a lot of complaints both publicly and privately about issues with the new spout. In retrospect I think we released it prematurely and would have been better off releasing it with some sort of “beta” or “preview” caveat until it further stabilized. Hindsight is 20/20. I’m hoping the fixes coming in the 1.1.1 release will ease some of the pain.

When I say I’m okay with deprecating the old spout, I’m saying that I’m okay with signaling to users that eventually in the (potentially very distant) future the old spout may be dropped. There are components of Storm that were deprecated 5 years ago that are still around, and I’m okay with that. IMHO it’s way too early to be discussing removal.

-Taylor

> On Jun 28, 2017, at 3:04 PM, Alexandre Vermeerbergen <av...@gmail.com> wrote:
> 
> Hello,
> 
> If that matters, our current experiences with StormKafkaClient
> isdisappointing (see my recent posts "Lag issues using Storm 1.1.1 latest
> build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this mailing
> list).
> 
> Our current experience is that the old StormKafka spout always beats the
> new one in term of performance & stability.
> 
> Therefore, I am surprised when I see talks about deprecation of the old
> StormKafka spout when the new one which just came "General Available" with
> Storm 1.1.0, is not stable, and it's not better when we try it from current
> 1.1.x builds to take into account recently closed JIRAs.
> 
> We're even considering writing our own Kafka spout with Kafka 0.10.x API to
> overcome the incompatibility of the old StormKafka spout with Kafka 0.10
> libraries.
> 
> Thus, for people which are comfortable with old Kafka spout, I'like to give
> a -1 (non binding) to the proposal of withdrawal of the old StormKafka
> spout until the new one converges.
> 
> Best regards,
> Alexandre Vermeerbergen
> 
> 
> 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:
> 
>> 
>>> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <hl...@hortonworks.com>
>> wrote:
>>> 
>>> I still need to go over the entire discussion thread in more detail, but
>> one thing I would like to bring up right way is the proposal to DEPRECATE,
>> and eventually remove, the KafkaSpout with the old Kafka Consumer APIs. The
>> storm-kafka-client KafkaSpout is getting stabilized, and I think we are all
>> in agreement that the storm-kafka KafkaSpout has presented continuous
>> maintainability problems with some fixes that got in not being backwards
>> compatible.
>> 
>> I’m fine with deprecating the old KafkaSpout, but I feel the decision to
>> actually remove it needs to take into account the user community. The main
>> sticking point here is compatibility with earlier versions of Kafka. Like
>> with JDK versions, there are many valid reasons whey users may not be in a
>> position to upgrade to a newer version of Kafka. Outright removal could
>> leave some users in the lurch.
>> 
>> Ideally, we could just poll the user community to get an idea of how much
>> of the user base depends on the old KafkaSpout and use the results to guide
>> our decision. Unfortunately, at least in my past experience, polling the
>> user@ list doesn’t elicit much of a response and the results don’t
>> provide an accurate view of the user community.
>> 
>> 
>>> 
>>> I am pretty confident how things are looking at this point for the
>> KafkaSpout. The Trident Kafka Spout is likely in between alpha and beta,
>> and that should be taken into account. I just recently submitted a PR<
>> https://github.com/apache/storm/pull/2174> with some improvements to the
>> Trident Kafka Spout (including the refactoring done to support manual
>> partition assignment), and there are some customers using it in
>> pre-production. However, it definitely would benefit from some more testing.
>>> 
>>> Thanks,
>>> Hugo
>> 
>> -Taylor
>> 
>> 
>>> 
>>> On Jun 28, 2017, at 7:48 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
>> mailto:evans@yahoo-inc.com.INVALID>> wrote:
>>> 
>>> +1.
>>> If the 1.1 and 1.2 lines start to become difficult to maintain we can
>> look at putting them in maintenance mode too once we have a 2.x release.
>>> I am a little nervous about merging a new feature into 1.x branch
>> without first going to master, but I hope that it will not be too much work
>> to port it to master, and I trust the devs on that branch to do the right
>> thing.
>>> On a related note we have not done much with feature branches before so
>> I am not sure what we want to do about merging in the new metrics API
>> branch to 1.x.  I know for me I have not had time to keep up with the
>> development work going on there.  I would at least like to have a pull
>> request put up for review before we merge it in.  This would fit with our
>> current bylaws that do not mention feature branches.  If all of the changes
>> have already followed the review process then technically I think it is OK
>> to just merge it in, but I still would like to take some time to look at
>> the changes, and especially the new APIs.
>>> 
>>> - Bobby
>>> 
>>> 
>>> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <
>> kabhwan@gmail.com<ma...@gmail.com>> wrote:
>>> 
>>> That's great news that metrics work is ready!
>>> 
>>> I'm +1 to Taylor's proposal, but in order to respect semantic
>> versioning, I
>>> propose some modifications from Taylor's proposal:
>>> 
>>> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back
>> only
>>> bug fixes to the 1.1.x-branch
>>> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
>>> 
>>> If we also agree above, I would like to volunteer the back-port work.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 6월 28일 (수) 오전 10:09, Harsha <storm@harsha.io<mailto:storm@
>> harsha.io>>님이 작성:
>>> 
>>> +1 for above stated approach on releasing 1.2.0 with metrics
>>> -Harsha
>>> 
>>> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
>>> The work on metrics is ready for a pull request to 1.x-branch from the
>>> feature branch. I’ve held off because we haven’t reached consensus on a
>>> path forward with the 1.x release lines .
>>> 
>>> I’d like to propose the following for the 1.x line:
>>> 
>>> 1. Create a branch for 1.2 so we have a branch to review the metrics
>>> stuff.
>>> 2. Release 1.1.1
>>> 3. Review/merge metrics work. Port metrics to master.
>>> 4. Release 1.2.0
>>> 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
>>> (we would only support 1.2.x and 1.1.x which are very closely aligned).
>>> 
>>> Dropping support for 1.0.x line would eliminate the need to maintain one
>>> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
>>> very closely aligned. I just up merged metrics_v2 against 1.x-branch
>>> after a while, and there were no conflicts.
>>> 
>>> That would give us a little more bandwidth to focus on 2.0 and needed bug
>>> fixes to the 1.x line like some of the issues raised with
>>> storm-kafka-client. We could even start releasing alpha/beta versions of
>>> 2.0 in parallel to the steps above.
>>> 
>>> Any thoughts on that approach?
>>> 
>>> -Taylor
>>> 
>>> 
>>> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:kabh
>> wan@gmail.com>> wrote:
>>> 
>>> Yes I prefer option 1, but it might depend on the progress of metrics
>>> V2.
>>> If it can be done within predictable near future I'm OK to pick option
>>> 2,
>>> but if not, we may be better to focus releasing 2.0.0 and make it
>>> really
>>> happen.
>>> 
>>> Whichever we go, I feel it's time to track remaining work on Storm
>>> 2.0.0. I
>>> found some bugs on master branch so filed issues, and we've remaining
>>> port
>>> work (UI and logviewer). We've some other improvements target for
>>> 2.0.0:
>>> worker redesign, beam integration, and so on, and we don't track its
>>> progress at all. I don't think we should wait for features which
>>> progress
>>> is not transparent (in other words we don't know when it will be
>>> finished).
>>> 
>>> - Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgoetz@gmail.com<mailto:ptgo
>> etz@gmail.com>>님이 작성:
>>> 
>>> Bobby/Jungtaek,
>>> 
>>> Are you saying you want to forego the 1.2 “metrics_v2” release and
>>> include
>>> it only in 2.0? (I ask because that work is already based on
>>> 1.x-branch,
>>> and forward-porting it to master is relatively simple.) I’d kind of
>>> like
>>> that work go out soon.
>>> 
>>> If we go with option 1, I would want to see a 2.0 release (even if
>>> it’s a
>>> “beta” or “preview) before putting the 1.x line into maintenance mode.
>>> 
>>> -Taylor
>>> 
>>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
>> mailto:evans@yahoo-inc.com.INVALID>
>>> 
>>> wrote:
>>> 
>>> I see 2 ways to address this.
>>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
>>> don't
>>> backport anything except bug fixes.2) We backport a lot of the
>>> backwards
>>> compatible changes from 2.x to 1.x.
>>> My personal preference is 1.  It makes it clear the direction we
>>> want to
>>> go in.  The biggest issue is that we probably would want to do a 2.x
>>> release sooner rather then later.  Even if we don't get all of the
>>> features
>>> that people want, if we just get a release out we can add in new
>>> features
>>> if they are backwards compatible, or we can create a 3.x line that
>>> would
>>> have the breaking changes in it.
>>> 
>>> - Bobby
>>> 
>>> 
>>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
>>> kabhwan@gmail.com<ma...@gmail.com>> wrote:
>>> 
>>> I'd like to bump this again instead of initiating new discussion
>>> thread.
>>> 
>>> I had having hard time to create and apply pull requests for both
>>> master
>>> and 1.x-branch and that's really painful and sometimes blocker for
>>> me to
>>> do
>>> merge step.
>>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
>>> even
>>> IDE can't switch between the branch smoothly. We didn't even address
>>> checkstyle issue yet, but after addressing, it could be "completely"
>>> diverged. JDK version is another major issue, since the pull requests
>>> targeted for master branch are not checked against JDK 7, and some of
>>> them
>>> make some issues regarding JDK version while porting back.
>>> 
>>> So personally I really would like to see the plan for 1.x version
>>> line
>>> changed - skipping any minor releases including 1.2.0 - and have epic
>>> issue
>>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
>>> proposed plan was having 2.0.0 directly after 1.0.0)
>>> 
>>> Would like to hear everyone's opinions. If we have consensus to not
>>> having
>>> any minor releases for 1.x version line, I will not port back
>>> non-bugfix
>>> pull requests to 1.x-branch, and guide contributors to create pull
>>> requests
>>> against master branch, not 1.x version line.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
>>> avermeerbergen@gmail.com<ma...@gmail.com>>님이
>>> 작성:
>>> 
>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
>>> system,
>>> we're very interested anything that can provide better throughput.
>>> 
>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <roshan@hortonworks.com<mailto:
>> roshan@hortonworks.com>>:
>>> 
>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
>>> be
>>> delivered sooner and my in-progress implementation suggests that it
>>> will
>>> yield substantial latency improvements. The 2.0 beta phase would
>>> really
>>> help kick the tires on the revised messaging system and the
>>> performance
>>> improvements will also be a good incentive for trying out the 2.0
>>> line.
>>> 
>>> I notice multiple other bottlenecks that are holding back
>>> throughput a
>>> lot, which can be addressed in a subsequent 2.x minor release.
>>> -roshan
>>> 
>>> 
>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabhwan@gmail.com<mailto:kabh
>> wan@gmail.com>> wrote:
>>> 
>>>   I also would love to see metrics V2 code sooner or later too.
>>> If we
>>> can get
>>>   it before releasing 2.0.0 that will be great, and then maybe we
>>> could
>>> just
>>>   move toward to 2.0.0, not adding any improvements to 1.x version
>>> line.
>>>   (And that's what I would want to.)
>>> 
>>>   If we would really want to have 1.2.0, I suggest that we make
>>> the
>>> 1.1.1
>>>   version correct right now rather than after releasing 1.1.1. We
>>> also
>>> merged
>>>   non-bugfix things to 1.x-branch but that's not what users
>>> expect. I
>>> agree
>>>   that work may be painful, but anyway need to do it.
>>> 
>>>   - Jungtaek Lim (HeartSaVioR)
>>> 
>>>   2017년 6월 3일 (토) 오전 3:49, Bobby Evans
>>> <ev...@yahoo-inc.com.invalid>
>>> 님이
>>> 작성:
>>> 
>>> I would love to see the metrics V2 code come out sooner rather
>>> than
>>> later.  +1.
>>> My biggest blocker for a 1.x release is
>>> https://github.com/apache/storm/pull/2142  Even though the pull
>>> request
>>> says it is minor it showed that we messed up pushing back some
>>> changes for
>>> pacemaker to open source (the code does not run at all which for
>>> me
>>> is a
>>> blocker) and I really want to get that fully fixed/tested before
>>> another
>>> release.
>>> As for 2.x I think we are very close to being able to so a 2.x
>>> alpha
>>> release.  I would like to see metrics V2 merged in simply because
>>> it
>>> is a
>>> big change for user facing APIs.  But after that I would love to
>>> see
>>> us
>>> starting to push forward on getting that out.
>>> 
>>> 
>>> - Bobby
>>> 
>>> 
>>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>>> ptgoetz@gmail.com<ma...@gmail.com>> wrote:
>>> 
>>> I’d like to bump this thread and start a discussion around our
>>> next
>>> release. Here are my thoughts.
>>> 
>>> There are a number of important fixes in 1.x-branch so I’d like
>>> to
>>> consider releasing 1.1.1 soon. I’d appreciate input on any open
>>> issues that
>>> should be resolved for that release.
>>> 
>>> I’d like us to consider releasing the metrics improvements in
>>> STORM-2153
>>> [1] as version 1.2.0. That work is in the metrics_v2 feature
>>> branch
>>> right
>>> now and would need to be reviewed and merged. That work is
>>> against
>>> the
>>> 1.x-branch right now. I would recommend porting it to master
>>> *after*
>>> the
>>> review/merge since there will likely be changes as a result of
>>> the
>>> review.
>>> 
>>> Maybe related to or not, but would we want to create a new
>>> branch
>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>> 
>>> 
>>> 
>>> If wee agree to the above, I would say yes. After the 1.1.1
>>> release,
>>> we
>>> could create a 1.1.x-branch that would be the maintenance/release
>>> branch
>>> for that version line. 1.x-branch would then become the target
>>> for
>>> the
>>> 1.2.0 release.
>>> 
>>> There are a few fixes in the 0.10.x branch that probably warrant
>>> a
>>> release. After that we may want to back away from that version
>>> line
>>> a bit
>>> so we can focus more on newer versions.
>>> 
>>> In the past, we’ve shied away form doing “beta” releases, but I’m
>>> wondering if we might want to revisit that for the 2.0 release —
>>> the
>>> idea
>>> being that it would give early adopter users a chance to kick the
>>> tires on
>>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
>>> help
>>> make the
>>> final release more solid. I’m on the fence here and could go
>>> either
>>> way.
>>> 
>>> I’d appreciate any input others may have.
>>> 
>>> 
>>> Thanks,
>>> 
>>> -Taylor
>>> 
>>> 
>>> [1] https://issues.apache.org/jira/browse/STORM-2153
>>> 
>>> 
>>> 
>>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabhwan@gmail.com<mailto:kabh
>> wan@gmail.com>>
>>> wrote:
>>> 
>>> Maybe related to or not, but would we want to create a new
>>> branch
>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>> 
>>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
>>> so
>>> hence
>>> the
>>> question.
>>> 
>>> - Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
>>> hlouro@hortonworks.com<ma...@hortonworks.com>>님이
>>> 작성:
>>> 
>>> +1 for finishing the porting to Java ahead of anything else -
>>> it
>>> will
>>> be a
>>> significant milestone. I have a JIRA assigned concerning to
>>> the
>>> porting. I
>>> will work on it for the 2.0 release.
>>> 
>>> it’s a priority to guarantee no performance regressions. As
>>> part
>>> of this
>>> endeavor, explore an automated (or easy) way to run and assert
>>> major
>>> performance benchmarks. Ideally any contributor should be able
>>> to
>>> fairly
>>> easily test the impact of changes under certain performance
>>> test
>>> scenarios.
>>> 
>>> Beam Runner work should take into account the impact of
>>> incorporating
>>> new
>>> JStorm features and Storm Worker Redesign<
>>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
>>> efficient
>>> to
>>> start doing it, to  find out that it will have to chance in
>>> face
>>> of
>>> Storm
>>> and worker redesign. That is, it should be done after it’s
>>> building
>>> blocks
>>> are stable.
>>> 
>>> Thanks,
>>> Hugo
>>> 
>>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
>>> arunm@apache.org<ma...@apache.org>
>>> <mailto:
>>> arunm@apache.org>> wrote:
>>> 
>>> +1 to release with the porting completed. I think its mainly
>>> the
>>> UI
>>> server
>>> and log viewer that’s pending.
>>> 
>>> We can start doing the regression and performance tests for
>>> whatever is
>>> already ported.
>>> 
>>> If anyone is running the master branch in their pre-prod /
>>> prod
>>> environments, it will be good to know and give us more
>>> confidence.
>>> 
>>> The other features can be added in follow up releases.
>>> 
>>> Regards,
>>> Arun
>>> 
>>> 
>>> On 3/24/17, 11:47 AM, "Satish Duggana" <
>>> satish.duggana@gmail.com
>>> <mailto:
>>> satish.duggana@gmail.com>> wrote:
>>> 
>>> +1 to have 2.0 with porting and performance(it should be at
>>> least
>>> as
>>> good
>>> as 1.x release) issues addressed
>>> 
>>> We can target other tasks(mentioned by Taylor and Jungtaek)
>>> for
>>> 2.x-branch.
>>> 
>>> 
>>> Exactly-once support:
>>> While thinking through the exactlyonce support design, it is
>>> realized
>>> better to avoid acking tuples and implement exactly once by
>>> snapshotting
>>> barriers. It seems JStorm folks followed similar design, they
>>> claim it
>>> gives better performance. This feature is essential for beam
>>> runner and
>>> we
>>> can decide on respective approaches though.
>>> 
>>> Beam Runner
>>> Lets hold on this for now and keep it in Storm till 2.x. We
>>> should avoid
>>> having a minimal beam runner in haste. It is better to address
>>> STORM-2284,
>>> exactly-once and other windowing enhancements to enable beam
>>> runner.
>>> 
>>> JStorm
>>> Agree with Jungtaek on looking at the latest JStorm and
>>> align/scope with
>>> the features for 2.x.
>>> 
>>> STORM-2284
>>> We may want to look at JStorm worker before working on
>>> respective
>>> components in this epic to pull appropriate enhancements.
>>> 
>>> YARN/MESOS
>>> Supporting Storm on YARN/Mesos for 2.x.
>>> 
>>> Thanks,
>>> Satish.
>>> 
>>> 
>>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
>>> kabhwan@gmail.com
>>> <mailto:
>>> kabhwan@gmail.com>> wrote:
>>> 
>>> First of all, +1 to complete only port work and do sanity
>>> check
>>> (including
>>> performance regression), and release.
>>> 
>>> If we can get STORM-2284 within deterministic time frame (say
>>> 2~3
>>> months)
>>> that should be great, but if not I'd in favor of postponing
>>> that
>>> to
>>> later
>>> 2.x release.
>>> 
>>> JStorm released their new versions after code donation. So
>>> there're more
>>> things we could get ideas from, or even adopt from.
>>> https://github.com/alibaba/jstorm/blob/master/history.md
>>> As you noticed from release note link, we also need to update
>>> phase 2
>>> since
>>> they already changed what we're planning to do in phase 2. For
>>> example,
>>> they changed backpressure to end-to-end, and changed to use
>>> snapshot
>>> rather
>>> than acker.
>>> May be sure, JStorm pulled many features from today's Storm,
>>> like
>>> Flux,
>>> Windowing, more shuffle groupings, log search, log level
>>> change,
>>> and so
>>> on.
>>> 
>>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
>>> is
>>> due to
>>> the
>>> limitation of Spout lifecycle (all the things are done in
>>> single
>>> thread),
>>> and STORM-1358 <
>>> https://issues.apache.org/jira/browse/STORM-1358
>>> (JStorm's
>>> multi-thread Spout) can remedy this (despite that Spout
>>> implementation
>>> may
>>> need to guarantee thread-safety later). It's not a just
>>> improvement but
>>> close to design concern so would like to address sooner than
>>> other
>>> things
>>> in phase 2.
>>> 
>>> For Storm SQL side, I've lost progress but major work would be
>>> adopting
>>> group by with windowing. It was not available from Calcite but
>>> will be
>>> available at next release (1.12.0).
>>> I've filed this to STORM-2405
>>> <https://issues.apache.org/jira/browse/STORM-2405>, but
>>> windowing &
>>> micro
>>> batch is not intuitive, so I would like to change the
>>> underlying
>>> API to
>>> stream API in SQL. Also filed this to STORM-2406
>>> <https://issues.apache.org/jira/browse/STORM-2406>.
>>> 
>>> Just 2 cents btw, hopefully I would like to see metrics V2
>>> sooner
>>> since
>>> we
>>> lost metrics even when doing normal operation like restarting
>>> worker,
>>> rebalancing, and so on. Eventually we need to fight with
>>> dynamic
>>> scaling,
>>> and then metrics will be broken often.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
>>> storm@harsha.io
>>> 님이
>>> 작성:
>>> 
>>> Storm 2.0 migration to java in itself is a big win and would
>>> attract
>>> wider
>>> community and adoption. So my vote would be to resolve the
>>> first
>>> 3 items
>>> to
>>> get a release out.
>>> All the other featured mentioned are great to have but
>>> shouldn't
>>> be
>>> blockers for 2.0 release.
>>> 
>>> -Harsha
>>> 
>>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
>>> ptgoetz@gmail.com>
>>> wrote:
>>> 
>>> With the 1.1.0 release nearing completion, I’d like to turn
>>> our
>>> attention
>>> to 2.0 and develop a plan for what features, etc. to include.
>>> 
>>> The following 3 are what I feel are the minimum for a 2.0
>>> release.
>>> These
>>> could likely be resolved relatively quickly:
>>> 
>>> * Performance — I’ve not benchmarked the master branch vs.
>>> 1.0.x
>>> or
>>> 1.1.x
>>> in a while, but I feel it will be important to make sure there
>>> are no
>>> performance regressions, and would hope that we actually have
>>> a
>>> performance
>>> improvement over previous versions. To that end (e.g. if there
>>> is
>>> in
>>> fact a
>>> performance regression), the proposals that Roshan Naik put
>>> together
>>> for
>>> revising the threading and execution model (STORM-2307) and
>>> replacing
>>> Disruptor with JCTools (STORM-2306) warrant review and
>>> consideration.
>>> See
>>> also STORM-2284 which is the parent JIRA.
>>> 
>>> * Finish porting Storm UI to java (STORM-1311)
>>> 
>>> * Finish porting log viewer to java (STORM-1280)
>>> 
>>> The following are items that are nice to have in 2.0, but I
>>> don’t
>>> feel
>>> are
>>> absolutely necessary for an initial 2.0 release:
>>> 
>>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
>>> because
>>> it’s
>>> relevant) — Initially there seemed to be a lot of interest in
>>> this, but
>>> that seems to have trailed off. I spoke with some Beam
>>> developers
>>> and
>>> there
>>> seems to be interest from that community as well. Do we want
>>> to
>>> move
>>> that
>>> effort to the Beam community, or keep it here? Moving it to
>>> the
>>> Beam
>>> community might lead to better collaboration between projects.
>>> 
>>> * Bounded Spouts (needed for Beam Runner implementation) —
>>> Currently
>>> spouts are unbounded, there no end to the stream. Beam has the
>>> concept
>>> of
>>> bounded sources (roughly analogous to batch processing). To
>>> support
>>> that,
>>> we would need to implement a similar concept in Storm. One
>>> benefit of
>>> such
>>> a feature would be the ability to handle both bounded and
>>> unbounded
>>> workflows in Storm.
>>> 
>>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
>>> behind
>>> this
>>> effort. What improvements do you envision for 2.0?
>>> 
>>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
>>> targeting this
>>> for 1.2.0, but it’s designed to be easily portable to
>>> master/2.0.
>>> 
>>> * JStorm Migration — Original outline can be found here [1].
>>> Note
>>> a lot
>>> of
>>> the associated JIRAs below are assigned, but there hasn’t been
>>> any
>>> recent
>>> activity or pull requests, we should probably consider them
>>> unassigned
>>> and
>>> up for grabs.:
>>> 
>>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
>>> been the
>>> bane of a lot of Storm users almost since day one. We have
>>> largely
>>> addressed it by shading/relocating dependencies. It would be
>>> great to
>>> see
>>> this addressed once and for all.
>>> 
>>> * JStorm back pressure implementation (STORM-1324) — The
>>> current
>>> back
>>> pressure implementation leaves a bit to be desired, and the
>>> JStorm
>>> approach
>>> looks promising, though it also depends on the JStorm concept
>>> of
>>> “topology
>>> master” (STORM-1323), which may have some implications
>>> regarding
>>> security.
>>> 
>>> * Dynamic Topology Updates (STORM-1335) — This would provide a
>>> command
>>> to
>>> update topology jars and configuration without stopping the
>>> topology,
>>> and
>>> is well suited to leverage the blobstore. The restart command
>>> (that can
>>> also update the topology configuration) also looks compelling
>>> (STORM-1334).
>>> 
>>> * Additional Scheduler Implementations (STORM-1320)
>>> 
>>> * Additional Grouping Implementations (STORM-1328)
>>> 
>>> 
>>> As always I’m open to any opinions and suggestions.
>>> 
>>> -Taylor
>>> 
>>> [1]
>>> 
>>> https://cwiki.apache.org/confluence/pages/viewpage.
>>> action?pageId=61328109
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello,

If that matters, our current experiences with StormKafkaClient
isdisappointing (see my recent posts "Lag issues using Storm 1.1.1 latest
build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this mailing
list).

Our current experience is that the old StormKafka spout always beats the
new one in term of performance & stability.

Therefore, I am surprised when I see talks about deprecation of the old
StormKafka spout when the new one which just came "General Available" with
Storm 1.1.0, is not stable, and it's not better when we try it from current
1.1.x builds to take into account recently closed JIRAs.

We're even considering writing our own Kafka spout with Kafka 0.10.x API to
overcome the incompatibility of the old StormKafka spout with Kafka 0.10
libraries.

Thus, for people which are comfortable with old Kafka spout, I'like to give
a -1 (non binding) to the proposal of withdrawal of the old StormKafka
spout until the new one converges.

Best regards,
Alexandre Vermeerbergen


2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:

>
> > On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <hl...@hortonworks.com>
> wrote:
> >
> > I still need to go over the entire discussion thread in more detail, but
> one thing I would like to bring up right way is the proposal to DEPRECATE,
> and eventually remove, the KafkaSpout with the old Kafka Consumer APIs. The
> storm-kafka-client KafkaSpout is getting stabilized, and I think we are all
> in agreement that the storm-kafka KafkaSpout has presented continuous
> maintainability problems with some fixes that got in not being backwards
> compatible.
>
> I’m fine with deprecating the old KafkaSpout, but I feel the decision to
> actually remove it needs to take into account the user community. The main
> sticking point here is compatibility with earlier versions of Kafka. Like
> with JDK versions, there are many valid reasons whey users may not be in a
> position to upgrade to a newer version of Kafka. Outright removal could
> leave some users in the lurch.
>
> Ideally, we could just poll the user community to get an idea of how much
> of the user base depends on the old KafkaSpout and use the results to guide
> our decision. Unfortunately, at least in my past experience, polling the
> user@ list doesn’t elicit much of a response and the results don’t
> provide an accurate view of the user community.
>
>
> >
> > I am pretty confident how things are looking at this point for the
> KafkaSpout. The Trident Kafka Spout is likely in between alpha and beta,
> and that should be taken into account. I just recently submitted a PR<
> https://github.com/apache/storm/pull/2174> with some improvements to the
> Trident Kafka Spout (including the refactoring done to support manual
> partition assignment), and there are some customers using it in
> pre-production. However, it definitely would benefit from some more testing.
> >
> > Thanks,
> > Hugo
>
> -Taylor
>
>
> >
> > On Jun 28, 2017, at 7:48 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
> mailto:evans@yahoo-inc.com.INVALID>> wrote:
> >
> > +1.
> > If the 1.1 and 1.2 lines start to become difficult to maintain we can
> look at putting them in maintenance mode too once we have a 2.x release.
> > I am a little nervous about merging a new feature into 1.x branch
> without first going to master, but I hope that it will not be too much work
> to port it to master, and I trust the devs on that branch to do the right
> thing.
> > On a related note we have not done much with feature branches before so
> I am not sure what we want to do about merging in the new metrics API
> branch to 1.x.  I know for me I have not had time to keep up with the
> development work going on there.  I would at least like to have a pull
> request put up for review before we merge it in.  This would fit with our
> current bylaws that do not mention feature branches.  If all of the changes
> have already followed the review process then technically I think it is OK
> to just merge it in, but I still would like to take some time to look at
> the changes, and especially the new APIs.
> >
> > - Bobby
> >
> >
> > On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <
> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> >
> > That's great news that metrics work is ready!
> >
> > I'm +1 to Taylor's proposal, but in order to respect semantic
> versioning, I
> > propose some modifications from Taylor's proposal:
> >
> > - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back
> only
> > bug fixes to the 1.1.x-branch
> > - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
> >
> > If we also agree above, I would like to volunteer the back-port work.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 6월 28일 (수) 오전 10:09, Harsha <storm@harsha.io<mailto:storm@
> harsha.io>>님이 작성:
> >
> > +1 for above stated approach on releasing 1.2.0 with metrics
> > -Harsha
> >
> > On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> > The work on metrics is ready for a pull request to 1.x-branch from the
> > feature branch. I’ve held off because we haven’t reached consensus on a
> > path forward with the 1.x release lines .
> >
> > I’d like to propose the following for the 1.x line:
> >
> > 1. Create a branch for 1.2 so we have a branch to review the metrics
> > stuff.
> > 2. Release 1.1.1
> > 3. Review/merge metrics work. Port metrics to master.
> > 4. Release 1.2.0
> > 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
> > (we would only support 1.2.x and 1.1.x which are very closely aligned).
> >
> > Dropping support for 1.0.x line would eliminate the need to maintain one
> > of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> > very closely aligned. I just up merged metrics_v2 against 1.x-branch
> > after a while, and there were no conflicts.
> >
> > That would give us a little more bandwidth to focus on 2.0 and needed bug
> > fixes to the 1.x line like some of the issues raised with
> > storm-kafka-client. We could even start releasing alpha/beta versions of
> > 2.0 in parallel to the steps above.
> >
> > Any thoughts on that approach?
> >
> > -Taylor
> >
> >
> > On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:kabh
> wan@gmail.com>> wrote:
> >
> > Yes I prefer option 1, but it might depend on the progress of metrics
> > V2.
> > If it can be done within predictable near future I'm OK to pick option
> > 2,
> > but if not, we may be better to focus releasing 2.0.0 and make it
> > really
> > happen.
> >
> > Whichever we go, I feel it's time to track remaining work on Storm
> > 2.0.0. I
> > found some bugs on master branch so filed issues, and we've remaining
> > port
> > work (UI and logviewer). We've some other improvements target for
> > 2.0.0:
> > worker redesign, beam integration, and so on, and we don't track its
> > progress at all. I don't think we should wait for features which
> > progress
> > is not transparent (in other words we don't know when it will be
> > finished).
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgoetz@gmail.com<mailto:ptgo
> etz@gmail.com>>님이 작성:
> >
> > Bobby/Jungtaek,
> >
> > Are you saying you want to forego the 1.2 “metrics_v2” release and
> > include
> > it only in 2.0? (I ask because that work is already based on
> > 1.x-branch,
> > and forward-porting it to master is relatively simple.) I’d kind of
> > like
> > that work go out soon.
> >
> > If we go with option 1, I would want to see a 2.0 release (even if
> > it’s a
> > “beta” or “preview) before putting the 1.x line into maintenance mode.
> >
> > -Taylor
> >
> > On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID<
> mailto:evans@yahoo-inc.com.INVALID>
> >
> > wrote:
> >
> > I see 2 ways to address this.
> > 1) We put the 1.x line into maintenance mode like with 0.10.  We
> > don't
> > backport anything except bug fixes.2) We backport a lot of the
> > backwards
> > compatible changes from 2.x to 1.x.
> > My personal preference is 1.  It makes it clear the direction we
> > want to
> > go in.  The biggest issue is that we probably would want to do a 2.x
> > release sooner rather then later.  Even if we don't get all of the
> > features
> > that people want, if we just get a release out we can add in new
> > features
> > if they are backwards compatible, or we can create a 3.x line that
> > would
> > have the breaking changes in it.
> >
> > - Bobby
> >
> >
> > On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> > kabhwan@gmail.com<ma...@gmail.com>> wrote:
> >
> > I'd like to bump this again instead of initiating new discussion
> > thread.
> >
> > I had having hard time to create and apply pull requests for both
> > master
> > and 1.x-branch and that's really painful and sometimes blocker for
> > me to
> > do
> > merge step.
> > Two branches are heavily diverged more than between 0.10 and 1.0.0,
> > even
> > IDE can't switch between the branch smoothly. We didn't even address
> > checkstyle issue yet, but after addressing, it could be "completely"
> > diverged. JDK version is another major issue, since the pull requests
> > targeted for master branch are not checked against JDK 7, and some of
> > them
> > make some issues regarding JDK version while porting back.
> >
> > So personally I really would like to see the plan for 1.x version
> > line
> > changed - skipping any minor releases including 1.2.0 - and have epic
> > issue
> > for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> > proposed plan was having 2.0.0 directly after 1.0.0)
> >
> > Would like to hear everyone's opinions. If we have consensus to not
> > having
> > any minor releases for 1.x version line, I will not port back
> > non-bugfix
> > pull requests to 1.x-branch, and guide contributors to create pull
> > requests
> > against master branch, not 1.x version line.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com<ma...@gmail.com>>님이
> > 작성:
> >
> > +1 for Roshan's suggestion : in our Storm 1.x based supervision
> > system,
> > we're very interested anything that can provide better throughput.
> >
> > 2017-06-03 18:12 GMT+02:00 Roshan Naik <roshan@hortonworks.com<mailto:
> roshan@hortonworks.com>>:
> >
> > For 2.0 beta … it would be good to incorporate some of the Worker
> > improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> > be
> > delivered sooner and my in-progress implementation suggests that it
> > will
> > yield substantial latency improvements. The 2.0 beta phase would
> > really
> > help kick the tires on the revised messaging system and the
> > performance
> > improvements will also be a good incentive for trying out the 2.0
> > line.
> >
> > I notice multiple other bottlenecks that are holding back
> > throughput a
> > lot, which can be addressed in a subsequent 2.x minor release.
> > -roshan
> >
> >
> > On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabhwan@gmail.com<mailto:kabh
> wan@gmail.com>> wrote:
> >
> >    I also would love to see metrics V2 code sooner or later too.
> > If we
> > can get
> >    it before releasing 2.0.0 that will be great, and then maybe we
> > could
> > just
> >    move toward to 2.0.0, not adding any improvements to 1.x version
> > line.
> >    (And that's what I would want to.)
> >
> >    If we would really want to have 1.2.0, I suggest that we make
> > the
> > 1.1.1
> >    version correct right now rather than after releasing 1.1.1. We
> > also
> > merged
> >    non-bugfix things to 1.x-branch but that's not what users
> > expect. I
> > agree
> >    that work may be painful, but anyway need to do it.
> >
> >    - Jungtaek Lim (HeartSaVioR)
> >
> >    2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> > <ev...@yahoo-inc.com.invalid>
> > 님이
> > 작성:
> >
> > I would love to see the metrics V2 code come out sooner rather
> > than
> > later.  +1.
> > My biggest blocker for a 1.x release is
> > https://github.com/apache/storm/pull/2142  Even though the pull
> > request
> > says it is minor it showed that we messed up pushing back some
> > changes for
> > pacemaker to open source (the code does not run at all which for
> > me
> > is a
> > blocker) and I really want to get that fully fixed/tested before
> > another
> > release.
> > As for 2.x I think we are very close to being able to so a 2.x
> > alpha
> > release.  I would like to see metrics V2 merged in simply because
> > it
> > is a
> > big change for user facing APIs.  But after that I would love to
> > see
> > us
> > starting to push forward on getting that out.
> >
> >
> > - Bobby
> >
> >
> > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> > ptgoetz@gmail.com<ma...@gmail.com>> wrote:
> >
> > I’d like to bump this thread and start a discussion around our
> > next
> > release. Here are my thoughts.
> >
> > There are a number of important fixes in 1.x-branch so I’d like
> > to
> > consider releasing 1.1.1 soon. I’d appreciate input on any open
> > issues that
> > should be resolved for that release.
> >
> > I’d like us to consider releasing the metrics improvements in
> > STORM-2153
> > [1] as version 1.2.0. That work is in the metrics_v2 feature
> > branch
> > right
> > now and would need to be reviewed and merged. That work is
> > against
> > the
> > 1.x-branch right now. I would recommend porting it to master
> > *after*
> > the
> > review/merge since there will likely be changes as a result of
> > the
> > review.
> >
> > Maybe related to or not, but would we want to create a new
> > branch
> > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >
> >
> >
> > If wee agree to the above, I would say yes. After the 1.1.1
> > release,
> > we
> > could create a 1.1.x-branch that would be the maintenance/release
> > branch
> > for that version line. 1.x-branch would then become the target
> > for
> > the
> > 1.2.0 release.
> >
> > There are a few fixes in the 0.10.x branch that probably warrant
> > a
> > release. After that we may want to back away from that version
> > line
> > a bit
> > so we can focus more on newer versions.
> >
> > In the past, we’ve shied away form doing “beta” releases, but I’m
> > wondering if we might want to revisit that for the 2.0 release —
> > the
> > idea
> > being that it would give early adopter users a chance to kick the
> > tires on
> > what’s coming in 2.0 and provide feedback, find bugs, etc. to
> > help
> > make the
> > final release more solid. I’m on the fence here and could go
> > either
> > way.
> >
> > I’d appreciate any input others may have.
> >
> >
> > Thanks,
> >
> > -Taylor
> >
> >
> > [1] https://issues.apache.org/jira/browse/STORM-2153
> >
> >
> >
> > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabhwan@gmail.com<mailto:kabh
> wan@gmail.com>>
> > wrote:
> >
> > Maybe related to or not, but would we want to create a new
> > branch
> > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >
> > I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> > so
> > hence
> > the
> > question.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > hlouro@hortonworks.com<ma...@hortonworks.com>>님이
> > 작성:
> >
> > +1 for finishing the porting to Java ahead of anything else -
> > it
> > will
> > be a
> > significant milestone. I have a JIRA assigned concerning to
> > the
> > porting. I
> > will work on it for the 2.0 release.
> >
> > it’s a priority to guarantee no performance regressions. As
> > part
> > of this
> > endeavor, explore an automated (or easy) way to run and assert
> > major
> > performance benchmarks. Ideally any contributor should be able
> > to
> > fairly
> > easily test the impact of changes under certain performance
> > test
> > scenarios.
> >
> > Beam Runner work should take into account the impact of
> > incorporating
> > new
> > JStorm features and Storm Worker Redesign<
> > https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > efficient
> > to
> > start doing it, to  find out that it will have to chance in
> > face
> > of
> > Storm
> > and worker redesign. That is, it should be done after it’s
> > building
> > blocks
> > are stable.
> >
> > Thanks,
> > Hugo
> >
> > On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> > arunm@apache.org<ma...@apache.org>
> > <mailto:
> > arunm@apache.org>> wrote:
> >
> > +1 to release with the porting completed. I think its mainly
> > the
> > UI
> > server
> > and log viewer that’s pending.
> >
> > We can start doing the regression and performance tests for
> > whatever is
> > already ported.
> >
> > If anyone is running the master branch in their pre-prod /
> > prod
> > environments, it will be good to know and give us more
> > confidence.
> >
> > The other features can be added in follow up releases.
> >
> > Regards,
> > Arun
> >
> >
> > On 3/24/17, 11:47 AM, "Satish Duggana" <
> > satish.duggana@gmail.com
> > <mailto:
> > satish.duggana@gmail.com>> wrote:
> >
> > +1 to have 2.0 with porting and performance(it should be at
> > least
> > as
> > good
> > as 1.x release) issues addressed
> >
> > We can target other tasks(mentioned by Taylor and Jungtaek)
> > for
> > 2.x-branch.
> >
> >
> > Exactly-once support:
> > While thinking through the exactlyonce support design, it is
> > realized
> > better to avoid acking tuples and implement exactly once by
> > snapshotting
> > barriers. It seems JStorm folks followed similar design, they
> > claim it
> > gives better performance. This feature is essential for beam
> > runner and
> > we
> > can decide on respective approaches though.
> >
> > Beam Runner
> > Lets hold on this for now and keep it in Storm till 2.x. We
> > should avoid
> > having a minimal beam runner in haste. It is better to address
> > STORM-2284,
> > exactly-once and other windowing enhancements to enable beam
> > runner.
> >
> > JStorm
> > Agree with Jungtaek on looking at the latest JStorm and
> > align/scope with
> > the features for 2.x.
> >
> > STORM-2284
> > We may want to look at JStorm worker before working on
> > respective
> > components in this epic to pull appropriate enhancements.
> >
> > YARN/MESOS
> > Supporting Storm on YARN/Mesos for 2.x.
> >
> > Thanks,
> > Satish.
> >
> >
> > On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> > kabhwan@gmail.com
> > <mailto:
> > kabhwan@gmail.com>> wrote:
> >
> > First of all, +1 to complete only port work and do sanity
> > check
> > (including
> > performance regression), and release.
> >
> > If we can get STORM-2284 within deterministic time frame (say
> > 2~3
> > months)
> > that should be great, but if not I'd in favor of postponing
> > that
> > to
> > later
> > 2.x release.
> >
> > JStorm released their new versions after code donation. So
> > there're more
> > things we could get ideas from, or even adopt from.
> > https://github.com/alibaba/jstorm/blob/master/history.md
> > As you noticed from release note link, we also need to update
> > phase 2
> > since
> > they already changed what we're planning to do in phase 2. For
> > example,
> > they changed backpressure to end-to-end, and changed to use
> > snapshot
> > rather
> > than acker.
> > May be sure, JStorm pulled many features from today's Storm,
> > like
> > Flux,
> > Windowing, more shuffle groupings, log search, log level
> > change,
> > and so
> > on.
> >
> > STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> > is
> > due to
> > the
> > limitation of Spout lifecycle (all the things are done in
> > single
> > thread),
> > and STORM-1358 <
> > https://issues.apache.org/jira/browse/STORM-1358
> > (JStorm's
> > multi-thread Spout) can remedy this (despite that Spout
> > implementation
> > may
> > need to guarantee thread-safety later). It's not a just
> > improvement but
> > close to design concern so would like to address sooner than
> > other
> > things
> > in phase 2.
> >
> > For Storm SQL side, I've lost progress but major work would be
> > adopting
> > group by with windowing. It was not available from Calcite but
> > will be
> > available at next release (1.12.0).
> > I've filed this to STORM-2405
> > <https://issues.apache.org/jira/browse/STORM-2405>, but
> > windowing &
> > micro
> > batch is not intuitive, so I would like to change the
> > underlying
> > API to
> > stream API in SQL. Also filed this to STORM-2406
> > <https://issues.apache.org/jira/browse/STORM-2406>.
> >
> > Just 2 cents btw, hopefully I would like to see metrics V2
> > sooner
> > since
> > we
> > lost metrics even when doing normal operation like restarting
> > worker,
> > rebalancing, and so on. Eventually we need to fight with
> > dynamic
> > scaling,
> > and then metrics will be broken often.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> > storm@harsha.io
> > 님이
> > 작성:
> >
> > Storm 2.0 migration to java in itself is a big win and would
> > attract
> > wider
> > community and adoption. So my vote would be to resolve the
> > first
> > 3 items
> > to
> > get a release out.
> > All the other featured mentioned are great to have but
> > shouldn't
> > be
> > blockers for 2.0 release.
> >
> > -Harsha
> >
> > On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > ptgoetz@gmail.com>
> > wrote:
> >
> > With the 1.1.0 release nearing completion, I’d like to turn
> > our
> > attention
> > to 2.0 and develop a plan for what features, etc. to include.
> >
> > The following 3 are what I feel are the minimum for a 2.0
> > release.
> > These
> > could likely be resolved relatively quickly:
> >
> > * Performance — I’ve not benchmarked the master branch vs.
> > 1.0.x
> > or
> > 1.1.x
> > in a while, but I feel it will be important to make sure there
> > are no
> > performance regressions, and would hope that we actually have
> > a
> > performance
> > improvement over previous versions. To that end (e.g. if there
> > is
> > in
> > fact a
> > performance regression), the proposals that Roshan Naik put
> > together
> > for
> > revising the threading and execution model (STORM-2307) and
> > replacing
> > Disruptor with JCTools (STORM-2306) warrant review and
> > consideration.
> > See
> > also STORM-2284 which is the parent JIRA.
> >
> > * Finish porting Storm UI to java (STORM-1311)
> >
> > * Finish porting log viewer to java (STORM-1280)
> >
> > The following are items that are nice to have in 2.0, but I
> > don’t
> > feel
> > are
> > absolutely necessary for an initial 2.0 release:
> >
> > * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> > because
> > it’s
> > relevant) — Initially there seemed to be a lot of interest in
> > this, but
> > that seems to have trailed off. I spoke with some Beam
> > developers
> > and
> > there
> > seems to be interest from that community as well. Do we want
> > to
> > move
> > that
> > effort to the Beam community, or keep it here? Moving it to
> > the
> > Beam
> > community might lead to better collaboration between projects.
> >
> > * Bounded Spouts (needed for Beam Runner implementation) —
> > Currently
> > spouts are unbounded, there no end to the stream. Beam has the
> > concept
> > of
> > bounded sources (roughly analogous to batch processing). To
> > support
> > that,
> > we would need to implement a similar concept in Storm. One
> > benefit of
> > such
> > a feature would be the ability to handle both bounded and
> > unbounded
> > workflows in Storm.
> >
> > * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > behind
> > this
> > effort. What improvements do you envision for 2.0?
> >
> > * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > targeting this
> > for 1.2.0, but it’s designed to be easily portable to
> > master/2.0.
> >
> > * JStorm Migration — Original outline can be found here [1].
> > Note
> > a lot
> > of
> > the associated JIRAs below are assigned, but there hasn’t been
> > any
> > recent
> > activity or pull requests, we should probably consider them
> > unassigned
> > and
> > up for grabs.:
> >
> > * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > been the
> > bane of a lot of Storm users almost since day one. We have
> > largely
> > addressed it by shading/relocating dependencies. It would be
> > great to
> > see
> > this addressed once and for all.
> >
> > * JStorm back pressure implementation (STORM-1324) — The
> > current
> > back
> > pressure implementation leaves a bit to be desired, and the
> > JStorm
> > approach
> > looks promising, though it also depends on the JStorm concept
> > of
> > “topology
> > master” (STORM-1323), which may have some implications
> > regarding
> > security.
> >
> > * Dynamic Topology Updates (STORM-1335) — This would provide a
> > command
> > to
> > update topology jars and configuration without stopping the
> > topology,
> > and
> > is well suited to leverage the blobstore. The restart command
> > (that can
> > also update the topology configuration) also looks compelling
> > (STORM-1334).
> >
> > * Additional Scheduler Implementations (STORM-1320)
> >
> > * Additional Grouping Implementations (STORM-1328)
> >
> >
> > As always I’m open to any opinions and suggestions.
> >
> > -Taylor
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=61328109
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <hl...@hortonworks.com> wrote:
> 
> I still need to go over the entire discussion thread in more detail, but one thing I would like to bring up right way is the proposal to DEPRECATE, and eventually remove, the KafkaSpout with the old Kafka Consumer APIs. The storm-kafka-client KafkaSpout is getting stabilized, and I think we are all in agreement that the storm-kafka KafkaSpout has presented continuous maintainability problems with some fixes that got in not being backwards compatible.

I’m fine with deprecating the old KafkaSpout, but I feel the decision to actually remove it needs to take into account the user community. The main sticking point here is compatibility with earlier versions of Kafka. Like with JDK versions, there are many valid reasons whey users may not be in a position to upgrade to a newer version of Kafka. Outright removal could leave some users in the lurch.

Ideally, we could just poll the user community to get an idea of how much of the user base depends on the old KafkaSpout and use the results to guide our decision. Unfortunately, at least in my past experience, polling the user@ list doesn’t elicit much of a response and the results don’t provide an accurate view of the user community. 


> 
> I am pretty confident how things are looking at this point for the KafkaSpout. The Trident Kafka Spout is likely in between alpha and beta, and that should be taken into account. I just recently submitted a PR<https://github.com/apache/storm/pull/2174> with some improvements to the Trident Kafka Spout (including the refactoring done to support manual partition assignment), and there are some customers using it in pre-production. However, it definitely would benefit from some more testing.
> 
> Thanks,
> Hugo

-Taylor


> 
> On Jun 28, 2017, at 7:48 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
> 
> +1.
> If the 1.1 and 1.2 lines start to become difficult to maintain we can look at putting them in maintenance mode too once we have a 2.x release.
> I am a little nervous about merging a new feature into 1.x branch without first going to master, but I hope that it will not be too much work to port it to master, and I trust the devs on that branch to do the right thing.
> On a related note we have not done much with feature branches before so I am not sure what we want to do about merging in the new metrics API branch to 1.x.  I know for me I have not had time to keep up with the development work going on there.  I would at least like to have a pull request put up for review before we merge it in.  This would fit with our current bylaws that do not mention feature branches.  If all of the changes have already followed the review process then technically I think it is OK to just merge it in, but I still would like to take some time to look at the changes, and especially the new APIs.
> 
> - Bobby
> 
> 
> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <ka...@gmail.com>> wrote:
> 
> That's great news that metrics work is ready!
> 
> I'm +1 to Taylor's proposal, but in order to respect semantic versioning, I
> propose some modifications from Taylor's proposal:
> 
> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back only
> bug fixes to the 1.1.x-branch
> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT
> 
> If we also agree above, I would like to volunteer the back-port work.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 28일 (수) 오전 10:09, Harsha <st...@harsha.io>>님이 작성:
> 
> +1 for above stated approach on releasing 1.2.0 with metrics
> -Harsha
> 
> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> The work on metrics is ready for a pull request to 1.x-branch from the
> feature branch. I’ve held off because we haven’t reached consensus on a
> path forward with the 1.x release lines .
> 
> I’d like to propose the following for the 1.x line:
> 
> 1. Create a branch for 1.2 so we have a branch to review the metrics
> stuff.
> 2. Release 1.1.1
> 3. Review/merge metrics work. Port metrics to master.
> 4. Release 1.2.0
> 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
> (we would only support 1.2.x and 1.1.x which are very closely aligned).
> 
> Dropping support for 1.0.x line would eliminate the need to maintain one
> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> very closely aligned. I just up merged metrics_v2 against 1.x-branch
> after a while, and there were no conflicts.
> 
> That would give us a little more bandwidth to focus on 2.0 and needed bug
> fixes to the 1.x line like some of the issues raised with
> storm-kafka-client. We could even start releasing alpha/beta versions of
> 2.0 in parallel to the steps above.
> 
> Any thoughts on that approach?
> 
> -Taylor
> 
> 
> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <ka...@gmail.com>> wrote:
> 
> Yes I prefer option 1, but it might depend on the progress of metrics
> V2.
> If it can be done within predictable near future I'm OK to pick option
> 2,
> but if not, we may be better to focus releasing 2.0.0 and make it
> really
> happen.
> 
> Whichever we go, I feel it's time to track remaining work on Storm
> 2.0.0. I
> found some bugs on master branch so filed issues, and we've remaining
> port
> work (UI and logviewer). We've some other improvements target for
> 2.0.0:
> worker redesign, beam integration, and so on, and we don't track its
> progress at all. I don't think we should wait for features which
> progress
> is not transparent (in other words we don't know when it will be
> finished).
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>>님이 작성:
> 
> Bobby/Jungtaek,
> 
> Are you saying you want to forego the 1.2 “metrics_v2” release and
> include
> it only in 2.0? (I ask because that work is already based on
> 1.x-branch,
> and forward-porting it to master is relatively simple.) I’d kind of
> like
> that work go out soon.
> 
> If we go with option 1, I would want to see a 2.0 release (even if
> it’s a
> “beta” or “preview) before putting the 1.x line into maintenance mode.
> 
> -Taylor
> 
> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> 
> wrote:
> 
> I see 2 ways to address this.
> 1) We put the 1.x line into maintenance mode like with 0.10.  We
> don't
> backport anything except bug fixes.2) We backport a lot of the
> backwards
> compatible changes from 2.x to 1.x.
> My personal preference is 1.  It makes it clear the direction we
> want to
> go in.  The biggest issue is that we probably would want to do a 2.x
> release sooner rather then later.  Even if we don't get all of the
> features
> that people want, if we just get a release out we can add in new
> features
> if they are backwards compatible, or we can create a 3.x line that
> would
> have the breaking changes in it.
> 
> - Bobby
> 
> 
> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> kabhwan@gmail.com<ma...@gmail.com>> wrote:
> 
> I'd like to bump this again instead of initiating new discussion
> thread.
> 
> I had having hard time to create and apply pull requests for both
> master
> and 1.x-branch and that's really painful and sometimes blocker for
> me to
> do
> merge step.
> Two branches are heavily diverged more than between 0.10 and 1.0.0,
> even
> IDE can't switch between the branch smoothly. We didn't even address
> checkstyle issue yet, but after addressing, it could be "completely"
> diverged. JDK version is another major issue, since the pull requests
> targeted for master branch are not checked against JDK 7, and some of
> them
> make some issues regarding JDK version while porting back.
> 
> So personally I really would like to see the plan for 1.x version
> line
> changed - skipping any minor releases including 1.2.0 - and have epic
> issue
> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> proposed plan was having 2.0.0 directly after 1.0.0)
> 
> Would like to hear everyone's opinions. If we have consensus to not
> having
> any minor releases for 1.x version line, I will not port back
> non-bugfix
> pull requests to 1.x-branch, and guide contributors to create pull
> requests
> against master branch, not 1.x version line.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> avermeerbergen@gmail.com<ma...@gmail.com>>님이
> 작성:
> 
> +1 for Roshan's suggestion : in our Storm 1.x based supervision
> system,
> we're very interested anything that can provide better throughput.
> 
> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>>:
> 
> For 2.0 beta … it would be good to incorporate some of the Worker
> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> be
> delivered sooner and my in-progress implementation suggests that it
> will
> yield substantial latency improvements. The 2.0 beta phase would
> really
> help kick the tires on the revised messaging system and the
> performance
> improvements will also be a good incentive for trying out the 2.0
> line.
> 
> I notice multiple other bottlenecks that are holding back
> throughput a
> lot, which can be addressed in a subsequent 2.x minor release.
> -roshan
> 
> 
> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com>> wrote:
> 
>    I also would love to see metrics V2 code sooner or later too.
> If we
> can get
>    it before releasing 2.0.0 that will be great, and then maybe we
> could
> just
>    move toward to 2.0.0, not adding any improvements to 1.x version
> line.
>    (And that's what I would want to.)
> 
>    If we would really want to have 1.2.0, I suggest that we make
> the
> 1.1.1
>    version correct right now rather than after releasing 1.1.1. We
> also
> merged
>    non-bugfix things to 1.x-branch but that's not what users
> expect. I
> agree
>    that work may be painful, but anyway need to do it.
> 
>    - Jungtaek Lim (HeartSaVioR)
> 
>    2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> <ev...@yahoo-inc.com.invalid>
> 님이
> 작성:
> 
> I would love to see the metrics V2 code come out sooner rather
> than
> later.  +1.
> My biggest blocker for a 1.x release is
> https://github.com/apache/storm/pull/2142  Even though the pull
> request
> says it is minor it showed that we messed up pushing back some
> changes for
> pacemaker to open source (the code does not run at all which for
> me
> is a
> blocker) and I really want to get that fully fixed/tested before
> another
> release.
> As for 2.x I think we are very close to being able to so a 2.x
> alpha
> release.  I would like to see metrics V2 merged in simply because
> it
> is a
> big change for user facing APIs.  But after that I would love to
> see
> us
> starting to push forward on getting that out.
> 
> 
> - Bobby
> 
> 
> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> ptgoetz@gmail.com<ma...@gmail.com>> wrote:
> 
> I’d like to bump this thread and start a discussion around our
> next
> release. Here are my thoughts.
> 
> There are a number of important fixes in 1.x-branch so I’d like
> to
> consider releasing 1.1.1 soon. I’d appreciate input on any open
> issues that
> should be resolved for that release.
> 
> I’d like us to consider releasing the metrics improvements in
> STORM-2153
> [1] as version 1.2.0. That work is in the metrics_v2 feature
> branch
> right
> now and would need to be reviewed and merged. That work is
> against
> the
> 1.x-branch right now. I would recommend porting it to master
> *after*
> the
> review/merge since there will likely be changes as a result of
> the
> review.
> 
> Maybe related to or not, but would we want to create a new
> branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> 
> 
> 
> If wee agree to the above, I would say yes. After the 1.1.1
> release,
> we
> could create a 1.1.x-branch that would be the maintenance/release
> branch
> for that version line. 1.x-branch would then become the target
> for
> the
> 1.2.0 release.
> 
> There are a few fixes in the 0.10.x branch that probably warrant
> a
> release. After that we may want to back away from that version
> line
> a bit
> so we can focus more on newer versions.
> 
> In the past, we’ve shied away form doing “beta” releases, but I’m
> wondering if we might want to revisit that for the 2.0 release —
> the
> idea
> being that it would give early adopter users a chance to kick the
> tires on
> what’s coming in 2.0 and provide feedback, find bugs, etc. to
> help
> make the
> final release more solid. I’m on the fence here and could go
> either
> way.
> 
> I’d appreciate any input others may have.
> 
> 
> Thanks,
> 
> -Taylor
> 
> 
> [1] https://issues.apache.org/jira/browse/STORM-2153
> 
> 
> 
> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>>
> wrote:
> 
> Maybe related to or not, but would we want to create a new
> branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> 
> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> so
> hence
> the
> question.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> hlouro@hortonworks.com<ma...@hortonworks.com>>님이
> 작성:
> 
> +1 for finishing the porting to Java ahead of anything else -
> it
> will
> be a
> significant milestone. I have a JIRA assigned concerning to
> the
> porting. I
> will work on it for the 2.0 release.
> 
> it’s a priority to guarantee no performance regressions. As
> part
> of this
> endeavor, explore an automated (or easy) way to run and assert
> major
> performance benchmarks. Ideally any contributor should be able
> to
> fairly
> easily test the impact of changes under certain performance
> test
> scenarios.
> 
> Beam Runner work should take into account the impact of
> incorporating
> new
> JStorm features and Storm Worker Redesign<
> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> efficient
> to
> start doing it, to  find out that it will have to chance in
> face
> of
> Storm
> and worker redesign. That is, it should be done after it’s
> building
> blocks
> are stable.
> 
> Thanks,
> Hugo
> 
> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> arunm@apache.org<ma...@apache.org>
> <mailto:
> arunm@apache.org>> wrote:
> 
> +1 to release with the porting completed. I think its mainly
> the
> UI
> server
> and log viewer that’s pending.
> 
> We can start doing the regression and performance tests for
> whatever is
> already ported.
> 
> If anyone is running the master branch in their pre-prod /
> prod
> environments, it will be good to know and give us more
> confidence.
> 
> The other features can be added in follow up releases.
> 
> Regards,
> Arun
> 
> 
> On 3/24/17, 11:47 AM, "Satish Duggana" <
> satish.duggana@gmail.com
> <mailto:
> satish.duggana@gmail.com>> wrote:
> 
> +1 to have 2.0 with porting and performance(it should be at
> least
> as
> good
> as 1.x release) issues addressed
> 
> We can target other tasks(mentioned by Taylor and Jungtaek)
> for
> 2.x-branch.
> 
> 
> Exactly-once support:
> While thinking through the exactlyonce support design, it is
> realized
> better to avoid acking tuples and implement exactly once by
> snapshotting
> barriers. It seems JStorm folks followed similar design, they
> claim it
> gives better performance. This feature is essential for beam
> runner and
> we
> can decide on respective approaches though.
> 
> Beam Runner
> Lets hold on this for now and keep it in Storm till 2.x. We
> should avoid
> having a minimal beam runner in haste. It is better to address
> STORM-2284,
> exactly-once and other windowing enhancements to enable beam
> runner.
> 
> JStorm
> Agree with Jungtaek on looking at the latest JStorm and
> align/scope with
> the features for 2.x.
> 
> STORM-2284
> We may want to look at JStorm worker before working on
> respective
> components in this epic to pull appropriate enhancements.
> 
> YARN/MESOS
> Supporting Storm on YARN/Mesos for 2.x.
> 
> Thanks,
> Satish.
> 
> 
> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> kabhwan@gmail.com
> <mailto:
> kabhwan@gmail.com>> wrote:
> 
> First of all, +1 to complete only port work and do sanity
> check
> (including
> performance regression), and release.
> 
> If we can get STORM-2284 within deterministic time frame (say
> 2~3
> months)
> that should be great, but if not I'd in favor of postponing
> that
> to
> later
> 2.x release.
> 
> JStorm released their new versions after code donation. So
> there're more
> things we could get ideas from, or even adopt from.
> https://github.com/alibaba/jstorm/blob/master/history.md
> As you noticed from release note link, we also need to update
> phase 2
> since
> they already changed what we're planning to do in phase 2. For
> example,
> they changed backpressure to end-to-end, and changed to use
> snapshot
> rather
> than acker.
> May be sure, JStorm pulled many features from today's Storm,
> like
> Flux,
> Windowing, more shuffle groupings, log search, log level
> change,
> and so
> on.
> 
> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> is
> due to
> the
> limitation of Spout lifecycle (all the things are done in
> single
> thread),
> and STORM-1358 <
> https://issues.apache.org/jira/browse/STORM-1358
> (JStorm's
> multi-thread Spout) can remedy this (despite that Spout
> implementation
> may
> need to guarantee thread-safety later). It's not a just
> improvement but
> close to design concern so would like to address sooner than
> other
> things
> in phase 2.
> 
> For Storm SQL side, I've lost progress but major work would be
> adopting
> group by with windowing. It was not available from Calcite but
> will be
> available at next release (1.12.0).
> I've filed this to STORM-2405
> <https://issues.apache.org/jira/browse/STORM-2405>, but
> windowing &
> micro
> batch is not intuitive, so I would like to change the
> underlying
> API to
> stream API in SQL. Also filed this to STORM-2406
> <https://issues.apache.org/jira/browse/STORM-2406>.
> 
> Just 2 cents btw, hopefully I would like to see metrics V2
> sooner
> since
> we
> lost metrics even when doing normal operation like restarting
> worker,
> rebalancing, and so on. Eventually we need to fight with
> dynamic
> scaling,
> and then metrics will be broken often.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> storm@harsha.io
> 님이
> 작성:
> 
> Storm 2.0 migration to java in itself is a big win and would
> attract
> wider
> community and adoption. So my vote would be to resolve the
> first
> 3 items
> to
> get a release out.
> All the other featured mentioned are great to have but
> shouldn't
> be
> blockers for 2.0 release.
> 
> -Harsha
> 
> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> ptgoetz@gmail.com>
> wrote:
> 
> With the 1.1.0 release nearing completion, I’d like to turn
> our
> attention
> to 2.0 and develop a plan for what features, etc. to include.
> 
> The following 3 are what I feel are the minimum for a 2.0
> release.
> These
> could likely be resolved relatively quickly:
> 
> * Performance — I’ve not benchmarked the master branch vs.
> 1.0.x
> or
> 1.1.x
> in a while, but I feel it will be important to make sure there
> are no
> performance regressions, and would hope that we actually have
> a
> performance
> improvement over previous versions. To that end (e.g. if there
> is
> in
> fact a
> performance regression), the proposals that Roshan Naik put
> together
> for
> revising the threading and execution model (STORM-2307) and
> replacing
> Disruptor with JCTools (STORM-2306) warrant review and
> consideration.
> See
> also STORM-2284 which is the parent JIRA.
> 
> * Finish porting Storm UI to java (STORM-1311)
> 
> * Finish porting log viewer to java (STORM-1280)
> 
> The following are items that are nice to have in 2.0, but I
> don’t
> feel
> are
> absolutely necessary for an initial 2.0 release:
> 
> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> because
> it’s
> relevant) — Initially there seemed to be a lot of interest in
> this, but
> that seems to have trailed off. I spoke with some Beam
> developers
> and
> there
> seems to be interest from that community as well. Do we want
> to
> move
> that
> effort to the Beam community, or keep it here? Moving it to
> the
> Beam
> community might lead to better collaboration between projects.
> 
> * Bounded Spouts (needed for Beam Runner implementation) —
> Currently
> spouts are unbounded, there no end to the stream. Beam has the
> concept
> of
> bounded sources (roughly analogous to batch processing). To
> support
> that,
> we would need to implement a similar concept in Storm. One
> benefit of
> such
> a feature would be the ability to handle both bounded and
> unbounded
> workflows in Storm.
> 
> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> behind
> this
> effort. What improvements do you envision for 2.0?
> 
> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> targeting this
> for 1.2.0, but it’s designed to be easily portable to
> master/2.0.
> 
> * JStorm Migration — Original outline can be found here [1].
> Note
> a lot
> of
> the associated JIRAs below are assigned, but there hasn’t been
> any
> recent
> activity or pull requests, we should probably consider them
> unassigned
> and
> up for grabs.:
> 
> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> been the
> bane of a lot of Storm users almost since day one. We have
> largely
> addressed it by shading/relocating dependencies. It would be
> great to
> see
> this addressed once and for all.
> 
> * JStorm back pressure implementation (STORM-1324) — The
> current
> back
> pressure implementation leaves a bit to be desired, and the
> JStorm
> approach
> looks promising, though it also depends on the JStorm concept
> of
> “topology
> master” (STORM-1323), which may have some implications
> regarding
> security.
> 
> * Dynamic Topology Updates (STORM-1335) — This would provide a
> command
> to
> update topology jars and configuration without stopping the
> topology,
> and
> is well suited to leverage the blobstore. The restart command
> (that can
> also update the topology configuration) also looks compelling
> (STORM-1334).
> 
> * Additional Scheduler Implementations (STORM-1320)
> 
> * Additional Grouping Implementations (STORM-1328)
> 
> 
> As always I’m open to any opinions and suggestions.
> 
> -Taylor
> 
> [1]
> 
> https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=61328109
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Hugo Da Cruz Louro <hl...@hortonworks.com>.
I still need to go over the entire discussion thread in more detail, but one thing I would like to bring up right way is the proposal to DEPRECATE, and eventually remove, the KafkaSpout with the old Kafka Consumer APIs. The storm-kafka-client KafkaSpout is getting stabilized, and I think we are all in agreement that the storm-kafka KafkaSpout has presented continuous maintainability problems with some fixes that got in not being backwards compatible.

I am pretty confident how things are looking at this point for the KafkaSpout. The Trident Kafka Spout is likely in between alpha and beta, and that should be taken into account. I just recently submitted a PR<https://github.com/apache/storm/pull/2174> with some improvements to the Trident Kafka Spout (including the refactoring done to support manual partition assignment), and there are some customers using it in pre-production. However, it definitely would benefit from some more testing.

Thanks,
Hugo

On Jun 28, 2017, at 7:48 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:

+1.
If the 1.1 and 1.2 lines start to become difficult to maintain we can look at putting them in maintenance mode too once we have a 2.x release.
I am a little nervous about merging a new feature into 1.x branch without first going to master, but I hope that it will not be too much work to port it to master, and I trust the devs on that branch to do the right thing.
On a related note we have not done much with feature branches before so I am not sure what we want to do about merging in the new metrics API branch to 1.x.  I know for me I have not had time to keep up with the development work going on there.  I would at least like to have a pull request put up for review before we merge it in.  This would fit with our current bylaws that do not mention feature branches.  If all of the changes have already followed the review process then technically I think it is OK to just merge it in, but I still would like to take some time to look at the changes, and especially the new APIs.

- Bobby


On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <ka...@gmail.com>> wrote:

That's great news that metrics work is ready!

I'm +1 to Taylor's proposal, but in order to respect semantic versioning, I
propose some modifications from Taylor's proposal:

- create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back only
bug fixes to the 1.1.x-branch
- change the target version of 1.x-branch to 1.2.0-SNAPSHOT

If we also agree above, I would like to volunteer the back-port work.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 28일 (수) 오전 10:09, Harsha <st...@harsha.io>>님이 작성:

+1 for above stated approach on releasing 1.2.0 with metrics
-Harsha

On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
The work on metrics is ready for a pull request to 1.x-branch from the
feature branch. I’ve held off because we haven’t reached consensus on a
path forward with the 1.x release lines .

I’d like to propose the following for the 1.x line:

1. Create a branch for 1.2 so we have a branch to review the metrics
stuff.
2. Release 1.1.1
3. Review/merge metrics work. Port metrics to master.
4. Release 1.2.0
5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
(we would only support 1.2.x and 1.1.x which are very closely aligned).

Dropping support for 1.0.x line would eliminate the need to maintain one
of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
very closely aligned. I just up merged metrics_v2 against 1.x-branch
after a while, and there were no conflicts.

That would give us a little more bandwidth to focus on 2.0 and needed bug
fixes to the 1.x line like some of the issues raised with
storm-kafka-client. We could even start releasing alpha/beta versions of
2.0 in parallel to the steps above.

Any thoughts on that approach?

-Taylor


On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <ka...@gmail.com>> wrote:

Yes I prefer option 1, but it might depend on the progress of metrics
V2.
If it can be done within predictable near future I'm OK to pick option
2,
but if not, we may be better to focus releasing 2.0.0 and make it
really
happen.

Whichever we go, I feel it's time to track remaining work on Storm
2.0.0. I
found some bugs on master branch so filed issues, and we've remaining
port
work (UI and logviewer). We've some other improvements target for
2.0.0:
worker redesign, beam integration, and so on, and we don't track its
progress at all. I don't think we should wait for features which
progress
is not transparent (in other words we don't know when it will be
finished).

- Jungtaek Lim (HeartSaVioR)

2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>>님이 작성:

Bobby/Jungtaek,

Are you saying you want to forego the 1.2 “metrics_v2” release and
include
it only in 2.0? (I ask because that work is already based on
1.x-branch,
and forward-porting it to master is relatively simple.) I’d kind of
like
that work go out soon.

If we go with option 1, I would want to see a 2.0 release (even if
it’s a
“beta” or “preview) before putting the 1.x line into maintenance mode.

-Taylor

On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>

wrote:

I see 2 ways to address this.
1) We put the 1.x line into maintenance mode like with 0.10.  We
don't
backport anything except bug fixes.2) We backport a lot of the
backwards
compatible changes from 2.x to 1.x.
My personal preference is 1.  It makes it clear the direction we
want to
go in.  The biggest issue is that we probably would want to do a 2.x
release sooner rather then later.  Even if we don't get all of the
features
that people want, if we just get a release out we can add in new
features
if they are backwards compatible, or we can create a 3.x line that
would
have the breaking changes in it.

- Bobby


On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
kabhwan@gmail.com<ma...@gmail.com>> wrote:

I'd like to bump this again instead of initiating new discussion
thread.

I had having hard time to create and apply pull requests for both
master
and 1.x-branch and that's really painful and sometimes blocker for
me to
do
merge step.
Two branches are heavily diverged more than between 0.10 and 1.0.0,
even
IDE can't switch between the branch smoothly. We didn't even address
checkstyle issue yet, but after addressing, it could be "completely"
diverged. JDK version is another major issue, since the pull requests
targeted for master branch are not checked against JDK 7, and some of
them
make some issues regarding JDK version while porting back.

So personally I really would like to see the plan for 1.x version
line
changed - skipping any minor releases including 1.2.0 - and have epic
issue
for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
proposed plan was having 2.0.0 directly after 1.0.0)

Would like to hear everyone's opinions. If we have consensus to not
having
any minor releases for 1.x version line, I will not port back
non-bugfix
pull requests to 1.x-branch, and guide contributors to create pull
requests
against master branch, not 1.x version line.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
avermeerbergen@gmail.com<ma...@gmail.com>>님이
작성:

+1 for Roshan's suggestion : in our Storm 1.x based supervision
system,
we're very interested anything that can provide better throughput.

2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>>:

For 2.0 beta … it would be good to incorporate some of the Worker
improvements (STORM-2284)  IMO. Changes to messaging subsystem can
be
delivered sooner and my in-progress implementation suggests that it
will
yield substantial latency improvements. The 2.0 beta phase would
really
help kick the tires on the revised messaging system and the
performance
improvements will also be a good incentive for trying out the 2.0
line.

I notice multiple other bottlenecks that are holding back
throughput a
lot, which can be addressed in a subsequent 2.x minor release.
-roshan


On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com>> wrote:

    I also would love to see metrics V2 code sooner or later too.
If we
can get
    it before releasing 2.0.0 that will be great, and then maybe we
could
just
    move toward to 2.0.0, not adding any improvements to 1.x version
line.
    (And that's what I would want to.)

    If we would really want to have 1.2.0, I suggest that we make
the
1.1.1
    version correct right now rather than after releasing 1.1.1. We
also
merged
    non-bugfix things to 1.x-branch but that's not what users
expect. I
agree
    that work may be painful, but anyway need to do it.

    - Jungtaek Lim (HeartSaVioR)

    2017년 6월 3일 (토) 오전 3:49, Bobby Evans
<ev...@yahoo-inc.com.invalid>
님이
작성:

I would love to see the metrics V2 code come out sooner rather
than
later.  +1.
My biggest blocker for a 1.x release is
https://github.com/apache/storm/pull/2142  Even though the pull
request
says it is minor it showed that we messed up pushing back some
changes for
pacemaker to open source (the code does not run at all which for
me
is a
blocker) and I really want to get that fully fixed/tested before
another
release.
As for 2.x I think we are very close to being able to so a 2.x
alpha
release.  I would like to see metrics V2 merged in simply because
it
is a
big change for user facing APIs.  But after that I would love to
see
us
starting to push forward on getting that out.


- Bobby


On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
ptgoetz@gmail.com<ma...@gmail.com>> wrote:

I’d like to bump this thread and start a discussion around our
next
release. Here are my thoughts.

There are a number of important fixes in 1.x-branch so I’d like
to
consider releasing 1.1.1 soon. I’d appreciate input on any open
issues that
should be resolved for that release.

I’d like us to consider releasing the metrics improvements in
STORM-2153
[1] as version 1.2.0. That work is in the metrics_v2 feature
branch
right
now and would need to be reviewed and merged. That work is
against
the
1.x-branch right now. I would recommend porting it to master
*after*
the
review/merge since there will likely be changes as a result of
the
review.

Maybe related to or not, but would we want to create a new
branch
"1.1.x-branch", and make "1.x-branch" target for 1.2?



If wee agree to the above, I would say yes. After the 1.1.1
release,
we
could create a 1.1.x-branch that would be the maintenance/release
branch
for that version line. 1.x-branch would then become the target
for
the
1.2.0 release.

There are a few fixes in the 0.10.x branch that probably warrant
a
release. After that we may want to back away from that version
line
a bit
so we can focus more on newer versions.

In the past, we’ve shied away form doing “beta” releases, but I’m
wondering if we might want to revisit that for the 2.0 release —
the
idea
being that it would give early adopter users a chance to kick the
tires on
what’s coming in 2.0 and provide feedback, find bugs, etc. to
help
make the
final release more solid. I’m on the fence here and could go
either
way.

I’d appreciate any input others may have.


Thanks,

-Taylor


[1] https://issues.apache.org/jira/browse/STORM-2153



On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>>
wrote:

Maybe related to or not, but would we want to create a new
branch
"1.1.x-branch", and make "1.x-branch" target for 1.2?

I'm not clear we don't release 1.2 for moving toward to 2.0.0,
so
hence
the
question.

- Jungtaek Lim (HeartSaVioR)

2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
hlouro@hortonworks.com<ma...@hortonworks.com>>님이
작성:

+1 for finishing the porting to Java ahead of anything else -
it
will
be a
significant milestone. I have a JIRA assigned concerning to
the
porting. I
will work on it for the 2.0 release.

it’s a priority to guarantee no performance regressions. As
part
of this
endeavor, explore an automated (or easy) way to run and assert
major
performance benchmarks. Ideally any contributor should be able
to
fairly
easily test the impact of changes under certain performance
test
scenarios.

Beam Runner work should take into account the impact of
incorporating
new
JStorm features and Storm Worker Redesign<
https://issues.apache.org/jira/browse/STORM-2284>. Not very
efficient
to
start doing it, to  find out that it will have to chance in
face
of
Storm
and worker redesign. That is, it should be done after it’s
building
blocks
are stable.

Thanks,
Hugo

On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
arunm@apache.org<ma...@apache.org>
<mailto:
arunm@apache.org>> wrote:

+1 to release with the porting completed. I think its mainly
the
UI
server
and log viewer that’s pending.

We can start doing the regression and performance tests for
whatever is
already ported.

If anyone is running the master branch in their pre-prod /
prod
environments, it will be good to know and give us more
confidence.

The other features can be added in follow up releases.

Regards,
Arun


On 3/24/17, 11:47 AM, "Satish Duggana" <
satish.duggana@gmail.com
<mailto:
satish.duggana@gmail.com>> wrote:

+1 to have 2.0 with porting and performance(it should be at
least
as
good
as 1.x release) issues addressed

We can target other tasks(mentioned by Taylor and Jungtaek)
for
2.x-branch.


Exactly-once support:
While thinking through the exactlyonce support design, it is
realized
better to avoid acking tuples and implement exactly once by
snapshotting
barriers. It seems JStorm folks followed similar design, they
claim it
gives better performance. This feature is essential for beam
runner and
we
can decide on respective approaches though.

Beam Runner
Lets hold on this for now and keep it in Storm till 2.x. We
should avoid
having a minimal beam runner in haste. It is better to address
STORM-2284,
exactly-once and other windowing enhancements to enable beam
runner.

JStorm
Agree with Jungtaek on looking at the latest JStorm and
align/scope with
the features for 2.x.

STORM-2284
We may want to look at JStorm worker before working on
respective
components in this epic to pull appropriate enhancements.

YARN/MESOS
Supporting Storm on YARN/Mesos for 2.x.

Thanks,
Satish.


On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
kabhwan@gmail.com
<mailto:
kabhwan@gmail.com>> wrote:

First of all, +1 to complete only port work and do sanity
check
(including
performance regression), and release.

If we can get STORM-2284 within deterministic time frame (say
2~3
months)
that should be great, but if not I'd in favor of postponing
that
to
later
2.x release.

JStorm released their new versions after code donation. So
there're more
things we could get ideas from, or even adopt from.
https://github.com/alibaba/jstorm/blob/master/history.md
As you noticed from release note link, we also need to update
phase 2
since
they already changed what we're planning to do in phase 2. For
example,
they changed backpressure to end-to-end, and changed to use
snapshot
rather
than acker.
May be sure, JStorm pulled many features from today's Storm,
like
Flux,
Windowing, more shuffle groupings, log search, log level
change,
and so
on.

STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
is
due to
the
limitation of Spout lifecycle (all the things are done in
single
thread),
and STORM-1358 <
https://issues.apache.org/jira/browse/STORM-1358
(JStorm's
multi-thread Spout) can remedy this (despite that Spout
implementation
may
need to guarantee thread-safety later). It's not a just
improvement but
close to design concern so would like to address sooner than
other
things
in phase 2.

For Storm SQL side, I've lost progress but major work would be
adopting
group by with windowing. It was not available from Calcite but
will be
available at next release (1.12.0).
I've filed this to STORM-2405
<https://issues.apache.org/jira/browse/STORM-2405>, but
windowing &
micro
batch is not intuitive, so I would like to change the
underlying
API to
stream API in SQL. Also filed this to STORM-2406
<https://issues.apache.org/jira/browse/STORM-2406>.

Just 2 cents btw, hopefully I would like to see metrics V2
sooner
since
we
lost metrics even when doing normal operation like restarting
worker,
rebalancing, and so on. Eventually we need to fight with
dynamic
scaling,
and then metrics will be broken often.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
storm@harsha.io
님이
작성:

Storm 2.0 migration to java in itself is a big win and would
attract
wider
community and adoption. So my vote would be to resolve the
first
3 items
to
get a release out.
All the other featured mentioned are great to have but
shouldn't
be
blockers for 2.0 release.

-Harsha

On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
ptgoetz@gmail.com>
wrote:

With the 1.1.0 release nearing completion, I’d like to turn
our
attention
to 2.0 and develop a plan for what features, etc. to include.

The following 3 are what I feel are the minimum for a 2.0
release.
These
could likely be resolved relatively quickly:

* Performance — I’ve not benchmarked the master branch vs.
1.0.x
or
1.1.x
in a while, but I feel it will be important to make sure there
are no
performance regressions, and would hope that we actually have
a
performance
improvement over previous versions. To that end (e.g. if there
is
in
fact a
performance regression), the proposals that Roshan Naik put
together
for
revising the threading and execution model (STORM-2307) and
replacing
Disruptor with JCTools (STORM-2306) warrant review and
consideration.
See
also STORM-2284 which is the parent JIRA.

* Finish porting Storm UI to java (STORM-1311)

* Finish porting log viewer to java (STORM-1280)

The following are items that are nice to have in 2.0, but I
don’t
feel
are
absolutely necessary for an initial 2.0 release:

* Beam Runner (I wouldn’t tie this to 2.0, mentioning it
because
it’s
relevant) — Initially there seemed to be a lot of interest in
this, but
that seems to have trailed off. I spoke with some Beam
developers
and
there
seems to be interest from that community as well. Do we want
to
move
that
effort to the Beam community, or keep it here? Moving it to
the
Beam
community might lead to better collaboration between projects.

* Bounded Spouts (needed for Beam Runner implementation) —
Currently
spouts are unbounded, there no end to the stream. Beam has the
concept
of
bounded sources (roughly analogous to batch processing). To
support
that,
we would need to implement a similar concept in Storm. One
benefit of
such
a feature would be the ability to handle both bounded and
unbounded
workflows in Storm.

* Storm-SQL — Jungtaek/Xin: You have been the primary drivers
behind
this
effort. What improvements do you envision for 2.0?

* Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
targeting this
for 1.2.0, but it’s designed to be easily portable to
master/2.0.

* JStorm Migration — Original outline can be found here [1].
Note
a lot
of
the associated JIRAs below are assigned, but there hasn’t been
any
recent
activity or pull requests, we should probably consider them
unassigned
and
up for grabs.:

* Worker Classloader Isolation (STORM-1338) — Lack of this has
been the
bane of a lot of Storm users almost since day one. We have
largely
addressed it by shading/relocating dependencies. It would be
great to
see
this addressed once and for all.

* JStorm back pressure implementation (STORM-1324) — The
current
back
pressure implementation leaves a bit to be desired, and the
JStorm
approach
looks promising, though it also depends on the JStorm concept
of
“topology
master” (STORM-1323), which may have some implications
regarding
security.

* Dynamic Topology Updates (STORM-1335) — This would provide a
command
to
update topology jars and configuration without stopping the
topology,
and
is well suited to leverage the blobstore. The restart command
(that can
also update the topology configuration) also looks compelling
(STORM-1334).

* Additional Scheduler Implementations (STORM-1320)

* Additional Grouping Implementations (STORM-1328)


As always I’m open to any opinions and suggestions.

-Taylor

[1]

https://cwiki.apache.org/confluence/pages/viewpage.
action?pageId=61328109



















Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
+1.
If the 1.1 and 1.2 lines start to become difficult to maintain we can look at putting them in maintenance mode too once we have a 2.x release.
I am a little nervous about merging a new feature into 1.x branch without first going to master, but I hope that it will not be too much work to port it to master, and I trust the devs on that branch to do the right thing.
On a related note we have not done much with feature branches before so I am not sure what we want to do about merging in the new metrics API branch to 1.x.  I know for me I have not had time to keep up with the development work going on there.  I would at least like to have a pull request put up for review before we merge it in.  This would fit with our current bylaws that do not mention feature branches.  If all of the changes have already followed the review process then technically I think it is OK to just merge it in, but I still would like to take some time to look at the changes, and especially the new APIs.

- Bobby


On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <ka...@gmail.com> wrote:

That's great news that metrics work is ready!

I'm +1 to Taylor's proposal, but in order to respect semantic versioning, I
propose some modifications from Taylor's proposal:

- create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back only
bug fixes to the 1.1.x-branch
- change the target version of 1.x-branch to 1.2.0-SNAPSHOT

If we also agree above, I would like to volunteer the back-port work.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 28일 (수) 오전 10:09, Harsha <st...@harsha.io>님이 작성:

> +1 for above stated approach on releasing 1.2.0 with metrics
> -Harsha
>
> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> > The work on metrics is ready for a pull request to 1.x-branch from the
> > feature branch. I’ve held off because we haven’t reached consensus on a
> > path forward with the 1.x release lines .
> >
> > I’d like to propose the following for the 1.x line:
> >
> > 1. Create a branch for 1.2 so we have a branch to review the metrics
> > stuff.
> > 2. Release 1.1.1
> > 3. Review/merge metrics work. Port metrics to master.
> > 4. Release 1.2.0
> > 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
> > (we would only support 1.2.x and 1.1.x which are very closely aligned).
> >
> > Dropping support for 1.0.x line would eliminate the need to maintain one
> > of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> > very closely aligned. I just up merged metrics_v2 against 1.x-branch
> > after a while, and there were no conflicts.
> >
> > That would give us a little more bandwidth to focus on 2.0 and needed bug
> > fixes to the 1.x line like some of the issues raised with
> > storm-kafka-client. We could even start releasing alpha/beta versions of
> > 2.0 in parallel to the steps above.
> >
> > Any thoughts on that approach?
> >
> > -Taylor
> >
> >
> > > On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> > >
> > > Yes I prefer option 1, but it might depend on the progress of metrics
> V2.
> > > If it can be done within predictable near future I'm OK to pick option
> 2,
> > > but if not, we may be better to focus releasing 2.0.0 and make it
> really
> > > happen.
> > >
> > > Whichever we go, I feel it's time to track remaining work on Storm
> 2.0.0. I
> > > found some bugs on master branch so filed issues, and we've remaining
> port
> > > work (UI and logviewer). We've some other improvements target for
> 2.0.0:
> > > worker redesign, beam integration, and so on, and we don't track its
> > > progress at all. I don't think we should wait for features which
> progress
> > > is not transparent (in other words we don't know when it will be
> finished).
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> > >
> > >> Bobby/Jungtaek,
> > >>
> > >> Are you saying you want to forego the 1.2 “metrics_v2” release and
> include
> > >> it only in 2.0? (I ask because that work is already based on
> 1.x-branch,
> > >> and forward-porting it to master is relatively simple.) I’d kind of
> like
> > >> that work go out soon.
> > >>
> > >> If we go with option 1, I would want to see a 2.0 release (even if
> it’s a
> > >> “beta” or “preview) before putting the 1.x line into maintenance mode.
> > >>
> > >> -Taylor
> > >>
> > >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID
> >
> > >> wrote:
> > >>>
> > >>> I see 2 ways to address this.
> > >>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
> don't
> > >> backport anything except bug fixes.2) We backport a lot of the
> backwards
> > >> compatible changes from 2.x to 1.x.
> > >>> My personal preference is 1.  It makes it clear the direction we
> want to
> > >> go in.  The biggest issue is that we probably would want to do a 2.x
> > >> release sooner rather then later.  Even if we don't get all of the
> features
> > >> that people want, if we just get a release out we can add in new
> features
> > >> if they are backwards compatible, or we can create a 3.x line that
> would
> > >> have the breaking changes in it.
> > >>>
> > >>> - Bobby
> > >>>
> > >>>
> > >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> > >> kabhwan@gmail.com> wrote:
> > >>>
> > >>> I'd like to bump this again instead of initiating new discussion
> thread.
> > >>>
> > >>> I had having hard time to create and apply pull requests for both
> master
> > >>> and 1.x-branch and that's really painful and sometimes blocker for
> me to
> > >> do
> > >>> merge step.
> > >>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
> even
> > >>> IDE can't switch between the branch smoothly. We didn't even address
> > >>> checkstyle issue yet, but after addressing, it could be "completely"
> > >>> diverged. JDK version is another major issue, since the pull requests
> > >>> targeted for master branch are not checked against JDK 7, and some of
> > >> them
> > >>> make some issues regarding JDK version while porting back.
> > >>>
> > >>> So personally I really would like to see the plan for 1.x version
> line
> > >>> changed - skipping any minor releases including 1.2.0 - and have epic
> > >> issue
> > >>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> > >>> proposed plan was having 2.0.0 directly after 1.0.0)
> > >>>
> > >>> Would like to hear everyone's opinions. If we have consensus to not
> > >> having
> > >>> any minor releases for 1.x version line, I will not port back
> non-bugfix
> > >>> pull requests to 1.x-branch, and guide contributors to create pull
> > >> requests
> > >>> against master branch, not 1.x version line.
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> > >> avermeerbergen@gmail.com>님이
> > >>> 작성:
> > >>>
> > >>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
> system,
> > >>>> we're very interested anything that can provide better throughput.
> > >>>>
> > >>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
> > >>>>
> > >>>>> For 2.0 beta … it would be good to incorporate some of the Worker
> > >>>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> be
> > >>>>> delivered sooner and my in-progress implementation suggests that it
> > >> will
> > >>>>> yield substantial latency improvements. The 2.0 beta phase would
> really
> > >>>>> help kick the tires on the revised messaging system and the
> performance
> > >>>>> improvements will also be a good incentive for trying out the 2.0
> line.
> > >>>>>
> > >>>>> I notice multiple other bottlenecks that are holding back
> throughput a
> > >>>>> lot, which can be addressed in a subsequent 2.x minor release.
> > >>>>> -roshan
> > >>>>>
> > >>>>>
> > >>>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> > >>>>>
> > >>>>>    I also would love to see metrics V2 code sooner or later too.
> If we
> > >>>>> can get
> > >>>>>    it before releasing 2.0.0 that will be great, and then maybe we
> > >> could
> > >>>>> just
> > >>>>>    move toward to 2.0.0, not adding any improvements to 1.x version
> > >>>> line.
> > >>>>>    (And that's what I would want to.)
> > >>>>>
> > >>>>>    If we would really want to have 1.2.0, I suggest that we make
> the
> > >>>> 1.1.1
> > >>>>>    version correct right now rather than after releasing 1.1.1. We
> > >> also
> > >>>>> merged
> > >>>>>    non-bugfix things to 1.x-branch but that's not what users
> expect. I
> > >>>>> agree
> > >>>>>    that work may be painful, but anyway need to do it.
> > >>>>>
> > >>>>>    - Jungtaek Lim (HeartSaVioR)
> > >>>>>
> > >>>>>    2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> <evans@yahoo-inc.com.invalid
> > >>> 님이
> > >>>>> 작성:
> > >>>>>
> > >>>>>> I would love to see the metrics V2 code come out sooner rather
> > >> than
> > >>>>>> later.  +1.
> > >>>>>> My biggest blocker for a 1.x release is
> > >>>>>> https://github.com/apache/storm/pull/2142  Even though the pull
> > >>>>> request
> > >>>>>> says it is minor it showed that we messed up pushing back some
> > >>>>> changes for
> > >>>>>> pacemaker to open source (the code does not run at all which for
> > >> me
> > >>>>> is a
> > >>>>>> blocker) and I really want to get that fully fixed/tested before
> > >>>>> another
> > >>>>>> release.
> > >>>>>> As for 2.x I think we are very close to being able to so a 2.x
> > >>>> alpha
> > >>>>>> release.  I would like to see metrics V2 merged in simply because
> > >>>> it
> > >>>>> is a
> > >>>>>> big change for user facing APIs.  But after that I would love to
> > >>>> see
> > >>>>> us
> > >>>>>> starting to push forward on getting that out.
> > >>>>>>
> > >>>>>>
> > >>>>>> - Bobby
> > >>>>>>
> > >>>>>>
> > >>>>>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> > >>>>>> ptgoetz@gmail.com> wrote:
> > >>>>>>
> > >>>>>> I’d like to bump this thread and start a discussion around our
> > >> next
> > >>>>>> release. Here are my thoughts.
> > >>>>>>
> > >>>>>> There are a number of important fixes in 1.x-branch so I’d like
> > >> to
> > >>>>>> consider releasing 1.1.1 soon. I’d appreciate input on any open
> > >>>>> issues that
> > >>>>>> should be resolved for that release.
> > >>>>>>
> > >>>>>> I’d like us to consider releasing the metrics improvements in
> > >>>>> STORM-2153
> > >>>>>> [1] as version 1.2.0. That work is in the metrics_v2 feature
> > >> branch
> > >>>>> right
> > >>>>>> now and would need to be reviewed and merged. That work is
> > >> against
> > >>>>> the
> > >>>>>> 1.x-branch right now. I would recommend porting it to master
> > >>>> *after*
> > >>>>> the
> > >>>>>> review/merge since there will likely be changes as a result of
> > >> the
> > >>>>> review.
> > >>>>>>
> > >>>>>>> Maybe related to or not, but would we want to create a new
> > >> branch
> > >>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> If wee agree to the above, I would say yes. After the 1.1.1
> > >>>> release,
> > >>>>> we
> > >>>>>> could create a 1.1.x-branch that would be the maintenance/release
> > >>>>> branch
> > >>>>>> for that version line. 1.x-branch would then become the target
> > >> for
> > >>>>> the
> > >>>>>> 1.2.0 release.
> > >>>>>>
> > >>>>>> There are a few fixes in the 0.10.x branch that probably warrant
> > >> a
> > >>>>>> release. After that we may want to back away from that version
> > >> line
> > >>>>> a bit
> > >>>>>> so we can focus more on newer versions.
> > >>>>>>
> > >>>>>> In the past, we’ve shied away form doing “beta” releases, but I’m
> > >>>>>> wondering if we might want to revisit that for the 2.0 release —
> > >>>> the
> > >>>>> idea
> > >>>>>> being that it would give early adopter users a chance to kick the
> > >>>>> tires on
> > >>>>>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
> > >> help
> > >>>>> make the
> > >>>>>> final release more solid. I’m on the fence here and could go
> > >> either
> > >>>>> way.
> > >>>>>>
> > >>>>>> I’d appreciate any input others may have.
> > >>>>>>
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> -Taylor
> > >>>>>>
> > >>>>>>
> > >>>>>> [1] https://issues.apache.org/jira/browse/STORM-2153
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>> Maybe related to or not, but would we want to create a new
> > >> branch
> > >>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >>>>>>>
> > >>>>>>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> > >> so
> > >>>>> hence
> > >>>>>> the
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> - Jungtaek Lim (HeartSaVioR)
> > >>>>>>>
> > >>>>>>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > >>>>> hlouro@hortonworks.com>님이
> > >>>>>> 작성:
> > >>>>>>>
> > >>>>>>>> +1 for finishing the porting to Java ahead of anything else -
> > >> it
> > >>>>> will
> > >>>>>> be a
> > >>>>>>>> significant milestone. I have a JIRA assigned concerning to
> > >> the
> > >>>>>> porting. I
> > >>>>>>>> will work on it for the 2.0 release.
> > >>>>>>>>
> > >>>>>>>> it’s a priority to guarantee no performance regressions. As
> > >> part
> > >>>>> of this
> > >>>>>>>> endeavor, explore an automated (or easy) way to run and assert
> > >>>>> major
> > >>>>>>>> performance benchmarks. Ideally any contributor should be able
> > >>>> to
> > >>>>> fairly
> > >>>>>>>> easily test the impact of changes under certain performance
> > >> test
> > >>>>>> scenarios.
> > >>>>>>>>
> > >>>>>>>> Beam Runner work should take into account the impact of
> > >>>>> incorporating
> > >>>>>> new
> > >>>>>>>> JStorm features and Storm Worker Redesign<
> > >>>>>>>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > >>>>> efficient
> > >>>>>> to
> > >>>>>>>> start doing it, to  find out that it will have to chance in
> > >> face
> > >>>>> of
> > >>>>>> Storm
> > >>>>>>>> and worker redesign. That is, it should be done after it’s
> > >>>>> building
> > >>>>>> blocks
> > >>>>>>>> are stable.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Hugo
> > >>>>>>>>
> > >>>>>>>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> > >> arunm@apache.org
> > >>>>> <mailto:
> > >>>>>>>> arunm@apache.org>> wrote:
> > >>>>>>>>
> > >>>>>>>> +1 to release with the porting completed. I think its mainly
> > >> the
> > >>>>> UI
> > >>>>>> server
> > >>>>>>>> and log viewer that’s pending.
> > >>>>>>>>
> > >>>>>>>> We can start doing the regression and performance tests for
> > >>>>> whatever is
> > >>>>>>>> already ported.
> > >>>>>>>>
> > >>>>>>>> If anyone is running the master branch in their pre-prod /
> > >> prod
> > >>>>>>>> environments, it will be good to know and give us more
> > >>>> confidence.
> > >>>>>>>>
> > >>>>>>>> The other features can be added in follow up releases.
> > >>>>>>>>
> > >>>>>>>> Regards,
> > >>>>>>>> Arun
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 3/24/17, 11:47 AM, "Satish Duggana" <
> > >>>> satish.duggana@gmail.com
> > >>>>>> <mailto:
> > >>>>>>>> satish.duggana@gmail.com>> wrote:
> > >>>>>>>>
> > >>>>>>>> +1 to have 2.0 with porting and performance(it should be at
> > >>>> least
> > >>>>> as
> > >>>>>> good
> > >>>>>>>> as 1.x release) issues addressed
> > >>>>>>>>
> > >>>>>>>> We can target other tasks(mentioned by Taylor and Jungtaek)
> > >> for
> > >>>>>> 2.x-branch.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Exactly-once support:
> > >>>>>>>> While thinking through the exactlyonce support design, it is
> > >>>>> realized
> > >>>>>>>> better to avoid acking tuples and implement exactly once by
> > >>>>> snapshotting
> > >>>>>>>> barriers. It seems JStorm folks followed similar design, they
> > >>>>> claim it
> > >>>>>>>> gives better performance. This feature is essential for beam
> > >>>>> runner and
> > >>>>>> we
> > >>>>>>>> can decide on respective approaches though.
> > >>>>>>>>
> > >>>>>>>> Beam Runner
> > >>>>>>>> Lets hold on this for now and keep it in Storm till 2.x. We
> > >>>>> should avoid
> > >>>>>>>> having a minimal beam runner in haste. It is better to address
> > >>>>>> STORM-2284,
> > >>>>>>>> exactly-once and other windowing enhancements to enable beam
> > >>>>> runner.
> > >>>>>>>>
> > >>>>>>>> JStorm
> > >>>>>>>> Agree with Jungtaek on looking at the latest JStorm and
> > >>>>> align/scope with
> > >>>>>>>> the features for 2.x.
> > >>>>>>>>
> > >>>>>>>> STORM-2284
> > >>>>>>>> We may want to look at JStorm worker before working on
> > >>>> respective
> > >>>>>>>> components in this epic to pull appropriate enhancements.
> > >>>>>>>>
> > >>>>>>>> YARN/MESOS
> > >>>>>>>> Supporting Storm on YARN/Mesos for 2.x.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Satish.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> > >>>> kabhwan@gmail.com
> > >>>>>> <mailto:
> > >>>>>>>> kabhwan@gmail.com>> wrote:
> > >>>>>>>>
> > >>>>>>>> First of all, +1 to complete only port work and do sanity
> > >> check
> > >>>>>> (including
> > >>>>>>>> performance regression), and release.
> > >>>>>>>>
> > >>>>>>>> If we can get STORM-2284 within deterministic time frame (say
> > >>>> 2~3
> > >>>>>> months)
> > >>>>>>>> that should be great, but if not I'd in favor of postponing
> > >> that
> > >>>>> to
> > >>>>>> later
> > >>>>>>>> 2.x release.
> > >>>>>>>>
> > >>>>>>>> JStorm released their new versions after code donation. So
> > >>>>> there're more
> > >>>>>>>> things we could get ideas from, or even adopt from.
> > >>>>>>>> https://github.com/alibaba/jstorm/blob/master/history.md
> > >>>>>>>> As you noticed from release note link, we also need to update
> > >>>>> phase 2
> > >>>>>> since
> > >>>>>>>> they already changed what we're planning to do in phase 2. For
> > >>>>> example,
> > >>>>>>>> they changed backpressure to end-to-end, and changed to use
> > >>>>> snapshot
> > >>>>>> rather
> > >>>>>>>> than acker.
> > >>>>>>>> May be sure, JStorm pulled many features from today's Storm,
> > >>>> like
> > >>>>> Flux,
> > >>>>>>>> Windowing, more shuffle groupings, log search, log level
> > >> change,
> > >>>>> and so
> > >>>>>> on.
> > >>>>>>>>
> > >>>>>>>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> > >>>> is
> > >>>>> due to
> > >>>>>>>> the
> > >>>>>>>> limitation of Spout lifecycle (all the things are done in
> > >> single
> > >>>>>> thread),
> > >>>>>>>> and STORM-1358 <
> > >>>> https://issues.apache.org/jira/browse/STORM-1358
> > >>>>>>> (JStorm's
> > >>>>>>>> multi-thread Spout) can remedy this (despite that Spout
> > >>>>> implementation
> > >>>>>> may
> > >>>>>>>> need to guarantee thread-safety later). It's not a just
> > >>>>> improvement but
> > >>>>>>>> close to design concern so would like to address sooner than
> > >>>> other
> > >>>>>> things
> > >>>>>>>> in phase 2.
> > >>>>>>>>
> > >>>>>>>> For Storm SQL side, I've lost progress but major work would be
> > >>>>> adopting
> > >>>>>>>> group by with windowing. It was not available from Calcite but
> > >>>>> will be
> > >>>>>>>> available at next release (1.12.0).
> > >>>>>>>> I've filed this to STORM-2405
> > >>>>>>>> <https://issues.apache.org/jira/browse/STORM-2405>, but
> > >>>>> windowing &
> > >>>>>> micro
> > >>>>>>>> batch is not intuitive, so I would like to change the
> > >> underlying
> > >>>>> API to
> > >>>>>>>> stream API in SQL. Also filed this to STORM-2406
> > >>>>>>>> <https://issues.apache.org/jira/browse/STORM-2406>.
> > >>>>>>>>
> > >>>>>>>> Just 2 cents btw, hopefully I would like to see metrics V2
> > >>>> sooner
> > >>>>> since
> > >>>>>> we
> > >>>>>>>> lost metrics even when doing normal operation like restarting
> > >>>>> worker,
> > >>>>>>>> rebalancing, and so on. Eventually we need to fight with
> > >> dynamic
> > >>>>>> scaling,
> > >>>>>>>> and then metrics will be broken often.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Jungtaek Lim (HeartSaVioR)
> > >>>>>>>>
> > >>>>>>>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> > >> storm@harsha.io
> > >>>>> 님이
> > >>>>> 작성:
> > >>>>>>>>
> > >>>>>>>> Storm 2.0 migration to java in itself is a big win and would
> > >>>>> attract
> > >>>>>>>> wider
> > >>>>>>>> community and adoption. So my vote would be to resolve the
> > >> first
> > >>>>> 3 items
> > >>>>>>>> to
> > >>>>>>>> get a release out.
> > >>>>>>>> All the other featured mentioned are great to have but
> > >> shouldn't
> > >>>>> be
> > >>>>>>>> blockers for 2.0 release.
> > >>>>>>>>
> > >>>>>>>> -Harsha
> > >>>>>>>>
> > >>>>>>>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > >>>>> ptgoetz@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> With the 1.1.0 release nearing completion, I’d like to turn
> > >> our
> > >>>>>>>> attention
> > >>>>>>>> to 2.0 and develop a plan for what features, etc. to include.
> > >>>>>>>>
> > >>>>>>>> The following 3 are what I feel are the minimum for a 2.0
> > >>>> release.
> > >>>>>>>> These
> > >>>>>>>> could likely be resolved relatively quickly:
> > >>>>>>>>
> > >>>>>>>> * Performance — I’ve not benchmarked the master branch vs.
> > >> 1.0.x
> > >>>>> or
> > >>>>>>>> 1.1.x
> > >>>>>>>> in a while, but I feel it will be important to make sure there
> > >>>>> are no
> > >>>>>>>> performance regressions, and would hope that we actually have
> > >> a
> > >>>>>>>> performance
> > >>>>>>>> improvement over previous versions. To that end (e.g. if there
> > >>>> is
> > >>>>> in
> > >>>>>>>> fact a
> > >>>>>>>> performance regression), the proposals that Roshan Naik put
> > >>>>> together
> > >>>>>>>> for
> > >>>>>>>> revising the threading and execution model (STORM-2307) and
> > >>>>> replacing
> > >>>>>>>> Disruptor with JCTools (STORM-2306) warrant review and
> > >>>>> consideration.
> > >>>>>>>> See
> > >>>>>>>> also STORM-2284 which is the parent JIRA.
> > >>>>>>>>
> > >>>>>>>> * Finish porting Storm UI to java (STORM-1311)
> > >>>>>>>>
> > >>>>>>>> * Finish porting log viewer to java (STORM-1280)
> > >>>>>>>>
> > >>>>>>>> The following are items that are nice to have in 2.0, but I
> > >>>> don’t
> > >>>>> feel
> > >>>>>>>> are
> > >>>>>>>> absolutely necessary for an initial 2.0 release:
> > >>>>>>>>
> > >>>>>>>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> > >> because
> > >>>>> it’s
> > >>>>>>>> relevant) — Initially there seemed to be a lot of interest in
> > >>>>> this, but
> > >>>>>>>> that seems to have trailed off. I spoke with some Beam
> > >>>> developers
> > >>>>> and
> > >>>>>>>> there
> > >>>>>>>> seems to be interest from that community as well. Do we want
> > >> to
> > >>>>> move
> > >>>>>>>> that
> > >>>>>>>> effort to the Beam community, or keep it here? Moving it to
> > >> the
> > >>>>> Beam
> > >>>>>>>> community might lead to better collaboration between projects.
> > >>>>>>>>
> > >>>>>>>> * Bounded Spouts (needed for Beam Runner implementation) —
> > >>>>> Currently
> > >>>>>>>> spouts are unbounded, there no end to the stream. Beam has the
> > >>>>> concept
> > >>>>>>>> of
> > >>>>>>>> bounded sources (roughly analogous to batch processing). To
> > >>>>> support
> > >>>>>>>> that,
> > >>>>>>>> we would need to implement a similar concept in Storm. One
> > >>>>> benefit of
> > >>>>>>>> such
> > >>>>>>>> a feature would be the ability to handle both bounded and
> > >>>>> unbounded
> > >>>>>>>> workflows in Storm.
> > >>>>>>>>
> > >>>>>>>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > >>>>> behind
> > >>>>>>>> this
> > >>>>>>>> effort. What improvements do you envision for 2.0?
> > >>>>>>>>
> > >>>>>>>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > >>>>> targeting this
> > >>>>>>>> for 1.2.0, but it’s designed to be easily portable to
> > >>>> master/2.0.
> > >>>>>>>>
> > >>>>>>>> * JStorm Migration — Original outline can be found here [1].
> > >>>> Note
> > >>>>> a lot
> > >>>>>>>> of
> > >>>>>>>> the associated JIRAs below are assigned, but there hasn’t been
> > >>>> any
> > >>>>>>>> recent
> > >>>>>>>> activity or pull requests, we should probably consider them
> > >>>>> unassigned
> > >>>>>>>> and
> > >>>>>>>> up for grabs.:
> > >>>>>>>>
> > >>>>>>>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > >>>>> been the
> > >>>>>>>> bane of a lot of Storm users almost since day one. We have
> > >>>> largely
> > >>>>>>>> addressed it by shading/relocating dependencies. It would be
> > >>>>> great to
> > >>>>>>>> see
> > >>>>>>>> this addressed once and for all.
> > >>>>>>>>
> > >>>>>>>> * JStorm back pressure implementation (STORM-1324) — The
> > >> current
> > >>>>> back
> > >>>>>>>> pressure implementation leaves a bit to be desired, and the
> > >>>> JStorm
> > >>>>>>>> approach
> > >>>>>>>> looks promising, though it also depends on the JStorm concept
> > >> of
> > >>>>>>>> “topology
> > >>>>>>>> master” (STORM-1323), which may have some implications
> > >> regarding
> > >>>>>>>> security.
> > >>>>>>>>
> > >>>>>>>> * Dynamic Topology Updates (STORM-1335) — This would provide a
> > >>>>> command
> > >>>>>>>> to
> > >>>>>>>> update topology jars and configuration without stopping the
> > >>>>> topology,
> > >>>>>>>> and
> > >>>>>>>> is well suited to leverage the blobstore. The restart command
> > >>>>> (that can
> > >>>>>>>> also update the topology configuration) also looks compelling
> > >>>>>>>> (STORM-1334).
> > >>>>>>>>
> > >>>>>>>> * Additional Scheduler Implementations (STORM-1320)
> > >>>>>>>>
> > >>>>>>>> * Additional Grouping Implementations (STORM-1328)
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> As always I’m open to any opinions and suggestions.
> > >>>>>>>>
> > >>>>>>>> -Taylor
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>>
> > >>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.
> > >>>>>>>> action?pageId=61328109
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>
> > >>
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Jungtaek Lim <ka...@gmail.com>.
That's great news that metrics work is ready!

I'm +1 to Taylor's proposal, but in order to respect semantic versioning, I
propose some modifications from Taylor's proposal:

- create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back only
bug fixes to the 1.1.x-branch
- change the target version of 1.x-branch to 1.2.0-SNAPSHOT

If we also agree above, I would like to volunteer the back-port work.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 28일 (수) 오전 10:09, Harsha <st...@harsha.io>님이 작성:

> +1 for above stated approach on releasing 1.2.0 with metrics
> -Harsha
>
> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> > The work on metrics is ready for a pull request to 1.x-branch from the
> > feature branch. I’ve held off because we haven’t reached consensus on a
> > path forward with the 1.x release lines .
> >
> > I’d like to propose the following for the 1.x line:
> >
> > 1. Create a branch for 1.2 so we have a branch to review the metrics
> > stuff.
> > 2. Release 1.1.1
> > 3. Review/merge metrics work. Port metrics to master.
> > 4. Release 1.2.0
> > 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
> > (we would only support 1.2.x and 1.1.x which are very closely aligned).
> >
> > Dropping support for 1.0.x line would eliminate the need to maintain one
> > of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> > very closely aligned. I just up merged metrics_v2 against 1.x-branch
> > after a while, and there were no conflicts.
> >
> > That would give us a little more bandwidth to focus on 2.0 and needed bug
> > fixes to the 1.x line like some of the issues raised with
> > storm-kafka-client. We could even start releasing alpha/beta versions of
> > 2.0 in parallel to the steps above.
> >
> > Any thoughts on that approach?
> >
> > -Taylor
> >
> >
> > > On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> > >
> > > Yes I prefer option 1, but it might depend on the progress of metrics
> V2.
> > > If it can be done within predictable near future I'm OK to pick option
> 2,
> > > but if not, we may be better to focus releasing 2.0.0 and make it
> really
> > > happen.
> > >
> > > Whichever we go, I feel it's time to track remaining work on Storm
> 2.0.0. I
> > > found some bugs on master branch so filed issues, and we've remaining
> port
> > > work (UI and logviewer). We've some other improvements target for
> 2.0.0:
> > > worker redesign, beam integration, and so on, and we don't track its
> > > progress at all. I don't think we should wait for features which
> progress
> > > is not transparent (in other words we don't know when it will be
> finished).
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> > >
> > >> Bobby/Jungtaek,
> > >>
> > >> Are you saying you want to forego the 1.2 “metrics_v2” release and
> include
> > >> it only in 2.0? (I ask because that work is already based on
> 1.x-branch,
> > >> and forward-porting it to master is relatively simple.) I’d kind of
> like
> > >> that work go out soon.
> > >>
> > >> If we go with option 1, I would want to see a 2.0 release (even if
> it’s a
> > >> “beta” or “preview) before putting the 1.x line into maintenance mode.
> > >>
> > >> -Taylor
> > >>
> > >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <evans@yahoo-inc.com.INVALID
> >
> > >> wrote:
> > >>>
> > >>> I see 2 ways to address this.
> > >>> 1) We put the 1.x line into maintenance mode like with 0.10.  We
> don't
> > >> backport anything except bug fixes.2) We backport a lot of the
> backwards
> > >> compatible changes from 2.x to 1.x.
> > >>> My personal preference is 1.  It makes it clear the direction we
> want to
> > >> go in.  The biggest issue is that we probably would want to do a 2.x
> > >> release sooner rather then later.  Even if we don't get all of the
> features
> > >> that people want, if we just get a release out we can add in new
> features
> > >> if they are backwards compatible, or we can create a 3.x line that
> would
> > >> have the breaking changes in it.
> > >>>
> > >>> - Bobby
> > >>>
> > >>>
> > >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> > >> kabhwan@gmail.com> wrote:
> > >>>
> > >>> I'd like to bump this again instead of initiating new discussion
> thread.
> > >>>
> > >>> I had having hard time to create and apply pull requests for both
> master
> > >>> and 1.x-branch and that's really painful and sometimes blocker for
> me to
> > >> do
> > >>> merge step.
> > >>> Two branches are heavily diverged more than between 0.10 and 1.0.0,
> even
> > >>> IDE can't switch between the branch smoothly. We didn't even address
> > >>> checkstyle issue yet, but after addressing, it could be "completely"
> > >>> diverged. JDK version is another major issue, since the pull requests
> > >>> targeted for master branch are not checked against JDK 7, and some of
> > >> them
> > >>> make some issues regarding JDK version while porting back.
> > >>>
> > >>> So personally I really would like to see the plan for 1.x version
> line
> > >>> changed - skipping any minor releases including 1.2.0 - and have epic
> > >> issue
> > >>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> > >>> proposed plan was having 2.0.0 directly after 1.0.0)
> > >>>
> > >>> Would like to hear everyone's opinions. If we have consensus to not
> > >> having
> > >>> any minor releases for 1.x version line, I will not port back
> non-bugfix
> > >>> pull requests to 1.x-branch, and guide contributors to create pull
> > >> requests
> > >>> against master branch, not 1.x version line.
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> > >> avermeerbergen@gmail.com>님이
> > >>> 작성:
> > >>>
> > >>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision
> system,
> > >>>> we're very interested anything that can provide better throughput.
> > >>>>
> > >>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
> > >>>>
> > >>>>> For 2.0 beta … it would be good to incorporate some of the Worker
> > >>>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can
> be
> > >>>>> delivered sooner and my in-progress implementation suggests that it
> > >> will
> > >>>>> yield substantial latency improvements. The 2.0 beta phase would
> really
> > >>>>> help kick the tires on the revised messaging system and the
> performance
> > >>>>> improvements will also be a good incentive for trying out the 2.0
> line.
> > >>>>>
> > >>>>> I notice multiple other bottlenecks that are holding back
> throughput a
> > >>>>> lot, which can be addressed in a subsequent 2.x minor release.
> > >>>>> -roshan
> > >>>>>
> > >>>>>
> > >>>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> > >>>>>
> > >>>>>    I also would love to see metrics V2 code sooner or later too.
> If we
> > >>>>> can get
> > >>>>>    it before releasing 2.0.0 that will be great, and then maybe we
> > >> could
> > >>>>> just
> > >>>>>    move toward to 2.0.0, not adding any improvements to 1.x version
> > >>>> line.
> > >>>>>    (And that's what I would want to.)
> > >>>>>
> > >>>>>    If we would really want to have 1.2.0, I suggest that we make
> the
> > >>>> 1.1.1
> > >>>>>    version correct right now rather than after releasing 1.1.1. We
> > >> also
> > >>>>> merged
> > >>>>>    non-bugfix things to 1.x-branch but that's not what users
> expect. I
> > >>>>> agree
> > >>>>>    that work may be painful, but anyway need to do it.
> > >>>>>
> > >>>>>    - Jungtaek Lim (HeartSaVioR)
> > >>>>>
> > >>>>>    2017년 6월 3일 (토) 오전 3:49, Bobby Evans
> <evans@yahoo-inc.com.invalid
> > >>> 님이
> > >>>>> 작성:
> > >>>>>
> > >>>>>> I would love to see the metrics V2 code come out sooner rather
> > >> than
> > >>>>>> later.  +1.
> > >>>>>> My biggest blocker for a 1.x release is
> > >>>>>> https://github.com/apache/storm/pull/2142  Even though the pull
> > >>>>> request
> > >>>>>> says it is minor it showed that we messed up pushing back some
> > >>>>> changes for
> > >>>>>> pacemaker to open source (the code does not run at all which for
> > >> me
> > >>>>> is a
> > >>>>>> blocker) and I really want to get that fully fixed/tested before
> > >>>>> another
> > >>>>>> release.
> > >>>>>> As for 2.x I think we are very close to being able to so a 2.x
> > >>>> alpha
> > >>>>>> release.  I would like to see metrics V2 merged in simply because
> > >>>> it
> > >>>>> is a
> > >>>>>> big change for user facing APIs.  But after that I would love to
> > >>>> see
> > >>>>> us
> > >>>>>> starting to push forward on getting that out.
> > >>>>>>
> > >>>>>>
> > >>>>>> - Bobby
> > >>>>>>
> > >>>>>>
> > >>>>>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> > >>>>>> ptgoetz@gmail.com> wrote:
> > >>>>>>
> > >>>>>> I’d like to bump this thread and start a discussion around our
> > >> next
> > >>>>>> release. Here are my thoughts.
> > >>>>>>
> > >>>>>> There are a number of important fixes in 1.x-branch so I’d like
> > >> to
> > >>>>>> consider releasing 1.1.1 soon. I’d appreciate input on any open
> > >>>>> issues that
> > >>>>>> should be resolved for that release.
> > >>>>>>
> > >>>>>> I’d like us to consider releasing the metrics improvements in
> > >>>>> STORM-2153
> > >>>>>> [1] as version 1.2.0. That work is in the metrics_v2 feature
> > >> branch
> > >>>>> right
> > >>>>>> now and would need to be reviewed and merged. That work is
> > >> against
> > >>>>> the
> > >>>>>> 1.x-branch right now. I would recommend porting it to master
> > >>>> *after*
> > >>>>> the
> > >>>>>> review/merge since there will likely be changes as a result of
> > >> the
> > >>>>> review.
> > >>>>>>
> > >>>>>>> Maybe related to or not, but would we want to create a new
> > >> branch
> > >>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> If wee agree to the above, I would say yes. After the 1.1.1
> > >>>> release,
> > >>>>> we
> > >>>>>> could create a 1.1.x-branch that would be the maintenance/release
> > >>>>> branch
> > >>>>>> for that version line. 1.x-branch would then become the target
> > >> for
> > >>>>> the
> > >>>>>> 1.2.0 release.
> > >>>>>>
> > >>>>>> There are a few fixes in the 0.10.x branch that probably warrant
> > >> a
> > >>>>>> release. After that we may want to back away from that version
> > >> line
> > >>>>> a bit
> > >>>>>> so we can focus more on newer versions.
> > >>>>>>
> > >>>>>> In the past, we’ve shied away form doing “beta” releases, but I’m
> > >>>>>> wondering if we might want to revisit that for the 2.0 release —
> > >>>> the
> > >>>>> idea
> > >>>>>> being that it would give early adopter users a chance to kick the
> > >>>>> tires on
> > >>>>>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
> > >> help
> > >>>>> make the
> > >>>>>> final release more solid. I’m on the fence here and could go
> > >> either
> > >>>>> way.
> > >>>>>>
> > >>>>>> I’d appreciate any input others may have.
> > >>>>>>
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> -Taylor
> > >>>>>>
> > >>>>>>
> > >>>>>> [1] https://issues.apache.org/jira/browse/STORM-2153
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>> Maybe related to or not, but would we want to create a new
> > >> branch
> > >>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> > >>>>>>>
> > >>>>>>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> > >> so
> > >>>>> hence
> > >>>>>> the
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> - Jungtaek Lim (HeartSaVioR)
> > >>>>>>>
> > >>>>>>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > >>>>> hlouro@hortonworks.com>님이
> > >>>>>> 작성:
> > >>>>>>>
> > >>>>>>>> +1 for finishing the porting to Java ahead of anything else -
> > >> it
> > >>>>> will
> > >>>>>> be a
> > >>>>>>>> significant milestone. I have a JIRA assigned concerning to
> > >> the
> > >>>>>> porting. I
> > >>>>>>>> will work on it for the 2.0 release.
> > >>>>>>>>
> > >>>>>>>> it’s a priority to guarantee no performance regressions. As
> > >> part
> > >>>>> of this
> > >>>>>>>> endeavor, explore an automated (or easy) way to run and assert
> > >>>>> major
> > >>>>>>>> performance benchmarks. Ideally any contributor should be able
> > >>>> to
> > >>>>> fairly
> > >>>>>>>> easily test the impact of changes under certain performance
> > >> test
> > >>>>>> scenarios.
> > >>>>>>>>
> > >>>>>>>> Beam Runner work should take into account the impact of
> > >>>>> incorporating
> > >>>>>> new
> > >>>>>>>> JStorm features and Storm Worker Redesign<
> > >>>>>>>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > >>>>> efficient
> > >>>>>> to
> > >>>>>>>> start doing it, to  find out that it will have to chance in
> > >> face
> > >>>>> of
> > >>>>>> Storm
> > >>>>>>>> and worker redesign. That is, it should be done after it’s
> > >>>>> building
> > >>>>>> blocks
> > >>>>>>>> are stable.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Hugo
> > >>>>>>>>
> > >>>>>>>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> > >> arunm@apache.org
> > >>>>> <mailto:
> > >>>>>>>> arunm@apache.org>> wrote:
> > >>>>>>>>
> > >>>>>>>> +1 to release with the porting completed. I think its mainly
> > >> the
> > >>>>> UI
> > >>>>>> server
> > >>>>>>>> and log viewer that’s pending.
> > >>>>>>>>
> > >>>>>>>> We can start doing the regression and performance tests for
> > >>>>> whatever is
> > >>>>>>>> already ported.
> > >>>>>>>>
> > >>>>>>>> If anyone is running the master branch in their pre-prod /
> > >> prod
> > >>>>>>>> environments, it will be good to know and give us more
> > >>>> confidence.
> > >>>>>>>>
> > >>>>>>>> The other features can be added in follow up releases.
> > >>>>>>>>
> > >>>>>>>> Regards,
> > >>>>>>>> Arun
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 3/24/17, 11:47 AM, "Satish Duggana" <
> > >>>> satish.duggana@gmail.com
> > >>>>>> <mailto:
> > >>>>>>>> satish.duggana@gmail.com>> wrote:
> > >>>>>>>>
> > >>>>>>>> +1 to have 2.0 with porting and performance(it should be at
> > >>>> least
> > >>>>> as
> > >>>>>> good
> > >>>>>>>> as 1.x release) issues addressed
> > >>>>>>>>
> > >>>>>>>> We can target other tasks(mentioned by Taylor and Jungtaek)
> > >> for
> > >>>>>> 2.x-branch.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Exactly-once support:
> > >>>>>>>> While thinking through the exactlyonce support design, it is
> > >>>>> realized
> > >>>>>>>> better to avoid acking tuples and implement exactly once by
> > >>>>> snapshotting
> > >>>>>>>> barriers. It seems JStorm folks followed similar design, they
> > >>>>> claim it
> > >>>>>>>> gives better performance. This feature is essential for beam
> > >>>>> runner and
> > >>>>>> we
> > >>>>>>>> can decide on respective approaches though.
> > >>>>>>>>
> > >>>>>>>> Beam Runner
> > >>>>>>>> Lets hold on this for now and keep it in Storm till 2.x. We
> > >>>>> should avoid
> > >>>>>>>> having a minimal beam runner in haste. It is better to address
> > >>>>>> STORM-2284,
> > >>>>>>>> exactly-once and other windowing enhancements to enable beam
> > >>>>> runner.
> > >>>>>>>>
> > >>>>>>>> JStorm
> > >>>>>>>> Agree with Jungtaek on looking at the latest JStorm and
> > >>>>> align/scope with
> > >>>>>>>> the features for 2.x.
> > >>>>>>>>
> > >>>>>>>> STORM-2284
> > >>>>>>>> We may want to look at JStorm worker before working on
> > >>>> respective
> > >>>>>>>> components in this epic to pull appropriate enhancements.
> > >>>>>>>>
> > >>>>>>>> YARN/MESOS
> > >>>>>>>> Supporting Storm on YARN/Mesos for 2.x.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Satish.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> > >>>> kabhwan@gmail.com
> > >>>>>> <mailto:
> > >>>>>>>> kabhwan@gmail.com>> wrote:
> > >>>>>>>>
> > >>>>>>>> First of all, +1 to complete only port work and do sanity
> > >> check
> > >>>>>> (including
> > >>>>>>>> performance regression), and release.
> > >>>>>>>>
> > >>>>>>>> If we can get STORM-2284 within deterministic time frame (say
> > >>>> 2~3
> > >>>>>> months)
> > >>>>>>>> that should be great, but if not I'd in favor of postponing
> > >> that
> > >>>>> to
> > >>>>>> later
> > >>>>>>>> 2.x release.
> > >>>>>>>>
> > >>>>>>>> JStorm released their new versions after code donation. So
> > >>>>> there're more
> > >>>>>>>> things we could get ideas from, or even adopt from.
> > >>>>>>>> https://github.com/alibaba/jstorm/blob/master/history.md
> > >>>>>>>> As you noticed from release note link, we also need to update
> > >>>>> phase 2
> > >>>>>> since
> > >>>>>>>> they already changed what we're planning to do in phase 2. For
> > >>>>> example,
> > >>>>>>>> they changed backpressure to end-to-end, and changed to use
> > >>>>> snapshot
> > >>>>>> rather
> > >>>>>>>> than acker.
> > >>>>>>>> May be sure, JStorm pulled many features from today's Storm,
> > >>>> like
> > >>>>> Flux,
> > >>>>>>>> Windowing, more shuffle groupings, log search, log level
> > >> change,
> > >>>>> and so
> > >>>>>> on.
> > >>>>>>>>
> > >>>>>>>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> > >>>> is
> > >>>>> due to
> > >>>>>>>> the
> > >>>>>>>> limitation of Spout lifecycle (all the things are done in
> > >> single
> > >>>>>> thread),
> > >>>>>>>> and STORM-1358 <
> > >>>> https://issues.apache.org/jira/browse/STORM-1358
> > >>>>>>> (JStorm's
> > >>>>>>>> multi-thread Spout) can remedy this (despite that Spout
> > >>>>> implementation
> > >>>>>> may
> > >>>>>>>> need to guarantee thread-safety later). It's not a just
> > >>>>> improvement but
> > >>>>>>>> close to design concern so would like to address sooner than
> > >>>> other
> > >>>>>> things
> > >>>>>>>> in phase 2.
> > >>>>>>>>
> > >>>>>>>> For Storm SQL side, I've lost progress but major work would be
> > >>>>> adopting
> > >>>>>>>> group by with windowing. It was not available from Calcite but
> > >>>>> will be
> > >>>>>>>> available at next release (1.12.0).
> > >>>>>>>> I've filed this to STORM-2405
> > >>>>>>>> <https://issues.apache.org/jira/browse/STORM-2405>, but
> > >>>>> windowing &
> > >>>>>> micro
> > >>>>>>>> batch is not intuitive, so I would like to change the
> > >> underlying
> > >>>>> API to
> > >>>>>>>> stream API in SQL. Also filed this to STORM-2406
> > >>>>>>>> <https://issues.apache.org/jira/browse/STORM-2406>.
> > >>>>>>>>
> > >>>>>>>> Just 2 cents btw, hopefully I would like to see metrics V2
> > >>>> sooner
> > >>>>> since
> > >>>>>> we
> > >>>>>>>> lost metrics even when doing normal operation like restarting
> > >>>>> worker,
> > >>>>>>>> rebalancing, and so on. Eventually we need to fight with
> > >> dynamic
> > >>>>>> scaling,
> > >>>>>>>> and then metrics will be broken often.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Jungtaek Lim (HeartSaVioR)
> > >>>>>>>>
> > >>>>>>>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> > >> storm@harsha.io
> > >>>>> 님이
> > >>>>> 작성:
> > >>>>>>>>
> > >>>>>>>> Storm 2.0 migration to java in itself is a big win and would
> > >>>>> attract
> > >>>>>>>> wider
> > >>>>>>>> community and adoption. So my vote would be to resolve the
> > >> first
> > >>>>> 3 items
> > >>>>>>>> to
> > >>>>>>>> get a release out.
> > >>>>>>>> All the other featured mentioned are great to have but
> > >> shouldn't
> > >>>>> be
> > >>>>>>>> blockers for 2.0 release.
> > >>>>>>>>
> > >>>>>>>> -Harsha
> > >>>>>>>>
> > >>>>>>>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > >>>>> ptgoetz@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> With the 1.1.0 release nearing completion, I’d like to turn
> > >> our
> > >>>>>>>> attention
> > >>>>>>>> to 2.0 and develop a plan for what features, etc. to include.
> > >>>>>>>>
> > >>>>>>>> The following 3 are what I feel are the minimum for a 2.0
> > >>>> release.
> > >>>>>>>> These
> > >>>>>>>> could likely be resolved relatively quickly:
> > >>>>>>>>
> > >>>>>>>> * Performance — I’ve not benchmarked the master branch vs.
> > >> 1.0.x
> > >>>>> or
> > >>>>>>>> 1.1.x
> > >>>>>>>> in a while, but I feel it will be important to make sure there
> > >>>>> are no
> > >>>>>>>> performance regressions, and would hope that we actually have
> > >> a
> > >>>>>>>> performance
> > >>>>>>>> improvement over previous versions. To that end (e.g. if there
> > >>>> is
> > >>>>> in
> > >>>>>>>> fact a
> > >>>>>>>> performance regression), the proposals that Roshan Naik put
> > >>>>> together
> > >>>>>>>> for
> > >>>>>>>> revising the threading and execution model (STORM-2307) and
> > >>>>> replacing
> > >>>>>>>> Disruptor with JCTools (STORM-2306) warrant review and
> > >>>>> consideration.
> > >>>>>>>> See
> > >>>>>>>> also STORM-2284 which is the parent JIRA.
> > >>>>>>>>
> > >>>>>>>> * Finish porting Storm UI to java (STORM-1311)
> > >>>>>>>>
> > >>>>>>>> * Finish porting log viewer to java (STORM-1280)
> > >>>>>>>>
> > >>>>>>>> The following are items that are nice to have in 2.0, but I
> > >>>> don’t
> > >>>>> feel
> > >>>>>>>> are
> > >>>>>>>> absolutely necessary for an initial 2.0 release:
> > >>>>>>>>
> > >>>>>>>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> > >> because
> > >>>>> it’s
> > >>>>>>>> relevant) — Initially there seemed to be a lot of interest in
> > >>>>> this, but
> > >>>>>>>> that seems to have trailed off. I spoke with some Beam
> > >>>> developers
> > >>>>> and
> > >>>>>>>> there
> > >>>>>>>> seems to be interest from that community as well. Do we want
> > >> to
> > >>>>> move
> > >>>>>>>> that
> > >>>>>>>> effort to the Beam community, or keep it here? Moving it to
> > >> the
> > >>>>> Beam
> > >>>>>>>> community might lead to better collaboration between projects.
> > >>>>>>>>
> > >>>>>>>> * Bounded Spouts (needed for Beam Runner implementation) —
> > >>>>> Currently
> > >>>>>>>> spouts are unbounded, there no end to the stream. Beam has the
> > >>>>> concept
> > >>>>>>>> of
> > >>>>>>>> bounded sources (roughly analogous to batch processing). To
> > >>>>> support
> > >>>>>>>> that,
> > >>>>>>>> we would need to implement a similar concept in Storm. One
> > >>>>> benefit of
> > >>>>>>>> such
> > >>>>>>>> a feature would be the ability to handle both bounded and
> > >>>>> unbounded
> > >>>>>>>> workflows in Storm.
> > >>>>>>>>
> > >>>>>>>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > >>>>> behind
> > >>>>>>>> this
> > >>>>>>>> effort. What improvements do you envision for 2.0?
> > >>>>>>>>
> > >>>>>>>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > >>>>> targeting this
> > >>>>>>>> for 1.2.0, but it’s designed to be easily portable to
> > >>>> master/2.0.
> > >>>>>>>>
> > >>>>>>>> * JStorm Migration — Original outline can be found here [1].
> > >>>> Note
> > >>>>> a lot
> > >>>>>>>> of
> > >>>>>>>> the associated JIRAs below are assigned, but there hasn’t been
> > >>>> any
> > >>>>>>>> recent
> > >>>>>>>> activity or pull requests, we should probably consider them
> > >>>>> unassigned
> > >>>>>>>> and
> > >>>>>>>> up for grabs.:
> > >>>>>>>>
> > >>>>>>>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > >>>>> been the
> > >>>>>>>> bane of a lot of Storm users almost since day one. We have
> > >>>> largely
> > >>>>>>>> addressed it by shading/relocating dependencies. It would be
> > >>>>> great to
> > >>>>>>>> see
> > >>>>>>>> this addressed once and for all.
> > >>>>>>>>
> > >>>>>>>> * JStorm back pressure implementation (STORM-1324) — The
> > >> current
> > >>>>> back
> > >>>>>>>> pressure implementation leaves a bit to be desired, and the
> > >>>> JStorm
> > >>>>>>>> approach
> > >>>>>>>> looks promising, though it also depends on the JStorm concept
> > >> of
> > >>>>>>>> “topology
> > >>>>>>>> master” (STORM-1323), which may have some implications
> > >> regarding
> > >>>>>>>> security.
> > >>>>>>>>
> > >>>>>>>> * Dynamic Topology Updates (STORM-1335) — This would provide a
> > >>>>> command
> > >>>>>>>> to
> > >>>>>>>> update topology jars and configuration without stopping the
> > >>>>> topology,
> > >>>>>>>> and
> > >>>>>>>> is well suited to leverage the blobstore. The restart command
> > >>>>> (that can
> > >>>>>>>> also update the topology configuration) also looks compelling
> > >>>>>>>> (STORM-1334).
> > >>>>>>>>
> > >>>>>>>> * Additional Scheduler Implementations (STORM-1320)
> > >>>>>>>>
> > >>>>>>>> * Additional Grouping Implementations (STORM-1328)
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> As always I’m open to any opinions and suggestions.
> > >>>>>>>>
> > >>>>>>>> -Taylor
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>>
> > >>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.
> > >>>>>>>> action?pageId=61328109
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>
> > >>
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Harsha <st...@harsha.io>.
+1 for above stated approach on releasing 1.2.0 with metrics 
-Harsha

On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> The work on metrics is ready for a pull request to 1.x-branch from the
> feature branch. I’ve held off because we haven’t reached consensus on a
> path forward with the 1.x release lines .
> 
> I’d like to propose the following for the 1.x line:
> 
> 1. Create a branch for 1.2 so we have a branch to review the metrics
> stuff.
> 2. Release 1.1.1
> 3. Review/merge metrics work. Port metrics to master.
> 4. Release 1.2.0
> 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
> (we would only support 1.2.x and 1.1.x which are very closely aligned).
> 
> Dropping support for 1.0.x line would eliminate the need to maintain one
> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> very closely aligned. I just up merged metrics_v2 against 1.x-branch
> after a while, and there were no conflicts.
> 
> That would give us a little more bandwidth to focus on 2.0 and needed bug
> fixes to the 1.x line like some of the issues raised with
> storm-kafka-client. We could even start releasing alpha/beta versions of
> 2.0 in parallel to the steps above.
> 
> Any thoughts on that approach?
> 
> -Taylor
> 
> 
> > On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> > 
> > Yes I prefer option 1, but it might depend on the progress of metrics V2.
> > If it can be done within predictable near future I'm OK to pick option 2,
> > but if not, we may be better to focus releasing 2.0.0 and make it really
> > happen.
> > 
> > Whichever we go, I feel it's time to track remaining work on Storm 2.0.0. I
> > found some bugs on master branch so filed issues, and we've remaining port
> > work (UI and logviewer). We've some other improvements target for 2.0.0:
> > worker redesign, beam integration, and so on, and we don't track its
> > progress at all. I don't think we should wait for features which progress
> > is not transparent (in other words we don't know when it will be finished).
> > 
> > - Jungtaek Lim (HeartSaVioR)
> > 
> > 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> > 
> >> Bobby/Jungtaek,
> >> 
> >> Are you saying you want to forego the 1.2 “metrics_v2” release and include
> >> it only in 2.0? (I ask because that work is already based on 1.x-branch,
> >> and forward-porting it to master is relatively simple.) I’d kind of like
> >> that work go out soon.
> >> 
> >> If we go with option 1, I would want to see a 2.0 release (even if it’s a
> >> “beta” or “preview) before putting the 1.x line into maintenance mode.
> >> 
> >> -Taylor
> >> 
> >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> >> wrote:
> >>> 
> >>> I see 2 ways to address this.
> >>> 1) We put the 1.x line into maintenance mode like with 0.10.  We don't
> >> backport anything except bug fixes.2) We backport a lot of the backwards
> >> compatible changes from 2.x to 1.x.
> >>> My personal preference is 1.  It makes it clear the direction we want to
> >> go in.  The biggest issue is that we probably would want to do a 2.x
> >> release sooner rather then later.  Even if we don't get all of the features
> >> that people want, if we just get a release out we can add in new features
> >> if they are backwards compatible, or we can create a 3.x line that would
> >> have the breaking changes in it.
> >>> 
> >>> - Bobby
> >>> 
> >>> 
> >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> >> kabhwan@gmail.com> wrote:
> >>> 
> >>> I'd like to bump this again instead of initiating new discussion thread.
> >>> 
> >>> I had having hard time to create and apply pull requests for both master
> >>> and 1.x-branch and that's really painful and sometimes blocker for me to
> >> do
> >>> merge step.
> >>> Two branches are heavily diverged more than between 0.10 and 1.0.0, even
> >>> IDE can't switch between the branch smoothly. We didn't even address
> >>> checkstyle issue yet, but after addressing, it could be "completely"
> >>> diverged. JDK version is another major issue, since the pull requests
> >>> targeted for master branch are not checked against JDK 7, and some of
> >> them
> >>> make some issues regarding JDK version while porting back.
> >>> 
> >>> So personally I really would like to see the plan for 1.x version line
> >>> changed - skipping any minor releases including 1.2.0 - and have epic
> >> issue
> >>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> >>> proposed plan was having 2.0.0 directly after 1.0.0)
> >>> 
> >>> Would like to hear everyone's opinions. If we have consensus to not
> >> having
> >>> any minor releases for 1.x version line, I will not port back non-bugfix
> >>> pull requests to 1.x-branch, and guide contributors to create pull
> >> requests
> >>> against master branch, not 1.x version line.
> >>> 
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>> 
> >>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> >> avermeerbergen@gmail.com>님이
> >>> 작성:
> >>> 
> >>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
> >>>> we're very interested anything that can provide better throughput.
> >>>> 
> >>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
> >>>> 
> >>>>> For 2.0 beta … it would be good to incorporate some of the Worker
> >>>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
> >>>>> delivered sooner and my in-progress implementation suggests that it
> >> will
> >>>>> yield substantial latency improvements. The 2.0 beta phase would really
> >>>>> help kick the tires on the revised messaging system and the performance
> >>>>> improvements will also be a good incentive for trying out the 2.0 line.
> >>>>> 
> >>>>> I notice multiple other bottlenecks that are holding back throughput a
> >>>>> lot, which can be addressed in a subsequent 2.x minor release.
> >>>>> -roshan
> >>>>> 
> >>>>> 
> >>>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>>>> 
> >>>>>    I also would love to see metrics V2 code sooner or later too. If we
> >>>>> can get
> >>>>>    it before releasing 2.0.0 that will be great, and then maybe we
> >> could
> >>>>> just
> >>>>>    move toward to 2.0.0, not adding any improvements to 1.x version
> >>>> line.
> >>>>>    (And that's what I would want to.)
> >>>>> 
> >>>>>    If we would really want to have 1.2.0, I suggest that we make the
> >>>> 1.1.1
> >>>>>    version correct right now rather than after releasing 1.1.1. We
> >> also
> >>>>> merged
> >>>>>    non-bugfix things to 1.x-branch but that's not what users expect. I
> >>>>> agree
> >>>>>    that work may be painful, but anyway need to do it.
> >>>>> 
> >>>>>    - Jungtaek Lim (HeartSaVioR)
> >>>>> 
> >>>>>    2017년 6월 3일 (토) 오전 3:49, Bobby Evans <evans@yahoo-inc.com.invalid
> >>> 님이
> >>>>> 작성:
> >>>>> 
> >>>>>> I would love to see the metrics V2 code come out sooner rather
> >> than
> >>>>>> later.  +1.
> >>>>>> My biggest blocker for a 1.x release is
> >>>>>> https://github.com/apache/storm/pull/2142  Even though the pull
> >>>>> request
> >>>>>> says it is minor it showed that we messed up pushing back some
> >>>>> changes for
> >>>>>> pacemaker to open source (the code does not run at all which for
> >> me
> >>>>> is a
> >>>>>> blocker) and I really want to get that fully fixed/tested before
> >>>>> another
> >>>>>> release.
> >>>>>> As for 2.x I think we are very close to being able to so a 2.x
> >>>> alpha
> >>>>>> release.  I would like to see metrics V2 merged in simply because
> >>>> it
> >>>>> is a
> >>>>>> big change for user facing APIs.  But after that I would love to
> >>>> see
> >>>>> us
> >>>>>> starting to push forward on getting that out.
> >>>>>> 
> >>>>>> 
> >>>>>> - Bobby
> >>>>>> 
> >>>>>> 
> >>>>>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> >>>>>> ptgoetz@gmail.com> wrote:
> >>>>>> 
> >>>>>> I’d like to bump this thread and start a discussion around our
> >> next
> >>>>>> release. Here are my thoughts.
> >>>>>> 
> >>>>>> There are a number of important fixes in 1.x-branch so I’d like
> >> to
> >>>>>> consider releasing 1.1.1 soon. I’d appreciate input on any open
> >>>>> issues that
> >>>>>> should be resolved for that release.
> >>>>>> 
> >>>>>> I’d like us to consider releasing the metrics improvements in
> >>>>> STORM-2153
> >>>>>> [1] as version 1.2.0. That work is in the metrics_v2 feature
> >> branch
> >>>>> right
> >>>>>> now and would need to be reviewed and merged. That work is
> >> against
> >>>>> the
> >>>>>> 1.x-branch right now. I would recommend porting it to master
> >>>> *after*
> >>>>> the
> >>>>>> review/merge since there will likely be changes as a result of
> >> the
> >>>>> review.
> >>>>>> 
> >>>>>>> Maybe related to or not, but would we want to create a new
> >> branch
> >>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> If wee agree to the above, I would say yes. After the 1.1.1
> >>>> release,
> >>>>> we
> >>>>>> could create a 1.1.x-branch that would be the maintenance/release
> >>>>> branch
> >>>>>> for that version line. 1.x-branch would then become the target
> >> for
> >>>>> the
> >>>>>> 1.2.0 release.
> >>>>>> 
> >>>>>> There are a few fixes in the 0.10.x branch that probably warrant
> >> a
> >>>>>> release. After that we may want to back away from that version
> >> line
> >>>>> a bit
> >>>>>> so we can focus more on newer versions.
> >>>>>> 
> >>>>>> In the past, we’ve shied away form doing “beta” releases, but I’m
> >>>>>> wondering if we might want to revisit that for the 2.0 release —
> >>>> the
> >>>>> idea
> >>>>>> being that it would give early adopter users a chance to kick the
> >>>>> tires on
> >>>>>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
> >> help
> >>>>> make the
> >>>>>> final release more solid. I’m on the fence here and could go
> >> either
> >>>>> way.
> >>>>>> 
> >>>>>> I’d appreciate any input others may have.
> >>>>>> 
> >>>>>> 
> >>>>>> Thanks,
> >>>>>> 
> >>>>>> -Taylor
> >>>>>> 
> >>>>>> 
> >>>>>> [1] https://issues.apache.org/jira/browse/STORM-2153
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
> >>>>> wrote:
> >>>>>>> 
> >>>>>>> Maybe related to or not, but would we want to create a new
> >> branch
> >>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >>>>>>> 
> >>>>>>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> >> so
> >>>>> hence
> >>>>>> the
> >>>>>>> question.
> >>>>>>> 
> >>>>>>> - Jungtaek Lim (HeartSaVioR)
> >>>>>>> 
> >>>>>>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> >>>>> hlouro@hortonworks.com>님이
> >>>>>> 작성:
> >>>>>>> 
> >>>>>>>> +1 for finishing the porting to Java ahead of anything else -
> >> it
> >>>>> will
> >>>>>> be a
> >>>>>>>> significant milestone. I have a JIRA assigned concerning to
> >> the
> >>>>>> porting. I
> >>>>>>>> will work on it for the 2.0 release.
> >>>>>>>> 
> >>>>>>>> it’s a priority to guarantee no performance regressions. As
> >> part
> >>>>> of this
> >>>>>>>> endeavor, explore an automated (or easy) way to run and assert
> >>>>> major
> >>>>>>>> performance benchmarks. Ideally any contributor should be able
> >>>> to
> >>>>> fairly
> >>>>>>>> easily test the impact of changes under certain performance
> >> test
> >>>>>> scenarios.
> >>>>>>>> 
> >>>>>>>> Beam Runner work should take into account the impact of
> >>>>> incorporating
> >>>>>> new
> >>>>>>>> JStorm features and Storm Worker Redesign<
> >>>>>>>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> >>>>> efficient
> >>>>>> to
> >>>>>>>> start doing it, to  find out that it will have to chance in
> >> face
> >>>>> of
> >>>>>> Storm
> >>>>>>>> and worker redesign. That is, it should be done after it’s
> >>>>> building
> >>>>>> blocks
> >>>>>>>> are stable.
> >>>>>>>> 
> >>>>>>>> Thanks,
> >>>>>>>> Hugo
> >>>>>>>> 
> >>>>>>>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> >> arunm@apache.org
> >>>>> <mailto:
> >>>>>>>> arunm@apache.org>> wrote:
> >>>>>>>> 
> >>>>>>>> +1 to release with the porting completed. I think its mainly
> >> the
> >>>>> UI
> >>>>>> server
> >>>>>>>> and log viewer that’s pending.
> >>>>>>>> 
> >>>>>>>> We can start doing the regression and performance tests for
> >>>>> whatever is
> >>>>>>>> already ported.
> >>>>>>>> 
> >>>>>>>> If anyone is running the master branch in their pre-prod /
> >> prod
> >>>>>>>> environments, it will be good to know and give us more
> >>>> confidence.
> >>>>>>>> 
> >>>>>>>> The other features can be added in follow up releases.
> >>>>>>>> 
> >>>>>>>> Regards,
> >>>>>>>> Arun
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> On 3/24/17, 11:47 AM, "Satish Duggana" <
> >>>> satish.duggana@gmail.com
> >>>>>> <mailto:
> >>>>>>>> satish.duggana@gmail.com>> wrote:
> >>>>>>>> 
> >>>>>>>> +1 to have 2.0 with porting and performance(it should be at
> >>>> least
> >>>>> as
> >>>>>> good
> >>>>>>>> as 1.x release) issues addressed
> >>>>>>>> 
> >>>>>>>> We can target other tasks(mentioned by Taylor and Jungtaek)
> >> for
> >>>>>> 2.x-branch.
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> Exactly-once support:
> >>>>>>>> While thinking through the exactlyonce support design, it is
> >>>>> realized
> >>>>>>>> better to avoid acking tuples and implement exactly once by
> >>>>> snapshotting
> >>>>>>>> barriers. It seems JStorm folks followed similar design, they
> >>>>> claim it
> >>>>>>>> gives better performance. This feature is essential for beam
> >>>>> runner and
> >>>>>> we
> >>>>>>>> can decide on respective approaches though.
> >>>>>>>> 
> >>>>>>>> Beam Runner
> >>>>>>>> Lets hold on this for now and keep it in Storm till 2.x. We
> >>>>> should avoid
> >>>>>>>> having a minimal beam runner in haste. It is better to address
> >>>>>> STORM-2284,
> >>>>>>>> exactly-once and other windowing enhancements to enable beam
> >>>>> runner.
> >>>>>>>> 
> >>>>>>>> JStorm
> >>>>>>>> Agree with Jungtaek on looking at the latest JStorm and
> >>>>> align/scope with
> >>>>>>>> the features for 2.x.
> >>>>>>>> 
> >>>>>>>> STORM-2284
> >>>>>>>> We may want to look at JStorm worker before working on
> >>>> respective
> >>>>>>>> components in this epic to pull appropriate enhancements.
> >>>>>>>> 
> >>>>>>>> YARN/MESOS
> >>>>>>>> Supporting Storm on YARN/Mesos for 2.x.
> >>>>>>>> 
> >>>>>>>> Thanks,
> >>>>>>>> Satish.
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> >>>> kabhwan@gmail.com
> >>>>>> <mailto:
> >>>>>>>> kabhwan@gmail.com>> wrote:
> >>>>>>>> 
> >>>>>>>> First of all, +1 to complete only port work and do sanity
> >> check
> >>>>>> (including
> >>>>>>>> performance regression), and release.
> >>>>>>>> 
> >>>>>>>> If we can get STORM-2284 within deterministic time frame (say
> >>>> 2~3
> >>>>>> months)
> >>>>>>>> that should be great, but if not I'd in favor of postponing
> >> that
> >>>>> to
> >>>>>> later
> >>>>>>>> 2.x release.
> >>>>>>>> 
> >>>>>>>> JStorm released their new versions after code donation. So
> >>>>> there're more
> >>>>>>>> things we could get ideas from, or even adopt from.
> >>>>>>>> https://github.com/alibaba/jstorm/blob/master/history.md
> >>>>>>>> As you noticed from release note link, we also need to update
> >>>>> phase 2
> >>>>>> since
> >>>>>>>> they already changed what we're planning to do in phase 2. For
> >>>>> example,
> >>>>>>>> they changed backpressure to end-to-end, and changed to use
> >>>>> snapshot
> >>>>>> rather
> >>>>>>>> than acker.
> >>>>>>>> May be sure, JStorm pulled many features from today's Storm,
> >>>> like
> >>>>> Flux,
> >>>>>>>> Windowing, more shuffle groupings, log search, log level
> >> change,
> >>>>> and so
> >>>>>> on.
> >>>>>>>> 
> >>>>>>>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> >>>> is
> >>>>> due to
> >>>>>>>> the
> >>>>>>>> limitation of Spout lifecycle (all the things are done in
> >> single
> >>>>>> thread),
> >>>>>>>> and STORM-1358 <
> >>>> https://issues.apache.org/jira/browse/STORM-1358
> >>>>>>> (JStorm's
> >>>>>>>> multi-thread Spout) can remedy this (despite that Spout
> >>>>> implementation
> >>>>>> may
> >>>>>>>> need to guarantee thread-safety later). It's not a just
> >>>>> improvement but
> >>>>>>>> close to design concern so would like to address sooner than
> >>>> other
> >>>>>> things
> >>>>>>>> in phase 2.
> >>>>>>>> 
> >>>>>>>> For Storm SQL side, I've lost progress but major work would be
> >>>>> adopting
> >>>>>>>> group by with windowing. It was not available from Calcite but
> >>>>> will be
> >>>>>>>> available at next release (1.12.0).
> >>>>>>>> I've filed this to STORM-2405
> >>>>>>>> <https://issues.apache.org/jira/browse/STORM-2405>, but
> >>>>> windowing &
> >>>>>> micro
> >>>>>>>> batch is not intuitive, so I would like to change the
> >> underlying
> >>>>> API to
> >>>>>>>> stream API in SQL. Also filed this to STORM-2406
> >>>>>>>> <https://issues.apache.org/jira/browse/STORM-2406>.
> >>>>>>>> 
> >>>>>>>> Just 2 cents btw, hopefully I would like to see metrics V2
> >>>> sooner
> >>>>> since
> >>>>>> we
> >>>>>>>> lost metrics even when doing normal operation like restarting
> >>>>> worker,
> >>>>>>>> rebalancing, and so on. Eventually we need to fight with
> >> dynamic
> >>>>>> scaling,
> >>>>>>>> and then metrics will be broken often.
> >>>>>>>> 
> >>>>>>>> Thanks,
> >>>>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>>>> 
> >>>>>>>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> >> storm@harsha.io
> >>>>> 님이
> >>>>> 작성:
> >>>>>>>> 
> >>>>>>>> Storm 2.0 migration to java in itself is a big win and would
> >>>>> attract
> >>>>>>>> wider
> >>>>>>>> community and adoption. So my vote would be to resolve the
> >> first
> >>>>> 3 items
> >>>>>>>> to
> >>>>>>>> get a release out.
> >>>>>>>> All the other featured mentioned are great to have but
> >> shouldn't
> >>>>> be
> >>>>>>>> blockers for 2.0 release.
> >>>>>>>> 
> >>>>>>>> -Harsha
> >>>>>>>> 
> >>>>>>>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> >>>>> ptgoetz@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>> 
> >>>>>>>> With the 1.1.0 release nearing completion, I’d like to turn
> >> our
> >>>>>>>> attention
> >>>>>>>> to 2.0 and develop a plan for what features, etc. to include.
> >>>>>>>> 
> >>>>>>>> The following 3 are what I feel are the minimum for a 2.0
> >>>> release.
> >>>>>>>> These
> >>>>>>>> could likely be resolved relatively quickly:
> >>>>>>>> 
> >>>>>>>> * Performance — I’ve not benchmarked the master branch vs.
> >> 1.0.x
> >>>>> or
> >>>>>>>> 1.1.x
> >>>>>>>> in a while, but I feel it will be important to make sure there
> >>>>> are no
> >>>>>>>> performance regressions, and would hope that we actually have
> >> a
> >>>>>>>> performance
> >>>>>>>> improvement over previous versions. To that end (e.g. if there
> >>>> is
> >>>>> in
> >>>>>>>> fact a
> >>>>>>>> performance regression), the proposals that Roshan Naik put
> >>>>> together
> >>>>>>>> for
> >>>>>>>> revising the threading and execution model (STORM-2307) and
> >>>>> replacing
> >>>>>>>> Disruptor with JCTools (STORM-2306) warrant review and
> >>>>> consideration.
> >>>>>>>> See
> >>>>>>>> also STORM-2284 which is the parent JIRA.
> >>>>>>>> 
> >>>>>>>> * Finish porting Storm UI to java (STORM-1311)
> >>>>>>>> 
> >>>>>>>> * Finish porting log viewer to java (STORM-1280)
> >>>>>>>> 
> >>>>>>>> The following are items that are nice to have in 2.0, but I
> >>>> don’t
> >>>>> feel
> >>>>>>>> are
> >>>>>>>> absolutely necessary for an initial 2.0 release:
> >>>>>>>> 
> >>>>>>>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> >> because
> >>>>> it’s
> >>>>>>>> relevant) — Initially there seemed to be a lot of interest in
> >>>>> this, but
> >>>>>>>> that seems to have trailed off. I spoke with some Beam
> >>>> developers
> >>>>> and
> >>>>>>>> there
> >>>>>>>> seems to be interest from that community as well. Do we want
> >> to
> >>>>> move
> >>>>>>>> that
> >>>>>>>> effort to the Beam community, or keep it here? Moving it to
> >> the
> >>>>> Beam
> >>>>>>>> community might lead to better collaboration between projects.
> >>>>>>>> 
> >>>>>>>> * Bounded Spouts (needed for Beam Runner implementation) —
> >>>>> Currently
> >>>>>>>> spouts are unbounded, there no end to the stream. Beam has the
> >>>>> concept
> >>>>>>>> of
> >>>>>>>> bounded sources (roughly analogous to batch processing). To
> >>>>> support
> >>>>>>>> that,
> >>>>>>>> we would need to implement a similar concept in Storm. One
> >>>>> benefit of
> >>>>>>>> such
> >>>>>>>> a feature would be the ability to handle both bounded and
> >>>>> unbounded
> >>>>>>>> workflows in Storm.
> >>>>>>>> 
> >>>>>>>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> >>>>> behind
> >>>>>>>> this
> >>>>>>>> effort. What improvements do you envision for 2.0?
> >>>>>>>> 
> >>>>>>>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> >>>>> targeting this
> >>>>>>>> for 1.2.0, but it’s designed to be easily portable to
> >>>> master/2.0.
> >>>>>>>> 
> >>>>>>>> * JStorm Migration — Original outline can be found here [1].
> >>>> Note
> >>>>> a lot
> >>>>>>>> of
> >>>>>>>> the associated JIRAs below are assigned, but there hasn’t been
> >>>> any
> >>>>>>>> recent
> >>>>>>>> activity or pull requests, we should probably consider them
> >>>>> unassigned
> >>>>>>>> and
> >>>>>>>> up for grabs.:
> >>>>>>>> 
> >>>>>>>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> >>>>> been the
> >>>>>>>> bane of a lot of Storm users almost since day one. We have
> >>>> largely
> >>>>>>>> addressed it by shading/relocating dependencies. It would be
> >>>>> great to
> >>>>>>>> see
> >>>>>>>> this addressed once and for all.
> >>>>>>>> 
> >>>>>>>> * JStorm back pressure implementation (STORM-1324) — The
> >> current
> >>>>> back
> >>>>>>>> pressure implementation leaves a bit to be desired, and the
> >>>> JStorm
> >>>>>>>> approach
> >>>>>>>> looks promising, though it also depends on the JStorm concept
> >> of
> >>>>>>>> “topology
> >>>>>>>> master” (STORM-1323), which may have some implications
> >> regarding
> >>>>>>>> security.
> >>>>>>>> 
> >>>>>>>> * Dynamic Topology Updates (STORM-1335) — This would provide a
> >>>>> command
> >>>>>>>> to
> >>>>>>>> update topology jars and configuration without stopping the
> >>>>> topology,
> >>>>>>>> and
> >>>>>>>> is well suited to leverage the blobstore. The restart command
> >>>>> (that can
> >>>>>>>> also update the topology configuration) also looks compelling
> >>>>>>>> (STORM-1334).
> >>>>>>>> 
> >>>>>>>> * Additional Scheduler Implementations (STORM-1320)
> >>>>>>>> 
> >>>>>>>> * Additional Grouping Implementations (STORM-1328)
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> As always I’m open to any opinions and suggestions.
> >>>>>>>> 
> >>>>>>>> -Taylor
> >>>>>>>> 
> >>>>>>>> [1]
> >>>>>>>> 
> >>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.
> >>>>>>>> action?pageId=61328109
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>>> 
> >> 
> >> 
> 

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
The work on metrics is ready for a pull request to 1.x-branch from the feature branch. I’ve held off because we haven’t reached consensus on a path forward with the 1.x release lines .

I’d like to propose the following for the 1.x line:

1. Create a branch for 1.2 so we have a branch to review the metrics stuff.
2. Release 1.1.1
3. Review/merge metrics work. Port metrics to master.
4. Release 1.2.0
5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x. (we would only support 1.2.x and 1.1.x which are very closely aligned).

Dropping support for 1.0.x line would eliminate the need to maintain one of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be very closely aligned. I just up merged metrics_v2 against 1.x-branch after a while, and there were no conflicts.

That would give us a little more bandwidth to focus on 2.0 and needed bug fixes to the 1.x line like some of the issues raised with storm-kafka-client. We could even start releasing alpha/beta versions of 2.0 in parallel to the steps above.

Any thoughts on that approach?

-Taylor


> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> Yes I prefer option 1, but it might depend on the progress of metrics V2.
> If it can be done within predictable near future I'm OK to pick option 2,
> but if not, we may be better to focus releasing 2.0.0 and make it really
> happen.
> 
> Whichever we go, I feel it's time to track remaining work on Storm 2.0.0. I
> found some bugs on master branch so filed issues, and we've remaining port
> work (UI and logviewer). We've some other improvements target for 2.0.0:
> worker redesign, beam integration, and so on, and we don't track its
> progress at all. I don't think we should wait for features which progress
> is not transparent (in other words we don't know when it will be finished).
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>님이 작성:
> 
>> Bobby/Jungtaek,
>> 
>> Are you saying you want to forego the 1.2 “metrics_v2” release and include
>> it only in 2.0? (I ask because that work is already based on 1.x-branch,
>> and forward-porting it to master is relatively simple.) I’d kind of like
>> that work go out soon.
>> 
>> If we go with option 1, I would want to see a 2.0 release (even if it’s a
>> “beta” or “preview) before putting the 1.x line into maintenance mode.
>> 
>> -Taylor
>> 
>>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
>> wrote:
>>> 
>>> I see 2 ways to address this.
>>> 1) We put the 1.x line into maintenance mode like with 0.10.  We don't
>> backport anything except bug fixes.2) We backport a lot of the backwards
>> compatible changes from 2.x to 1.x.
>>> My personal preference is 1.  It makes it clear the direction we want to
>> go in.  The biggest issue is that we probably would want to do a 2.x
>> release sooner rather then later.  Even if we don't get all of the features
>> that people want, if we just get a release out we can add in new features
>> if they are backwards compatible, or we can create a 3.x line that would
>> have the breaking changes in it.
>>> 
>>> - Bobby
>>> 
>>> 
>>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
>> kabhwan@gmail.com> wrote:
>>> 
>>> I'd like to bump this again instead of initiating new discussion thread.
>>> 
>>> I had having hard time to create and apply pull requests for both master
>>> and 1.x-branch and that's really painful and sometimes blocker for me to
>> do
>>> merge step.
>>> Two branches are heavily diverged more than between 0.10 and 1.0.0, even
>>> IDE can't switch between the branch smoothly. We didn't even address
>>> checkstyle issue yet, but after addressing, it could be "completely"
>>> diverged. JDK version is another major issue, since the pull requests
>>> targeted for master branch are not checked against JDK 7, and some of
>> them
>>> make some issues regarding JDK version while porting back.
>>> 
>>> So personally I really would like to see the plan for 1.x version line
>>> changed - skipping any minor releases including 1.2.0 - and have epic
>> issue
>>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
>>> proposed plan was having 2.0.0 directly after 1.0.0)
>>> 
>>> Would like to hear everyone's opinions. If we have consensus to not
>> having
>>> any minor releases for 1.x version line, I will not port back non-bugfix
>>> pull requests to 1.x-branch, and guide contributors to create pull
>> requests
>>> against master branch, not 1.x version line.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
>> avermeerbergen@gmail.com>님이
>>> 작성:
>>> 
>>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
>>>> we're very interested anything that can provide better throughput.
>>>> 
>>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
>>>> 
>>>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
>>>>> delivered sooner and my in-progress implementation suggests that it
>> will
>>>>> yield substantial latency improvements. The 2.0 beta phase would really
>>>>> help kick the tires on the revised messaging system and the performance
>>>>> improvements will also be a good incentive for trying out the 2.0 line.
>>>>> 
>>>>> I notice multiple other bottlenecks that are holding back throughput a
>>>>> lot, which can be addressed in a subsequent 2.x minor release.
>>>>> -roshan
>>>>> 
>>>>> 
>>>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>>> 
>>>>>    I also would love to see metrics V2 code sooner or later too. If we
>>>>> can get
>>>>>    it before releasing 2.0.0 that will be great, and then maybe we
>> could
>>>>> just
>>>>>    move toward to 2.0.0, not adding any improvements to 1.x version
>>>> line.
>>>>>    (And that's what I would want to.)
>>>>> 
>>>>>    If we would really want to have 1.2.0, I suggest that we make the
>>>> 1.1.1
>>>>>    version correct right now rather than after releasing 1.1.1. We
>> also
>>>>> merged
>>>>>    non-bugfix things to 1.x-branch but that's not what users expect. I
>>>>> agree
>>>>>    that work may be painful, but anyway need to do it.
>>>>> 
>>>>>    - Jungtaek Lim (HeartSaVioR)
>>>>> 
>>>>>    2017년 6월 3일 (토) 오전 3:49, Bobby Evans <evans@yahoo-inc.com.invalid
>>> 님이
>>>>> 작성:
>>>>> 
>>>>>> I would love to see the metrics V2 code come out sooner rather
>> than
>>>>>> later.  +1.
>>>>>> My biggest blocker for a 1.x release is
>>>>>> https://github.com/apache/storm/pull/2142  Even though the pull
>>>>> request
>>>>>> says it is minor it showed that we messed up pushing back some
>>>>> changes for
>>>>>> pacemaker to open source (the code does not run at all which for
>> me
>>>>> is a
>>>>>> blocker) and I really want to get that fully fixed/tested before
>>>>> another
>>>>>> release.
>>>>>> As for 2.x I think we are very close to being able to so a 2.x
>>>> alpha
>>>>>> release.  I would like to see metrics V2 merged in simply because
>>>> it
>>>>> is a
>>>>>> big change for user facing APIs.  But after that I would love to
>>>> see
>>>>> us
>>>>>> starting to push forward on getting that out.
>>>>>> 
>>>>>> 
>>>>>> - Bobby
>>>>>> 
>>>>>> 
>>>>>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>>>>>> ptgoetz@gmail.com> wrote:
>>>>>> 
>>>>>> I’d like to bump this thread and start a discussion around our
>> next
>>>>>> release. Here are my thoughts.
>>>>>> 
>>>>>> There are a number of important fixes in 1.x-branch so I’d like
>> to
>>>>>> consider releasing 1.1.1 soon. I’d appreciate input on any open
>>>>> issues that
>>>>>> should be resolved for that release.
>>>>>> 
>>>>>> I’d like us to consider releasing the metrics improvements in
>>>>> STORM-2153
>>>>>> [1] as version 1.2.0. That work is in the metrics_v2 feature
>> branch
>>>>> right
>>>>>> now and would need to be reviewed and merged. That work is
>> against
>>>>> the
>>>>>> 1.x-branch right now. I would recommend porting it to master
>>>> *after*
>>>>> the
>>>>>> review/merge since there will likely be changes as a result of
>> the
>>>>> review.
>>>>>> 
>>>>>>> Maybe related to or not, but would we want to create a new
>> branch
>>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> If wee agree to the above, I would say yes. After the 1.1.1
>>>> release,
>>>>> we
>>>>>> could create a 1.1.x-branch that would be the maintenance/release
>>>>> branch
>>>>>> for that version line. 1.x-branch would then become the target
>> for
>>>>> the
>>>>>> 1.2.0 release.
>>>>>> 
>>>>>> There are a few fixes in the 0.10.x branch that probably warrant
>> a
>>>>>> release. After that we may want to back away from that version
>> line
>>>>> a bit
>>>>>> so we can focus more on newer versions.
>>>>>> 
>>>>>> In the past, we’ve shied away form doing “beta” releases, but I’m
>>>>>> wondering if we might want to revisit that for the 2.0 release —
>>>> the
>>>>> idea
>>>>>> being that it would give early adopter users a chance to kick the
>>>>> tires on
>>>>>> what’s coming in 2.0 and provide feedback, find bugs, etc. to
>> help
>>>>> make the
>>>>>> final release more solid. I’m on the fence here and could go
>> either
>>>>> way.
>>>>>> 
>>>>>> I’d appreciate any input others may have.
>>>>>> 
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> -Taylor
>>>>>> 
>>>>>> 
>>>>>> [1] https://issues.apache.org/jira/browse/STORM-2153
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
>>>>> wrote:
>>>>>>> 
>>>>>>> Maybe related to or not, but would we want to create a new
>> branch
>>>>>>> "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>>>>> 
>>>>>>> I'm not clear we don't release 1.2 for moving toward to 2.0.0,
>> so
>>>>> hence
>>>>>> the
>>>>>>> question.
>>>>>>> 
>>>>>>> - Jungtaek Lim (HeartSaVioR)
>>>>>>> 
>>>>>>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
>>>>> hlouro@hortonworks.com>님이
>>>>>> 작성:
>>>>>>> 
>>>>>>>> +1 for finishing the porting to Java ahead of anything else -
>> it
>>>>> will
>>>>>> be a
>>>>>>>> significant milestone. I have a JIRA assigned concerning to
>> the
>>>>>> porting. I
>>>>>>>> will work on it for the 2.0 release.
>>>>>>>> 
>>>>>>>> it’s a priority to guarantee no performance regressions. As
>> part
>>>>> of this
>>>>>>>> endeavor, explore an automated (or easy) way to run and assert
>>>>> major
>>>>>>>> performance benchmarks. Ideally any contributor should be able
>>>> to
>>>>> fairly
>>>>>>>> easily test the impact of changes under certain performance
>> test
>>>>>> scenarios.
>>>>>>>> 
>>>>>>>> Beam Runner work should take into account the impact of
>>>>> incorporating
>>>>>> new
>>>>>>>> JStorm features and Storm Worker Redesign<
>>>>>>>> https://issues.apache.org/jira/browse/STORM-2284>. Not very
>>>>> efficient
>>>>>> to
>>>>>>>> start doing it, to  find out that it will have to chance in
>> face
>>>>> of
>>>>>> Storm
>>>>>>>> and worker redesign. That is, it should be done after it’s
>>>>> building
>>>>>> blocks
>>>>>>>> are stable.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Hugo
>>>>>>>> 
>>>>>>>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
>> arunm@apache.org
>>>>> <mailto:
>>>>>>>> arunm@apache.org>> wrote:
>>>>>>>> 
>>>>>>>> +1 to release with the porting completed. I think its mainly
>> the
>>>>> UI
>>>>>> server
>>>>>>>> and log viewer that’s pending.
>>>>>>>> 
>>>>>>>> We can start doing the regression and performance tests for
>>>>> whatever is
>>>>>>>> already ported.
>>>>>>>> 
>>>>>>>> If anyone is running the master branch in their pre-prod /
>> prod
>>>>>>>> environments, it will be good to know and give us more
>>>> confidence.
>>>>>>>> 
>>>>>>>> The other features can be added in follow up releases.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Arun
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 3/24/17, 11:47 AM, "Satish Duggana" <
>>>> satish.duggana@gmail.com
>>>>>> <mailto:
>>>>>>>> satish.duggana@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> +1 to have 2.0 with porting and performance(it should be at
>>>> least
>>>>> as
>>>>>> good
>>>>>>>> as 1.x release) issues addressed
>>>>>>>> 
>>>>>>>> We can target other tasks(mentioned by Taylor and Jungtaek)
>> for
>>>>>> 2.x-branch.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Exactly-once support:
>>>>>>>> While thinking through the exactlyonce support design, it is
>>>>> realized
>>>>>>>> better to avoid acking tuples and implement exactly once by
>>>>> snapshotting
>>>>>>>> barriers. It seems JStorm folks followed similar design, they
>>>>> claim it
>>>>>>>> gives better performance. This feature is essential for beam
>>>>> runner and
>>>>>> we
>>>>>>>> can decide on respective approaches though.
>>>>>>>> 
>>>>>>>> Beam Runner
>>>>>>>> Lets hold on this for now and keep it in Storm till 2.x. We
>>>>> should avoid
>>>>>>>> having a minimal beam runner in haste. It is better to address
>>>>>> STORM-2284,
>>>>>>>> exactly-once and other windowing enhancements to enable beam
>>>>> runner.
>>>>>>>> 
>>>>>>>> JStorm
>>>>>>>> Agree with Jungtaek on looking at the latest JStorm and
>>>>> align/scope with
>>>>>>>> the features for 2.x.
>>>>>>>> 
>>>>>>>> STORM-2284
>>>>>>>> We may want to look at JStorm worker before working on
>>>> respective
>>>>>>>> components in this epic to pull appropriate enhancements.
>>>>>>>> 
>>>>>>>> YARN/MESOS
>>>>>>>> Supporting Storm on YARN/Mesos for 2.x.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Satish.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
>>>> kabhwan@gmail.com
>>>>>> <mailto:
>>>>>>>> kabhwan@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> First of all, +1 to complete only port work and do sanity
>> check
>>>>>> (including
>>>>>>>> performance regression), and release.
>>>>>>>> 
>>>>>>>> If we can get STORM-2284 within deterministic time frame (say
>>>> 2~3
>>>>>> months)
>>>>>>>> that should be great, but if not I'd in favor of postponing
>> that
>>>>> to
>>>>>> later
>>>>>>>> 2.x release.
>>>>>>>> 
>>>>>>>> JStorm released their new versions after code donation. So
>>>>> there're more
>>>>>>>> things we could get ideas from, or even adopt from.
>>>>>>>> https://github.com/alibaba/jstorm/blob/master/history.md
>>>>>>>> As you noticed from release note link, we also need to update
>>>>> phase 2
>>>>>> since
>>>>>>>> they already changed what we're planning to do in phase 2. For
>>>>> example,
>>>>>>>> they changed backpressure to end-to-end, and changed to use
>>>>> snapshot
>>>>>> rather
>>>>>>>> than acker.
>>>>>>>> May be sure, JStorm pulled many features from today's Storm,
>>>> like
>>>>> Flux,
>>>>>>>> Windowing, more shuffle groupings, log search, log level
>> change,
>>>>> and so
>>>>>> on.
>>>>>>>> 
>>>>>>>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
>>>> is
>>>>> due to
>>>>>>>> the
>>>>>>>> limitation of Spout lifecycle (all the things are done in
>> single
>>>>>> thread),
>>>>>>>> and STORM-1358 <
>>>> https://issues.apache.org/jira/browse/STORM-1358
>>>>>>> (JStorm's
>>>>>>>> multi-thread Spout) can remedy this (despite that Spout
>>>>> implementation
>>>>>> may
>>>>>>>> need to guarantee thread-safety later). It's not a just
>>>>> improvement but
>>>>>>>> close to design concern so would like to address sooner than
>>>> other
>>>>>> things
>>>>>>>> in phase 2.
>>>>>>>> 
>>>>>>>> For Storm SQL side, I've lost progress but major work would be
>>>>> adopting
>>>>>>>> group by with windowing. It was not available from Calcite but
>>>>> will be
>>>>>>>> available at next release (1.12.0).
>>>>>>>> I've filed this to STORM-2405
>>>>>>>> <https://issues.apache.org/jira/browse/STORM-2405>, but
>>>>> windowing &
>>>>>> micro
>>>>>>>> batch is not intuitive, so I would like to change the
>> underlying
>>>>> API to
>>>>>>>> stream API in SQL. Also filed this to STORM-2406
>>>>>>>> <https://issues.apache.org/jira/browse/STORM-2406>.
>>>>>>>> 
>>>>>>>> Just 2 cents btw, hopefully I would like to see metrics V2
>>>> sooner
>>>>> since
>>>>>> we
>>>>>>>> lost metrics even when doing normal operation like restarting
>>>>> worker,
>>>>>>>> rebalancing, and so on. Eventually we need to fight with
>> dynamic
>>>>>> scaling,
>>>>>>>> and then metrics will be broken often.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>>>> 
>>>>>>>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
>> storm@harsha.io
>>>>> 님이
>>>>> 작성:
>>>>>>>> 
>>>>>>>> Storm 2.0 migration to java in itself is a big win and would
>>>>> attract
>>>>>>>> wider
>>>>>>>> community and adoption. So my vote would be to resolve the
>> first
>>>>> 3 items
>>>>>>>> to
>>>>>>>> get a release out.
>>>>>>>> All the other featured mentioned are great to have but
>> shouldn't
>>>>> be
>>>>>>>> blockers for 2.0 release.
>>>>>>>> 
>>>>>>>> -Harsha
>>>>>>>> 
>>>>>>>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
>>>>> ptgoetz@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> With the 1.1.0 release nearing completion, I’d like to turn
>> our
>>>>>>>> attention
>>>>>>>> to 2.0 and develop a plan for what features, etc. to include.
>>>>>>>> 
>>>>>>>> The following 3 are what I feel are the minimum for a 2.0
>>>> release.
>>>>>>>> These
>>>>>>>> could likely be resolved relatively quickly:
>>>>>>>> 
>>>>>>>> * Performance — I’ve not benchmarked the master branch vs.
>> 1.0.x
>>>>> or
>>>>>>>> 1.1.x
>>>>>>>> in a while, but I feel it will be important to make sure there
>>>>> are no
>>>>>>>> performance regressions, and would hope that we actually have
>> a
>>>>>>>> performance
>>>>>>>> improvement over previous versions. To that end (e.g. if there
>>>> is
>>>>> in
>>>>>>>> fact a
>>>>>>>> performance regression), the proposals that Roshan Naik put
>>>>> together
>>>>>>>> for
>>>>>>>> revising the threading and execution model (STORM-2307) and
>>>>> replacing
>>>>>>>> Disruptor with JCTools (STORM-2306) warrant review and
>>>>> consideration.
>>>>>>>> See
>>>>>>>> also STORM-2284 which is the parent JIRA.
>>>>>>>> 
>>>>>>>> * Finish porting Storm UI to java (STORM-1311)
>>>>>>>> 
>>>>>>>> * Finish porting log viewer to java (STORM-1280)
>>>>>>>> 
>>>>>>>> The following are items that are nice to have in 2.0, but I
>>>> don’t
>>>>> feel
>>>>>>>> are
>>>>>>>> absolutely necessary for an initial 2.0 release:
>>>>>>>> 
>>>>>>>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
>> because
>>>>> it’s
>>>>>>>> relevant) — Initially there seemed to be a lot of interest in
>>>>> this, but
>>>>>>>> that seems to have trailed off. I spoke with some Beam
>>>> developers
>>>>> and
>>>>>>>> there
>>>>>>>> seems to be interest from that community as well. Do we want
>> to
>>>>> move
>>>>>>>> that
>>>>>>>> effort to the Beam community, or keep it here? Moving it to
>> the
>>>>> Beam
>>>>>>>> community might lead to better collaboration between projects.
>>>>>>>> 
>>>>>>>> * Bounded Spouts (needed for Beam Runner implementation) —
>>>>> Currently
>>>>>>>> spouts are unbounded, there no end to the stream. Beam has the
>>>>> concept
>>>>>>>> of
>>>>>>>> bounded sources (roughly analogous to batch processing). To
>>>>> support
>>>>>>>> that,
>>>>>>>> we would need to implement a similar concept in Storm. One
>>>>> benefit of
>>>>>>>> such
>>>>>>>> a feature would be the ability to handle both bounded and
>>>>> unbounded
>>>>>>>> workflows in Storm.
>>>>>>>> 
>>>>>>>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
>>>>> behind
>>>>>>>> this
>>>>>>>> effort. What improvements do you envision for 2.0?
>>>>>>>> 
>>>>>>>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
>>>>> targeting this
>>>>>>>> for 1.2.0, but it’s designed to be easily portable to
>>>> master/2.0.
>>>>>>>> 
>>>>>>>> * JStorm Migration — Original outline can be found here [1].
>>>> Note
>>>>> a lot
>>>>>>>> of
>>>>>>>> the associated JIRAs below are assigned, but there hasn’t been
>>>> any
>>>>>>>> recent
>>>>>>>> activity or pull requests, we should probably consider them
>>>>> unassigned
>>>>>>>> and
>>>>>>>> up for grabs.:
>>>>>>>> 
>>>>>>>> * Worker Classloader Isolation (STORM-1338) — Lack of this has
>>>>> been the
>>>>>>>> bane of a lot of Storm users almost since day one. We have
>>>> largely
>>>>>>>> addressed it by shading/relocating dependencies. It would be
>>>>> great to
>>>>>>>> see
>>>>>>>> this addressed once and for all.
>>>>>>>> 
>>>>>>>> * JStorm back pressure implementation (STORM-1324) — The
>> current
>>>>> back
>>>>>>>> pressure implementation leaves a bit to be desired, and the
>>>> JStorm
>>>>>>>> approach
>>>>>>>> looks promising, though it also depends on the JStorm concept
>> of
>>>>>>>> “topology
>>>>>>>> master” (STORM-1323), which may have some implications
>> regarding
>>>>>>>> security.
>>>>>>>> 
>>>>>>>> * Dynamic Topology Updates (STORM-1335) — This would provide a
>>>>> command
>>>>>>>> to
>>>>>>>> update topology jars and configuration without stopping the
>>>>> topology,
>>>>>>>> and
>>>>>>>> is well suited to leverage the blobstore. The restart command
>>>>> (that can
>>>>>>>> also update the topology configuration) also looks compelling
>>>>>>>> (STORM-1334).
>>>>>>>> 
>>>>>>>> * Additional Scheduler Implementations (STORM-1320)
>>>>>>>> 
>>>>>>>> * Additional Grouping Implementations (STORM-1328)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> As always I’m open to any opinions and suggestions.
>>>>>>>> 
>>>>>>>> -Taylor
>>>>>>>> 
>>>>>>>> [1]
>>>>>>>> 
>>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.
>>>>>>>> action?pageId=61328109
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>> 
>> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Jungtaek Lim <ka...@gmail.com>.
Yes I prefer option 1, but it might depend on the progress of metrics V2.
If it can be done within predictable near future I'm OK to pick option 2,
but if not, we may be better to focus releasing 2.0.0 and make it really
happen.

Whichever we go, I feel it's time to track remaining work on Storm 2.0.0. I
found some bugs on master branch so filed issues, and we've remaining port
work (UI and logviewer). We've some other improvements target for 2.0.0:
worker redesign, beam integration, and so on, and we don't track its
progress at all. I don't think we should wait for features which progress
is not transparent (in other words we don't know when it will be finished).

- Jungtaek Lim (HeartSaVioR)

2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <pt...@gmail.com>님이 작성:

> Bobby/Jungtaek,
>
> Are you saying you want to forego the 1.2 “metrics_v2” release and include
> it only in 2.0? (I ask because that work is already based on 1.x-branch,
> and forward-porting it to master is relatively simple.) I’d kind of like
> that work go out soon.
>
> If we go with option 1, I would want to see a 2.0 release (even if it’s a
> “beta” or “preview) before putting the 1.x line into maintenance mode.
>
> -Taylor
>
> > On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> wrote:
> >
> > I see 2 ways to address this.
> > 1) We put the 1.x line into maintenance mode like with 0.10.  We don't
> backport anything except bug fixes.2) We backport a lot of the backwards
> compatible changes from 2.x to 1.x.
> > My personal preference is 1.  It makes it clear the direction we want to
> go in.  The biggest issue is that we probably would want to do a 2.x
> release sooner rather then later.  Even if we don't get all of the features
> that people want, if we just get a release out we can add in new features
> if they are backwards compatible, or we can create a 3.x line that would
> have the breaking changes in it.
> >
> > - Bobby
> >
> >
> > On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <
> kabhwan@gmail.com> wrote:
> >
> > I'd like to bump this again instead of initiating new discussion thread.
> >
> > I had having hard time to create and apply pull requests for both master
> > and 1.x-branch and that's really painful and sometimes blocker for me to
> do
> > merge step.
> > Two branches are heavily diverged more than between 0.10 and 1.0.0, even
> > IDE can't switch between the branch smoothly. We didn't even address
> > checkstyle issue yet, but after addressing, it could be "completely"
> > diverged. JDK version is another major issue, since the pull requests
> > targeted for master branch are not checked against JDK 7, and some of
> them
> > make some issues regarding JDK version while porting back.
> >
> > So personally I really would like to see the plan for 1.x version line
> > changed - skipping any minor releases including 1.2.0 - and have epic
> issue
> > for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> > proposed plan was having 2.0.0 directly after 1.0.0)
> >
> > Would like to hear everyone's opinions. If we have consensus to not
> having
> > any minor releases for 1.x version line, I will not port back non-bugfix
> > pull requests to 1.x-branch, and guide contributors to create pull
> requests
> > against master branch, not 1.x version line.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>님이
> > 작성:
> >
> >> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
> >> we're very interested anything that can provide better throughput.
> >>
> >> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
> >>
> >>> For 2.0 beta … it would be good to incorporate some of the Worker
> >>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
> >>> delivered sooner and my in-progress implementation suggests that it
> will
> >>> yield substantial latency improvements. The 2.0 beta phase would really
> >>> help kick the tires on the revised messaging system and the performance
> >>> improvements will also be a good incentive for trying out the 2.0 line.
> >>>
> >>> I notice multiple other bottlenecks that are holding back throughput a
> >>> lot, which can be addressed in a subsequent 2.x minor release.
> >>> -roshan
> >>>
> >>>
> >>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >>>
> >>>     I also would love to see metrics V2 code sooner or later too. If we
> >>> can get
> >>>     it before releasing 2.0.0 that will be great, and then maybe we
> could
> >>> just
> >>>     move toward to 2.0.0, not adding any improvements to 1.x version
> >> line.
> >>>     (And that's what I would want to.)
> >>>
> >>>     If we would really want to have 1.2.0, I suggest that we make the
> >> 1.1.1
> >>>     version correct right now rather than after releasing 1.1.1. We
> also
> >>> merged
> >>>     non-bugfix things to 1.x-branch but that's not what users expect. I
> >>> agree
> >>>     that work may be painful, but anyway need to do it.
> >>>
> >>>     - Jungtaek Lim (HeartSaVioR)
> >>>
> >>>     2017년 6월 3일 (토) 오전 3:49, Bobby Evans <evans@yahoo-inc.com.invalid
> >님이
> >>> 작성:
> >>>
> >>>     > I would love to see the metrics V2 code come out sooner rather
> than
> >>>     > later.  +1.
> >>>     > My biggest blocker for a 1.x release is
> >>>     > https://github.com/apache/storm/pull/2142  Even though the pull
> >>> request
> >>>     > says it is minor it showed that we messed up pushing back some
> >>> changes for
> >>>     > pacemaker to open source (the code does not run at all which for
> me
> >>> is a
> >>>     > blocker) and I really want to get that fully fixed/tested before
> >>> another
> >>>     > release.
> >>>     > As for 2.x I think we are very close to being able to so a 2.x
> >> alpha
> >>>     > release.  I would like to see metrics V2 merged in simply because
> >> it
> >>> is a
> >>>     > big change for user facing APIs.  But after that I would love to
> >> see
> >>> us
> >>>     > starting to push forward on getting that out.
> >>>     >
> >>>     >
> >>>     > - Bobby
> >>>     >
> >>>     >
> >>>     > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> >>>     > ptgoetz@gmail.com> wrote:
> >>>     >
> >>>     > I’d like to bump this thread and start a discussion around our
> next
> >>>     > release. Here are my thoughts.
> >>>     >
> >>>     > There are a number of important fixes in 1.x-branch so I’d like
> to
> >>>     > consider releasing 1.1.1 soon. I’d appreciate input on any open
> >>> issues that
> >>>     > should be resolved for that release.
> >>>     >
> >>>     > I’d like us to consider releasing the metrics improvements in
> >>> STORM-2153
> >>>     > [1] as version 1.2.0. That work is in the metrics_v2 feature
> branch
> >>> right
> >>>     > now and would need to be reviewed and merged. That work is
> against
> >>> the
> >>>     > 1.x-branch right now. I would recommend porting it to master
> >> *after*
> >>> the
> >>>     > review/merge since there will likely be changes as a result of
> the
> >>> review.
> >>>     >
> >>>     > > Maybe related to or not, but would we want to create a new
> branch
> >>>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >>>     >
> >>>     >
> >>>     >
> >>>     > If wee agree to the above, I would say yes. After the 1.1.1
> >> release,
> >>> we
> >>>     > could create a 1.1.x-branch that would be the maintenance/release
> >>> branch
> >>>     > for that version line. 1.x-branch would then become the target
> for
> >>> the
> >>>     > 1.2.0 release.
> >>>     >
> >>>     > There are a few fixes in the 0.10.x branch that probably warrant
> a
> >>>     > release. After that we may want to back away from that version
> line
> >>> a bit
> >>>     > so we can focus more on newer versions.
> >>>     >
> >>>     > In the past, we’ve shied away form doing “beta” releases, but I’m
> >>>     > wondering if we might want to revisit that for the 2.0 release —
> >> the
> >>> idea
> >>>     > being that it would give early adopter users a chance to kick the
> >>> tires on
> >>>     > what’s coming in 2.0 and provide feedback, find bugs, etc. to
> help
> >>> make the
> >>>     > final release more solid. I’m on the fence here and could go
> either
> >>> way.
> >>>     >
> >>>     > I’d appreciate any input others may have.
> >>>     >
> >>>     >
> >>>     > Thanks,
> >>>     >
> >>>     > -Taylor
> >>>     >
> >>>     >
> >>>     > [1] https://issues.apache.org/jira/browse/STORM-2153
> >>>     >
> >>>     >
> >>>     >
> >>>     > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
> >>> wrote:
> >>>     > >
> >>>     > > Maybe related to or not, but would we want to create a new
> branch
> >>>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >>>     > >
> >>>     > > I'm not clear we don't release 1.2 for moving toward to 2.0.0,
> so
> >>> hence
> >>>     > the
> >>>     > > question.
> >>>     > >
> >>>     > > - Jungtaek Lim (HeartSaVioR)
> >>>     > >
> >>>     > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> >>> hlouro@hortonworks.com>님이
> >>>     > 작성:
> >>>     > >
> >>>     > >> +1 for finishing the porting to Java ahead of anything else -
> it
> >>> will
> >>>     > be a
> >>>     > >> significant milestone. I have a JIRA assigned concerning to
> the
> >>>     > porting. I
> >>>     > >> will work on it for the 2.0 release.
> >>>     > >>
> >>>     > >> it’s a priority to guarantee no performance regressions. As
> part
> >>> of this
> >>>     > >> endeavor, explore an automated (or easy) way to run and assert
> >>> major
> >>>     > >> performance benchmarks. Ideally any contributor should be able
> >> to
> >>> fairly
> >>>     > >> easily test the impact of changes under certain performance
> test
> >>>     > scenarios.
> >>>     > >>
> >>>     > >> Beam Runner work should take into account the impact of
> >>> incorporating
> >>>     > new
> >>>     > >> JStorm features and Storm Worker Redesign<
> >>>     > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> >>> efficient
> >>>     > to
> >>>     > >> start doing it, to  find out that it will have to chance in
> face
> >>> of
> >>>     > Storm
> >>>     > >> and worker redesign. That is, it should be done after it’s
> >>> building
> >>>     > blocks
> >>>     > >> are stable.
> >>>     > >>
> >>>     > >> Thanks,
> >>>     > >> Hugo
> >>>     > >>
> >>>     > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <
> arunm@apache.org
> >>> <mailto:
> >>>     > >> arunm@apache.org>> wrote:
> >>>     > >>
> >>>     > >> +1 to release with the porting completed. I think its mainly
> the
> >>> UI
> >>>     > server
> >>>     > >> and log viewer that’s pending.
> >>>     > >>
> >>>     > >> We can start doing the regression and performance tests for
> >>> whatever is
> >>>     > >> already ported.
> >>>     > >>
> >>>     > >> If anyone is running the master branch in their pre-prod /
> prod
> >>>     > >> environments, it will be good to know and give us more
> >> confidence.
> >>>     > >>
> >>>     > >> The other features can be added in follow up releases.
> >>>     > >>
> >>>     > >> Regards,
> >>>     > >> Arun
> >>>     > >>
> >>>     > >>
> >>>     > >> On 3/24/17, 11:47 AM, "Satish Duggana" <
> >> satish.duggana@gmail.com
> >>>     > <mailto:
> >>>     > >> satish.duggana@gmail.com>> wrote:
> >>>     > >>
> >>>     > >> +1 to have 2.0 with porting and performance(it should be at
> >> least
> >>> as
> >>>     > good
> >>>     > >> as 1.x release) issues addressed
> >>>     > >>
> >>>     > >> We can target other tasks(mentioned by Taylor and Jungtaek)
> for
> >>>     > 2.x-branch.
> >>>     > >>
> >>>     > >>
> >>>     > >> Exactly-once support:
> >>>     > >> While thinking through the exactlyonce support design, it is
> >>> realized
> >>>     > >> better to avoid acking tuples and implement exactly once by
> >>> snapshotting
> >>>     > >> barriers. It seems JStorm folks followed similar design, they
> >>> claim it
> >>>     > >> gives better performance. This feature is essential for beam
> >>> runner and
> >>>     > we
> >>>     > >> can decide on respective approaches though.
> >>>     > >>
> >>>     > >> Beam Runner
> >>>     > >> Lets hold on this for now and keep it in Storm till 2.x. We
> >>> should avoid
> >>>     > >> having a minimal beam runner in haste. It is better to address
> >>>     > STORM-2284,
> >>>     > >> exactly-once and other windowing enhancements to enable beam
> >>> runner.
> >>>     > >>
> >>>     > >> JStorm
> >>>     > >> Agree with Jungtaek on looking at the latest JStorm and
> >>> align/scope with
> >>>     > >> the features for 2.x.
> >>>     > >>
> >>>     > >> STORM-2284
> >>>     > >> We may want to look at JStorm worker before working on
> >> respective
> >>>     > >> components in this epic to pull appropriate enhancements.
> >>>     > >>
> >>>     > >> YARN/MESOS
> >>>     > >> Supporting Storm on YARN/Mesos for 2.x.
> >>>     > >>
> >>>     > >> Thanks,
> >>>     > >> Satish.
> >>>     > >>
> >>>     > >>
> >>>     > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> >> kabhwan@gmail.com
> >>>     > <mailto:
> >>>     > >> kabhwan@gmail.com>> wrote:
> >>>     > >>
> >>>     > >> First of all, +1 to complete only port work and do sanity
> check
> >>>     > (including
> >>>     > >> performance regression), and release.
> >>>     > >>
> >>>     > >> If we can get STORM-2284 within deterministic time frame (say
> >> 2~3
> >>>     > months)
> >>>     > >> that should be great, but if not I'd in favor of postponing
> that
> >>> to
> >>>     > later
> >>>     > >> 2.x release.
> >>>     > >>
> >>>     > >> JStorm released their new versions after code donation. So
> >>> there're more
> >>>     > >> things we could get ideas from, or even adopt from.
> >>>     > >> https://github.com/alibaba/jstorm/blob/master/history.md
> >>>     > >> As you noticed from release note link, we also need to update
> >>> phase 2
> >>>     > since
> >>>     > >> they already changed what we're planning to do in phase 2. For
> >>> example,
> >>>     > >> they changed backpressure to end-to-end, and changed to use
> >>> snapshot
> >>>     > rather
> >>>     > >> than acker.
> >>>     > >> May be sure, JStorm pulled many features from today's Storm,
> >> like
> >>> Flux,
> >>>     > >> Windowing, more shuffle groupings, log search, log level
> change,
> >>> and so
> >>>     > on.
> >>>     > >>
> >>>     > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> >> is
> >>> due to
> >>>     > >> the
> >>>     > >> limitation of Spout lifecycle (all the things are done in
> single
> >>>     > thread),
> >>>     > >> and STORM-1358 <
> >> https://issues.apache.org/jira/browse/STORM-1358
> >>>     > >(JStorm's
> >>>     > >> multi-thread Spout) can remedy this (despite that Spout
> >>> implementation
> >>>     > may
> >>>     > >> need to guarantee thread-safety later). It's not a just
> >>> improvement but
> >>>     > >> close to design concern so would like to address sooner than
> >> other
> >>>     > things
> >>>     > >> in phase 2.
> >>>     > >>
> >>>     > >> For Storm SQL side, I've lost progress but major work would be
> >>> adopting
> >>>     > >> group by with windowing. It was not available from Calcite but
> >>> will be
> >>>     > >> available at next release (1.12.0).
> >>>     > >> I've filed this to STORM-2405
> >>>     > >> <https://issues.apache.org/jira/browse/STORM-2405>, but
> >>> windowing &
> >>>     > micro
> >>>     > >> batch is not intuitive, so I would like to change the
> underlying
> >>> API to
> >>>     > >> stream API in SQL. Also filed this to STORM-2406
> >>>     > >> <https://issues.apache.org/jira/browse/STORM-2406>.
> >>>     > >>
> >>>     > >> Just 2 cents btw, hopefully I would like to see metrics V2
> >> sooner
> >>> since
> >>>     > we
> >>>     > >> lost metrics even when doing normal operation like restarting
> >>> worker,
> >>>     > >> rebalancing, and so on. Eventually we need to fight with
> dynamic
> >>>     > scaling,
> >>>     > >> and then metrics will be broken often.
> >>>     > >>
> >>>     > >> Thanks,
> >>>     > >> Jungtaek Lim (HeartSaVioR)
> >>>     > >>
> >>>     > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <
> storm@harsha.io
> >>> 님이
> >>> 작성:
> >>>     > >>
> >>>     > >> Storm 2.0 migration to java in itself is a big win and would
> >>> attract
> >>>     > >> wider
> >>>     > >> community and adoption. So my vote would be to resolve the
> first
> >>> 3 items
> >>>     > >> to
> >>>     > >> get a release out.
> >>>     > >> All the other featured mentioned are great to have but
> shouldn't
> >>> be
> >>>     > >> blockers for 2.0 release.
> >>>     > >>
> >>>     > >> -Harsha
> >>>     > >>
> >>>     > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> >>> ptgoetz@gmail.com>
> >>>     > >> wrote:
> >>>     > >>
> >>>     > >> With the 1.1.0 release nearing completion, I’d like to turn
> our
> >>>     > >> attention
> >>>     > >> to 2.0 and develop a plan for what features, etc. to include.
> >>>     > >>
> >>>     > >> The following 3 are what I feel are the minimum for a 2.0
> >> release.
> >>>     > >> These
> >>>     > >> could likely be resolved relatively quickly:
> >>>     > >>
> >>>     > >> * Performance — I’ve not benchmarked the master branch vs.
> 1.0.x
> >>> or
> >>>     > >> 1.1.x
> >>>     > >> in a while, but I feel it will be important to make sure there
> >>> are no
> >>>     > >> performance regressions, and would hope that we actually have
> a
> >>>     > >> performance
> >>>     > >> improvement over previous versions. To that end (e.g. if there
> >> is
> >>> in
> >>>     > >> fact a
> >>>     > >> performance regression), the proposals that Roshan Naik put
> >>> together
> >>>     > >> for
> >>>     > >> revising the threading and execution model (STORM-2307) and
> >>> replacing
> >>>     > >> Disruptor with JCTools (STORM-2306) warrant review and
> >>> consideration.
> >>>     > >> See
> >>>     > >> also STORM-2284 which is the parent JIRA.
> >>>     > >>
> >>>     > >> * Finish porting Storm UI to java (STORM-1311)
> >>>     > >>
> >>>     > >> * Finish porting log viewer to java (STORM-1280)
> >>>     > >>
> >>>     > >> The following are items that are nice to have in 2.0, but I
> >> don’t
> >>> feel
> >>>     > >> are
> >>>     > >> absolutely necessary for an initial 2.0 release:
> >>>     > >>
> >>>     > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it
> because
> >>> it’s
> >>>     > >> relevant) — Initially there seemed to be a lot of interest in
> >>> this, but
> >>>     > >> that seems to have trailed off. I spoke with some Beam
> >> developers
> >>> and
> >>>     > >> there
> >>>     > >> seems to be interest from that community as well. Do we want
> to
> >>> move
> >>>     > >> that
> >>>     > >> effort to the Beam community, or keep it here? Moving it to
> the
> >>> Beam
> >>>     > >> community might lead to better collaboration between projects.
> >>>     > >>
> >>>     > >> * Bounded Spouts (needed for Beam Runner implementation) —
> >>> Currently
> >>>     > >> spouts are unbounded, there no end to the stream. Beam has the
> >>> concept
> >>>     > >> of
> >>>     > >> bounded sources (roughly analogous to batch processing). To
> >>> support
> >>>     > >> that,
> >>>     > >> we would need to implement a similar concept in Storm. One
> >>> benefit of
> >>>     > >> such
> >>>     > >> a feature would be the ability to handle both bounded and
> >>> unbounded
> >>>     > >> workflows in Storm.
> >>>     > >>
> >>>     > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> >>> behind
> >>>     > >> this
> >>>     > >> effort. What improvements do you envision for 2.0?
> >>>     > >>
> >>>     > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> >>> targeting this
> >>>     > >> for 1.2.0, but it’s designed to be easily portable to
> >> master/2.0.
> >>>     > >>
> >>>     > >> * JStorm Migration — Original outline can be found here [1].
> >> Note
> >>> a lot
> >>>     > >> of
> >>>     > >> the associated JIRAs below are assigned, but there hasn’t been
> >> any
> >>>     > >> recent
> >>>     > >> activity or pull requests, we should probably consider them
> >>> unassigned
> >>>     > >> and
> >>>     > >> up for grabs.:
> >>>     > >>
> >>>     > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> >>> been the
> >>>     > >> bane of a lot of Storm users almost since day one. We have
> >> largely
> >>>     > >> addressed it by shading/relocating dependencies. It would be
> >>> great to
> >>>     > >> see
> >>>     > >> this addressed once and for all.
> >>>     > >>
> >>>     > >> * JStorm back pressure implementation (STORM-1324) — The
> current
> >>> back
> >>>     > >> pressure implementation leaves a bit to be desired, and the
> >> JStorm
> >>>     > >> approach
> >>>     > >> looks promising, though it also depends on the JStorm concept
> of
> >>>     > >> “topology
> >>>     > >> master” (STORM-1323), which may have some implications
> regarding
> >>>     > >> security.
> >>>     > >>
> >>>     > >> * Dynamic Topology Updates (STORM-1335) — This would provide a
> >>> command
> >>>     > >> to
> >>>     > >> update topology jars and configuration without stopping the
> >>> topology,
> >>>     > >> and
> >>>     > >> is well suited to leverage the blobstore. The restart command
> >>> (that can
> >>>     > >> also update the topology configuration) also looks compelling
> >>>     > >> (STORM-1334).
> >>>     > >>
> >>>     > >> * Additional Scheduler Implementations (STORM-1320)
> >>>     > >>
> >>>     > >> * Additional Grouping Implementations (STORM-1328)
> >>>     > >>
> >>>     > >>
> >>>     > >> As always I’m open to any opinions and suggestions.
> >>>     > >>
> >>>     > >> -Taylor
> >>>     > >>
> >>>     > >> [1]
> >>>     > >>
> >>>     > >> https://cwiki.apache.org/confluence/pages/viewpage.
> >>>     > >> action?pageId=61328109
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     > >>
> >>>     >
> >>>
> >>>
> >>>
>
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
The new metrics APIs are not a breaking change. The only user-facing API change is the addition of methods to TopologyContext that allow for the registration of user-defined metrics [1].

-Taylor

[1] https://github.com/apache/storm/blob/09c420e6147b4f788ea20c34cced195d72655a28/storm-core/src/jvm/org/apache/storm/task/TopologyContext.java#L392 <https://github.com/apache/storm/blob/09c420e6147b4f788ea20c34cced195d72655a28/storm-core/src/jvm/org/apache/storm/task/TopologyContext.java#L392>

> On Jun 26, 2017, at 10:18 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> Yes I would prefer to see us get a 2.x alpha release out.  Are the new metrics APIs going to be a breaking change?  I assume not, because we wouldn't want to put them in 1.x otherwise.  If that is the case then we don't need to wait for them before doing a 2.x release.
> 
> - Bobby
> 
> 
> On Friday, June 23, 2017, 3:19:28 PM CDT, P. Taylor Goetz <pt...@gmail.com> wrote:
> 
> Bobby/Jungtaek,
> 
> Are you saying you want to forego the 1.2 “metrics_v2” release and include it only in 2.0? (I ask because that work is already based on 1.x-branch, and forward-porting it to master is relatively simple.) I’d kind of like that work go out soon. 
> 
> If we go with option 1, I would want to see a 2.0 release (even if it’s a “beta” or “preview) before putting the 1.x line into maintenance mode. 
> 
> -Taylor
> 
>> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>> 
>> I see 2 ways to address this.
>> 1) We put the 1.x line into maintenance mode like with 0.10.  We don't backport anything except bug fixes.2) We backport a lot of the backwards compatible changes from 2.x to 1.x.
>> My personal preference is 1.  It makes it clear the direction we want to go in.  The biggest issue is that we probably would want to do a 2.x release sooner rather then later.  Even if we don't get all of the features that people want, if we just get a release out we can add in new features if they are backwards compatible, or we can create a 3.x line that would have the breaking changes in it.
>> 
>> - Bobby
>> 
>> 
>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
>> 
>> I'd like to bump this again instead of initiating new discussion thread.
>> 
>> I had having hard time to create and apply pull requests for both master
>> and 1.x-branch and that's really painful and sometimes blocker for me to do
>> merge step.
>> Two branches are heavily diverged more than between 0.10 and 1.0.0, even
>> IDE can't switch between the branch smoothly. We didn't even address
>> checkstyle issue yet, but after addressing, it could be "completely"
>> diverged. JDK version is another major issue, since the pull requests
>> targeted for master branch are not checked against JDK 7, and some of them
>> make some issues regarding JDK version while porting back.
>> 
>> So personally I really would like to see the plan for 1.x version line
>> changed - skipping any minor releases including 1.2.0 - and have epic issue
>> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
>> proposed plan was having 2.0.0 directly after 1.0.0)
>> 
>> Would like to hear everyone's opinions. If we have consensus to not having
>> any minor releases for 1.x version line, I will not port back non-bugfix
>> pull requests to 1.x-branch, and guide contributors to create pull requests
>> against master branch, not 1.x version line.
>> 
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>> 
>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <av...@gmail.com>님이
>> 작성:
>> 
>>> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
>>> we're very interested anything that can provide better throughput.
>>> 
>>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
>>> 
>>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
>>>> delivered sooner and my in-progress implementation suggests that it will
>>>> yield substantial latency improvements. The 2.0 beta phase would really
>>>> help kick the tires on the revised messaging system and the performance
>>>> improvements will also be a good incentive for trying out the 2.0 line.
>>>> 
>>>> I notice multiple other bottlenecks that are holding back throughput a
>>>> lot, which can be addressed in a subsequent 2.x minor release.
>>>> -roshan
>>>> 
>>>> 
>>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>>> 
>>>>     I also would love to see metrics V2 code sooner or later too. If we
>>>> can get
>>>>     it before releasing 2.0.0 that will be great, and then maybe we could
>>>> just
>>>>     move toward to 2.0.0, not adding any improvements to 1.x version
>>> line.
>>>>     (And that's what I would want to.)
>>>> 
>>>>     If we would really want to have 1.2.0, I suggest that we make the
>>> 1.1.1
>>>>     version correct right now rather than after releasing 1.1.1. We also
>>>> merged
>>>>     non-bugfix things to 1.x-branch but that's not what users expect. I
>>>> agree
>>>>     that work may be painful, but anyway need to do it.
>>>> 
>>>>     - Jungtaek Lim (HeartSaVioR)
>>>> 
>>>>     2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
>>>> 작성:
>>>> 
>>>>     > I would love to see the metrics V2 code come out sooner rather than
>>>>     > later.  +1.
>>>>     > My biggest blocker for a 1.x release is
>>>>     > https://github.com/apache/storm/pull/2142  Even though the pull
>>>> request
>>>>     > says it is minor it showed that we messed up pushing back some
>>>> changes for
>>>>     > pacemaker to open source (the code does not run at all which for me
>>>> is a
>>>>     > blocker) and I really want to get that fully fixed/tested before
>>>> another
>>>>     > release.
>>>>     > As for 2.x I think we are very close to being able to so a 2.x
>>> alpha
>>>>     > release.  I would like to see metrics V2 merged in simply because
>>> it
>>>> is a
>>>>     > big change for user facing APIs.  But after that I would love to
>>> see
>>>> us
>>>>     > starting to push forward on getting that out.
>>>>     >
>>>>     >
>>>>     > - Bobby
>>>>     >
>>>>     >
>>>>     > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>>>>     > ptgoetz@gmail.com> wrote:
>>>>     >
>>>>     > I’d like to bump this thread and start a discussion around our next
>>>>     > release. Here are my thoughts.
>>>>     >
>>>>     > There are a number of important fixes in 1.x-branch so I’d like to
>>>>     > consider releasing 1.1.1 soon. I’d appreciate input on any open
>>>> issues that
>>>>     > should be resolved for that release.
>>>>     >
>>>>     > I’d like us to consider releasing the metrics improvements in
>>>> STORM-2153
>>>>     > [1] as version 1.2.0. That work is in the metrics_v2 feature branch
>>>> right
>>>>     > now and would need to be reviewed and merged. That work is against
>>>> the
>>>>     > 1.x-branch right now. I would recommend porting it to master
>>> *after*
>>>> the
>>>>     > review/merge since there will likely be changes as a result of the
>>>> review.
>>>>     >
>>>>     > > Maybe related to or not, but would we want to create a new branch
>>>>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>>     >
>>>>     >
>>>>     >
>>>>     > If wee agree to the above, I would say yes. After the 1.1.1
>>> release,
>>>> we
>>>>     > could create a 1.1.x-branch that would be the maintenance/release
>>>> branch
>>>>     > for that version line. 1.x-branch would then become the target for
>>>> the
>>>>     > 1.2.0 release.
>>>>     >
>>>>     > There are a few fixes in the 0.10.x branch that probably warrant a
>>>>     > release. After that we may want to back away from that version line
>>>> a bit
>>>>     > so we can focus more on newer versions.
>>>>     >
>>>>     > In the past, we’ve shied away form doing “beta” releases, but I’m
>>>>     > wondering if we might want to revisit that for the 2.0 release —
>>> the
>>>> idea
>>>>     > being that it would give early adopter users a chance to kick the
>>>> tires on
>>>>     > what’s coming in 2.0 and provide feedback, find bugs, etc. to help
>>>> make the
>>>>     > final release more solid. I’m on the fence here and could go either
>>>> way.
>>>>     >
>>>>     > I’d appreciate any input others may have.
>>>>     >
>>>>     >
>>>>     > Thanks,
>>>>     >
>>>>     > -Taylor
>>>>     >
>>>>     >
>>>>     > [1] https://issues.apache.org/jira/browse/STORM-2153
>>>>     >
>>>>     >
>>>>     >
>>>>     > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
>>>> wrote:
>>>>     > >
>>>>     > > Maybe related to or not, but would we want to create a new branch
>>>>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>>     > >
>>>>     > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so
>>>> hence
>>>>     > the
>>>>     > > question.
>>>>     > >
>>>>     > > - Jungtaek Lim (HeartSaVioR)
>>>>     > >
>>>>     > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
>>>> hlouro@hortonworks.com>님이
>>>>     > 작성:
>>>>     > >
>>>>     > >> +1 for finishing the porting to Java ahead of anything else - it
>>>> will
>>>>     > be a
>>>>     > >> significant milestone. I have a JIRA assigned concerning to the
>>>>     > porting. I
>>>>     > >> will work on it for the 2.0 release.
>>>>     > >>
>>>>     > >> it’s a priority to guarantee no performance regressions. As part
>>>> of this
>>>>     > >> endeavor, explore an automated (or easy) way to run and assert
>>>> major
>>>>     > >> performance benchmarks. Ideally any contributor should be able
>>> to
>>>> fairly
>>>>     > >> easily test the impact of changes under certain performance test
>>>>     > scenarios.
>>>>     > >>
>>>>     > >> Beam Runner work should take into account the impact of
>>>> incorporating
>>>>     > new
>>>>     > >> JStorm features and Storm Worker Redesign<
>>>>     > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very
>>>> efficient
>>>>     > to
>>>>     > >> start doing it, to  find out that it will have to chance in face
>>>> of
>>>>     > Storm
>>>>     > >> and worker redesign. That is, it should be done after it’s
>>>> building
>>>>     > blocks
>>>>     > >> are stable.
>>>>     > >>
>>>>     > >> Thanks,
>>>>     > >> Hugo
>>>>     > >>
>>>>     > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org
>>>> <mailto:
>>>>     > >> arunm@apache.org>> wrote:
>>>>     > >>
>>>>     > >> +1 to release with the porting completed. I think its mainly the
>>>> UI
>>>>     > server
>>>>     > >> and log viewer that’s pending.
>>>>     > >>
>>>>     > >> We can start doing the regression and performance tests for
>>>> whatever is
>>>>     > >> already ported.
>>>>     > >>
>>>>     > >> If anyone is running the master branch in their pre-prod / prod
>>>>     > >> environments, it will be good to know and give us more
>>> confidence.
>>>>     > >>
>>>>     > >> The other features can be added in follow up releases.
>>>>     > >>
>>>>     > >> Regards,
>>>>     > >> Arun
>>>>     > >>
>>>>     > >>
>>>>     > >> On 3/24/17, 11:47 AM, "Satish Duggana" <
>>> satish.duggana@gmail.com
>>>>     > <mailto:
>>>>     > >> satish.duggana@gmail.com>> wrote:
>>>>     > >>
>>>>     > >> +1 to have 2.0 with porting and performance(it should be at
>>> least
>>>> as
>>>>     > good
>>>>     > >> as 1.x release) issues addressed
>>>>     > >>
>>>>     > >> We can target other tasks(mentioned by Taylor and Jungtaek) for
>>>>     > 2.x-branch.
>>>>     > >>
>>>>     > >>
>>>>     > >> Exactly-once support:
>>>>     > >> While thinking through the exactlyonce support design, it is
>>>> realized
>>>>     > >> better to avoid acking tuples and implement exactly once by
>>>> snapshotting
>>>>     > >> barriers. It seems JStorm folks followed similar design, they
>>>> claim it
>>>>     > >> gives better performance. This feature is essential for beam
>>>> runner and
>>>>     > we
>>>>     > >> can decide on respective approaches though.
>>>>     > >>
>>>>     > >> Beam Runner
>>>>     > >> Lets hold on this for now and keep it in Storm till 2.x. We
>>>> should avoid
>>>>     > >> having a minimal beam runner in haste. It is better to address
>>>>     > STORM-2284,
>>>>     > >> exactly-once and other windowing enhancements to enable beam
>>>> runner.
>>>>     > >>
>>>>     > >> JStorm
>>>>     > >> Agree with Jungtaek on looking at the latest JStorm and
>>>> align/scope with
>>>>     > >> the features for 2.x.
>>>>     > >>
>>>>     > >> STORM-2284
>>>>     > >> We may want to look at JStorm worker before working on
>>> respective
>>>>     > >> components in this epic to pull appropriate enhancements.
>>>>     > >>
>>>>     > >> YARN/MESOS
>>>>     > >> Supporting Storm on YARN/Mesos for 2.x.
>>>>     > >>
>>>>     > >> Thanks,
>>>>     > >> Satish.
>>>>     > >>
>>>>     > >>
>>>>     > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
>>> kabhwan@gmail.com
>>>>     > <mailto:
>>>>     > >> kabhwan@gmail.com>> wrote:
>>>>     > >>
>>>>     > >> First of all, +1 to complete only port work and do sanity check
>>>>     > (including
>>>>     > >> performance regression), and release.
>>>>     > >>
>>>>     > >> If we can get STORM-2284 within deterministic time frame (say
>>> 2~3
>>>>     > months)
>>>>     > >> that should be great, but if not I'd in favor of postponing that
>>>> to
>>>>     > later
>>>>     > >> 2.x release.
>>>>     > >>
>>>>     > >> JStorm released their new versions after code donation. So
>>>> there're more
>>>>     > >> things we could get ideas from, or even adopt from.
>>>>     > >> https://github.com/alibaba/jstorm/blob/master/history.md
>>>>     > >> As you noticed from release note link, we also need to update
>>>> phase 2
>>>>     > since
>>>>     > >> they already changed what we're planning to do in phase 2. For
>>>> example,
>>>>     > >> they changed backpressure to end-to-end, and changed to use
>>>> snapshot
>>>>     > rather
>>>>     > >> than acker.
>>>>     > >> May be sure, JStorm pulled many features from today's Storm,
>>> like
>>>> Flux,
>>>>     > >> Windowing, more shuffle groupings, log search, log level change,
>>>> and so
>>>>     > on.
>>>>     > >>
>>>>     > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
>>> is
>>>> due to
>>>>     > >> the
>>>>     > >> limitation of Spout lifecycle (all the things are done in single
>>>>     > thread),
>>>>     > >> and STORM-1358 <
>>> https://issues.apache.org/jira/browse/STORM-1358
>>>>     > >(JStorm's
>>>>     > >> multi-thread Spout) can remedy this (despite that Spout
>>>> implementation
>>>>     > may
>>>>     > >> need to guarantee thread-safety later). It's not a just
>>>> improvement but
>>>>     > >> close to design concern so would like to address sooner than
>>> other
>>>>     > things
>>>>     > >> in phase 2.
>>>>     > >>
>>>>     > >> For Storm SQL side, I've lost progress but major work would be
>>>> adopting
>>>>     > >> group by with windowing. It was not available from Calcite but
>>>> will be
>>>>     > >> available at next release (1.12.0).
>>>>     > >> I've filed this to STORM-2405
>>>>     > >> <https://issues.apache.org/jira/browse/STORM-2405>, but
>>>> windowing &
>>>>     > micro
>>>>     > >> batch is not intuitive, so I would like to change the underlying
>>>> API to
>>>>     > >> stream API in SQL. Also filed this to STORM-2406
>>>>     > >> <https://issues.apache.org/jira/browse/STORM-2406>.
>>>>     > >>
>>>>     > >> Just 2 cents btw, hopefully I would like to see metrics V2
>>> sooner
>>>> since
>>>>     > we
>>>>     > >> lost metrics even when doing normal operation like restarting
>>>> worker,
>>>>     > >> rebalancing, and so on. Eventually we need to fight with dynamic
>>>>     > scaling,
>>>>     > >> and then metrics will be broken often.
>>>>     > >>
>>>>     > >> Thanks,
>>>>     > >> Jungtaek Lim (HeartSaVioR)
>>>>     > >>
>>>>     > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <storm@harsha.io
>>>> 님이
>>>> 작성:
>>>>     > >>
>>>>     > >> Storm 2.0 migration to java in itself is a big win and would
>>>> attract
>>>>     > >> wider
>>>>     > >> community and adoption. So my vote would be to resolve the first
>>>> 3 items
>>>>     > >> to
>>>>     > >> get a release out.
>>>>     > >> All the other featured mentioned are great to have but shouldn't
>>>> be
>>>>     > >> blockers for 2.0 release.
>>>>     > >>
>>>>     > >> -Harsha
>>>>     > >>
>>>>     > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
>>>> ptgoetz@gmail.com>
>>>>     > >> wrote:
>>>>     > >>
>>>>     > >> With the 1.1.0 release nearing completion, I’d like to turn our
>>>>     > >> attention
>>>>     > >> to 2.0 and develop a plan for what features, etc. to include.
>>>>     > >>
>>>>     > >> The following 3 are what I feel are the minimum for a 2.0
>>> release.
>>>>     > >> These
>>>>     > >> could likely be resolved relatively quickly:
>>>>     > >>
>>>>     > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x
>>>> or
>>>>     > >> 1.1.x
>>>>     > >> in a while, but I feel it will be important to make sure there
>>>> are no
>>>>     > >> performance regressions, and would hope that we actually have a
>>>>     > >> performance
>>>>     > >> improvement over previous versions. To that end (e.g. if there
>>> is
>>>> in
>>>>     > >> fact a
>>>>     > >> performance regression), the proposals that Roshan Naik put
>>>> together
>>>>     > >> for
>>>>     > >> revising the threading and execution model (STORM-2307) and
>>>> replacing
>>>>     > >> Disruptor with JCTools (STORM-2306) warrant review and
>>>> consideration.
>>>>     > >> See
>>>>     > >> also STORM-2284 which is the parent JIRA.
>>>>     > >>
>>>>     > >> * Finish porting Storm UI to java (STORM-1311)
>>>>     > >>
>>>>     > >> * Finish porting log viewer to java (STORM-1280)
>>>>     > >>
>>>>     > >> The following are items that are nice to have in 2.0, but I
>>> don’t
>>>> feel
>>>>     > >> are
>>>>     > >> absolutely necessary for an initial 2.0 release:
>>>>     > >>
>>>>     > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because
>>>> it’s
>>>>     > >> relevant) — Initially there seemed to be a lot of interest in
>>>> this, but
>>>>     > >> that seems to have trailed off. I spoke with some Beam
>>> developers
>>>> and
>>>>     > >> there
>>>>     > >> seems to be interest from that community as well. Do we want to
>>>> move
>>>>     > >> that
>>>>     > >> effort to the Beam community, or keep it here? Moving it to the
>>>> Beam
>>>>     > >> community might lead to better collaboration between projects.
>>>>     > >>
>>>>     > >> * Bounded Spouts (needed for Beam Runner implementation) —
>>>> Currently
>>>>     > >> spouts are unbounded, there no end to the stream. Beam has the
>>>> concept
>>>>     > >> of
>>>>     > >> bounded sources (roughly analogous to batch processing). To
>>>> support
>>>>     > >> that,
>>>>     > >> we would need to implement a similar concept in Storm. One
>>>> benefit of
>>>>     > >> such
>>>>     > >> a feature would be the ability to handle both bounded and
>>>> unbounded
>>>>     > >> workflows in Storm.
>>>>     > >>
>>>>     > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
>>>> behind
>>>>     > >> this
>>>>     > >> effort. What improvements do you envision for 2.0?
>>>>     > >>
>>>>     > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
>>>> targeting this
>>>>     > >> for 1.2.0, but it’s designed to be easily portable to
>>> master/2.0.
>>>>     > >>
>>>>     > >> * JStorm Migration — Original outline can be found here [1].
>>> Note
>>>> a lot
>>>>     > >> of
>>>>     > >> the associated JIRAs below are assigned, but there hasn’t been
>>> any
>>>>     > >> recent
>>>>     > >> activity or pull requests, we should probably consider them
>>>> unassigned
>>>>     > >> and
>>>>     > >> up for grabs.:
>>>>     > >>
>>>>     > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has
>>>> been the
>>>>     > >> bane of a lot of Storm users almost since day one. We have
>>> largely
>>>>     > >> addressed it by shading/relocating dependencies. It would be
>>>> great to
>>>>     > >> see
>>>>     > >> this addressed once and for all.
>>>>     > >>
>>>>     > >> * JStorm back pressure implementation (STORM-1324) — The current
>>>> back
>>>>     > >> pressure implementation leaves a bit to be desired, and the
>>> JStorm
>>>>     > >> approach
>>>>     > >> looks promising, though it also depends on the JStorm concept of
>>>>     > >> “topology
>>>>     > >> master” (STORM-1323), which may have some implications regarding
>>>>     > >> security.
>>>>     > >>
>>>>     > >> * Dynamic Topology Updates (STORM-1335) — This would provide a
>>>> command
>>>>     > >> to
>>>>     > >> update topology jars and configuration without stopping the
>>>> topology,
>>>>     > >> and
>>>>     > >> is well suited to leverage the blobstore. The restart command
>>>> (that can
>>>>     > >> also update the topology configuration) also looks compelling
>>>>     > >> (STORM-1334).
>>>>     > >>
>>>>     > >> * Additional Scheduler Implementations (STORM-1320)
>>>>     > >>
>>>>     > >> * Additional Grouping Implementations (STORM-1328)
>>>>     > >>
>>>>     > >>
>>>>     > >> As always I’m open to any opinions and suggestions.
>>>>     > >>
>>>>     > >> -Taylor
>>>>     > >>
>>>>     > >> [1]
>>>>     > >>
>>>>     > >> https://cwiki.apache.org/confluence/pages/viewpage.
>>>>     > >> action?pageId=61328109
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     > >>
>>>>     >
>>>> 
>>>> 
>>>> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Yes I would prefer to see us get a 2.x alpha release out.  Are the new metrics APIs going to be a breaking change?  I assume not, because we wouldn't want to put them in 1.x otherwise.  If that is the case then we don't need to wait for them before doing a 2.x release.

- Bobby


On Friday, June 23, 2017, 3:19:28 PM CDT, P. Taylor Goetz <pt...@gmail.com> wrote:

Bobby/Jungtaek,

Are you saying you want to forego the 1.2 “metrics_v2” release and include it only in 2.0? (I ask because that work is already based on 1.x-branch, and forward-porting it to master is relatively simple.) I’d kind of like that work go out soon. 

If we go with option 1, I would want to see a 2.0 release (even if it’s a “beta” or “preview) before putting the 1.x line into maintenance mode. 

-Taylor

> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> I see 2 ways to address this.
> 1) We put the 1.x line into maintenance mode like with 0.10.  We don't backport anything except bug fixes.2) We backport a lot of the backwards compatible changes from 2.x to 1.x.
> My personal preference is 1.  It makes it clear the direction we want to go in.  The biggest issue is that we probably would want to do a 2.x release sooner rather then later.  Even if we don't get all of the features that people want, if we just get a release out we can add in new features if they are backwards compatible, or we can create a 3.x line that would have the breaking changes in it.
> 
> - Bobby
> 
> 
> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> I'd like to bump this again instead of initiating new discussion thread.
> 
> I had having hard time to create and apply pull requests for both master
> and 1.x-branch and that's really painful and sometimes blocker for me to do
> merge step.
> Two branches are heavily diverged more than between 0.10 and 1.0.0, even
> IDE can't switch between the branch smoothly. We didn't even address
> checkstyle issue yet, but after addressing, it could be "completely"
> diverged. JDK version is another major issue, since the pull requests
> targeted for master branch are not checked against JDK 7, and some of them
> make some issues regarding JDK version while porting back.
> 
> So personally I really would like to see the plan for 1.x version line
> changed - skipping any minor releases including 1.2.0 - and have epic issue
> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> proposed plan was having 2.0.0 directly after 1.0.0)
> 
> Would like to hear everyone's opinions. If we have consensus to not having
> any minor releases for 1.x version line, I will not port back non-bugfix
> pull requests to 1.x-branch, and guide contributors to create pull requests
> against master branch, not 1.x version line.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <av...@gmail.com>님이
> 작성:
> 
>> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
>> we're very interested anything that can provide better throughput.
>> 
>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
>> 
>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
>>> delivered sooner and my in-progress implementation suggests that it will
>>> yield substantial latency improvements. The 2.0 beta phase would really
>>> help kick the tires on the revised messaging system and the performance
>>> improvements will also be a good incentive for trying out the 2.0 line.
>>> 
>>> I notice multiple other bottlenecks that are holding back throughput a
>>> lot, which can be addressed in a subsequent 2.x minor release.
>>> -roshan
>>> 
>>> 
>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>> 
>>>    I also would love to see metrics V2 code sooner or later too. If we
>>> can get
>>>    it before releasing 2.0.0 that will be great, and then maybe we could
>>> just
>>>    move toward to 2.0.0, not adding any improvements to 1.x version
>> line.
>>>    (And that's what I would want to.)
>>> 
>>>    If we would really want to have 1.2.0, I suggest that we make the
>> 1.1.1
>>>    version correct right now rather than after releasing 1.1.1. We also
>>> merged
>>>    non-bugfix things to 1.x-branch but that's not what users expect. I
>>> agree
>>>    that work may be painful, but anyway need to do it.
>>> 
>>>    - Jungtaek Lim (HeartSaVioR)
>>> 
>>>    2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
>>> 작성:
>>> 
>>>    > I would love to see the metrics V2 code come out sooner rather than
>>>    > later.  +1.
>>>    > My biggest blocker for a 1.x release is
>>>    > https://github.com/apache/storm/pull/2142  Even though the pull
>>> request
>>>    > says it is minor it showed that we messed up pushing back some
>>> changes for
>>>    > pacemaker to open source (the code does not run at all which for me
>>> is a
>>>    > blocker) and I really want to get that fully fixed/tested before
>>> another
>>>    > release.
>>>    > As for 2.x I think we are very close to being able to so a 2.x
>> alpha
>>>    > release.  I would like to see metrics V2 merged in simply because
>> it
>>> is a
>>>    > big change for user facing APIs.  But after that I would love to
>> see
>>> us
>>>    > starting to push forward on getting that out.
>>>    >
>>>    >
>>>    > - Bobby
>>>    >
>>>    >
>>>    > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>>>    > ptgoetz@gmail.com> wrote:
>>>    >
>>>    > I’d like to bump this thread and start a discussion around our next
>>>    > release. Here are my thoughts.
>>>    >
>>>    > There are a number of important fixes in 1.x-branch so I’d like to
>>>    > consider releasing 1.1.1 soon. I’d appreciate input on any open
>>> issues that
>>>    > should be resolved for that release.
>>>    >
>>>    > I’d like us to consider releasing the metrics improvements in
>>> STORM-2153
>>>    > [1] as version 1.2.0. That work is in the metrics_v2 feature branch
>>> right
>>>    > now and would need to be reviewed and merged. That work is against
>>> the
>>>    > 1.x-branch right now. I would recommend porting it to master
>> *after*
>>> the
>>>    > review/merge since there will likely be changes as a result of the
>>> review.
>>>    >
>>>    > > Maybe related to or not, but would we want to create a new branch
>>>    > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>    >
>>>    >
>>>    >
>>>    > If wee agree to the above, I would say yes. After the 1.1.1
>> release,
>>> we
>>>    > could create a 1.1.x-branch that would be the maintenance/release
>>> branch
>>>    > for that version line. 1.x-branch would then become the target for
>>> the
>>>    > 1.2.0 release.
>>>    >
>>>    > There are a few fixes in the 0.10.x branch that probably warrant a
>>>    > release. After that we may want to back away from that version line
>>> a bit
>>>    > so we can focus more on newer versions.
>>>    >
>>>    > In the past, we’ve shied away form doing “beta” releases, but I’m
>>>    > wondering if we might want to revisit that for the 2.0 release —
>> the
>>> idea
>>>    > being that it would give early adopter users a chance to kick the
>>> tires on
>>>    > what’s coming in 2.0 and provide feedback, find bugs, etc. to help
>>> make the
>>>    > final release more solid. I’m on the fence here and could go either
>>> way.
>>>    >
>>>    > I’d appreciate any input others may have.
>>>    >
>>>    >
>>>    > Thanks,
>>>    >
>>>    > -Taylor
>>>    >
>>>    >
>>>    > [1] https://issues.apache.org/jira/browse/STORM-2153
>>>    >
>>>    >
>>>    >
>>>    > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
>>> wrote:
>>>    > >
>>>    > > Maybe related to or not, but would we want to create a new branch
>>>    > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>    > >
>>>    > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so
>>> hence
>>>    > the
>>>    > > question.
>>>    > >
>>>    > > - Jungtaek Lim (HeartSaVioR)
>>>    > >
>>>    > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
>>> hlouro@hortonworks.com>님이
>>>    > 작성:
>>>    > >
>>>    > >> +1 for finishing the porting to Java ahead of anything else - it
>>> will
>>>    > be a
>>>    > >> significant milestone. I have a JIRA assigned concerning to the
>>>    > porting. I
>>>    > >> will work on it for the 2.0 release.
>>>    > >>
>>>    > >> it’s a priority to guarantee no performance regressions. As part
>>> of this
>>>    > >> endeavor, explore an automated (or easy) way to run and assert
>>> major
>>>    > >> performance benchmarks. Ideally any contributor should be able
>> to
>>> fairly
>>>    > >> easily test the impact of changes under certain performance test
>>>    > scenarios.
>>>    > >>
>>>    > >> Beam Runner work should take into account the impact of
>>> incorporating
>>>    > new
>>>    > >> JStorm features and Storm Worker Redesign<
>>>    > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very
>>> efficient
>>>    > to
>>>    > >> start doing it, to  find out that it will have to chance in face
>>> of
>>>    > Storm
>>>    > >> and worker redesign. That is, it should be done after it’s
>>> building
>>>    > blocks
>>>    > >> are stable.
>>>    > >>
>>>    > >> Thanks,
>>>    > >> Hugo
>>>    > >>
>>>    > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org
>>> <mailto:
>>>    > >> arunm@apache.org>> wrote:
>>>    > >>
>>>    > >> +1 to release with the porting completed. I think its mainly the
>>> UI
>>>    > server
>>>    > >> and log viewer that’s pending.
>>>    > >>
>>>    > >> We can start doing the regression and performance tests for
>>> whatever is
>>>    > >> already ported.
>>>    > >>
>>>    > >> If anyone is running the master branch in their pre-prod / prod
>>>    > >> environments, it will be good to know and give us more
>> confidence.
>>>    > >>
>>>    > >> The other features can be added in follow up releases.
>>>    > >>
>>>    > >> Regards,
>>>    > >> Arun
>>>    > >>
>>>    > >>
>>>    > >> On 3/24/17, 11:47 AM, "Satish Duggana" <
>> satish.duggana@gmail.com
>>>    > <mailto:
>>>    > >> satish.duggana@gmail.com>> wrote:
>>>    > >>
>>>    > >> +1 to have 2.0 with porting and performance(it should be at
>> least
>>> as
>>>    > good
>>>    > >> as 1.x release) issues addressed
>>>    > >>
>>>    > >> We can target other tasks(mentioned by Taylor and Jungtaek) for
>>>    > 2.x-branch.
>>>    > >>
>>>    > >>
>>>    > >> Exactly-once support:
>>>    > >> While thinking through the exactlyonce support design, it is
>>> realized
>>>    > >> better to avoid acking tuples and implement exactly once by
>>> snapshotting
>>>    > >> barriers. It seems JStorm folks followed similar design, they
>>> claim it
>>>    > >> gives better performance. This feature is essential for beam
>>> runner and
>>>    > we
>>>    > >> can decide on respective approaches though.
>>>    > >>
>>>    > >> Beam Runner
>>>    > >> Lets hold on this for now and keep it in Storm till 2.x. We
>>> should avoid
>>>    > >> having a minimal beam runner in haste. It is better to address
>>>    > STORM-2284,
>>>    > >> exactly-once and other windowing enhancements to enable beam
>>> runner.
>>>    > >>
>>>    > >> JStorm
>>>    > >> Agree with Jungtaek on looking at the latest JStorm and
>>> align/scope with
>>>    > >> the features for 2.x.
>>>    > >>
>>>    > >> STORM-2284
>>>    > >> We may want to look at JStorm worker before working on
>> respective
>>>    > >> components in this epic to pull appropriate enhancements.
>>>    > >>
>>>    > >> YARN/MESOS
>>>    > >> Supporting Storm on YARN/Mesos for 2.x.
>>>    > >>
>>>    > >> Thanks,
>>>    > >> Satish.
>>>    > >>
>>>    > >>
>>>    > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
>> kabhwan@gmail.com
>>>    > <mailto:
>>>    > >> kabhwan@gmail.com>> wrote:
>>>    > >>
>>>    > >> First of all, +1 to complete only port work and do sanity check
>>>    > (including
>>>    > >> performance regression), and release.
>>>    > >>
>>>    > >> If we can get STORM-2284 within deterministic time frame (say
>> 2~3
>>>    > months)
>>>    > >> that should be great, but if not I'd in favor of postponing that
>>> to
>>>    > later
>>>    > >> 2.x release.
>>>    > >>
>>>    > >> JStorm released their new versions after code donation. So
>>> there're more
>>>    > >> things we could get ideas from, or even adopt from.
>>>    > >> https://github.com/alibaba/jstorm/blob/master/history.md
>>>    > >> As you noticed from release note link, we also need to update
>>> phase 2
>>>    > since
>>>    > >> they already changed what we're planning to do in phase 2. For
>>> example,
>>>    > >> they changed backpressure to end-to-end, and changed to use
>>> snapshot
>>>    > rather
>>>    > >> than acker.
>>>    > >> May be sure, JStorm pulled many features from today's Storm,
>> like
>>> Flux,
>>>    > >> Windowing, more shuffle groupings, log search, log level change,
>>> and so
>>>    > on.
>>>    > >>
>>>    > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
>> is
>>> due to
>>>    > >> the
>>>    > >> limitation of Spout lifecycle (all the things are done in single
>>>    > thread),
>>>    > >> and STORM-1358 <
>> https://issues.apache.org/jira/browse/STORM-1358
>>>    > >(JStorm's
>>>    > >> multi-thread Spout) can remedy this (despite that Spout
>>> implementation
>>>    > may
>>>    > >> need to guarantee thread-safety later). It's not a just
>>> improvement but
>>>    > >> close to design concern so would like to address sooner than
>> other
>>>    > things
>>>    > >> in phase 2.
>>>    > >>
>>>    > >> For Storm SQL side, I've lost progress but major work would be
>>> adopting
>>>    > >> group by with windowing. It was not available from Calcite but
>>> will be
>>>    > >> available at next release (1.12.0).
>>>    > >> I've filed this to STORM-2405
>>>    > >> <https://issues.apache.org/jira/browse/STORM-2405>, but
>>> windowing &
>>>    > micro
>>>    > >> batch is not intuitive, so I would like to change the underlying
>>> API to
>>>    > >> stream API in SQL. Also filed this to STORM-2406
>>>    > >> <https://issues.apache.org/jira/browse/STORM-2406>.
>>>    > >>
>>>    > >> Just 2 cents btw, hopefully I would like to see metrics V2
>> sooner
>>> since
>>>    > we
>>>    > >> lost metrics even when doing normal operation like restarting
>>> worker,
>>>    > >> rebalancing, and so on. Eventually we need to fight with dynamic
>>>    > scaling,
>>>    > >> and then metrics will be broken often.
>>>    > >>
>>>    > >> Thanks,
>>>    > >> Jungtaek Lim (HeartSaVioR)
>>>    > >>
>>>    > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <storm@harsha.io
>>> 님이
>>> 작성:
>>>    > >>
>>>    > >> Storm 2.0 migration to java in itself is a big win and would
>>> attract
>>>    > >> wider
>>>    > >> community and adoption. So my vote would be to resolve the first
>>> 3 items
>>>    > >> to
>>>    > >> get a release out.
>>>    > >> All the other featured mentioned are great to have but shouldn't
>>> be
>>>    > >> blockers for 2.0 release.
>>>    > >>
>>>    > >> -Harsha
>>>    > >>
>>>    > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
>>> ptgoetz@gmail.com>
>>>    > >> wrote:
>>>    > >>
>>>    > >> With the 1.1.0 release nearing completion, I’d like to turn our
>>>    > >> attention
>>>    > >> to 2.0 and develop a plan for what features, etc. to include.
>>>    > >>
>>>    > >> The following 3 are what I feel are the minimum for a 2.0
>> release.
>>>    > >> These
>>>    > >> could likely be resolved relatively quickly:
>>>    > >>
>>>    > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x
>>> or
>>>    > >> 1.1.x
>>>    > >> in a while, but I feel it will be important to make sure there
>>> are no
>>>    > >> performance regressions, and would hope that we actually have a
>>>    > >> performance
>>>    > >> improvement over previous versions. To that end (e.g. if there
>> is
>>> in
>>>    > >> fact a
>>>    > >> performance regression), the proposals that Roshan Naik put
>>> together
>>>    > >> for
>>>    > >> revising the threading and execution model (STORM-2307) and
>>> replacing
>>>    > >> Disruptor with JCTools (STORM-2306) warrant review and
>>> consideration.
>>>    > >> See
>>>    > >> also STORM-2284 which is the parent JIRA.
>>>    > >>
>>>    > >> * Finish porting Storm UI to java (STORM-1311)
>>>    > >>
>>>    > >> * Finish porting log viewer to java (STORM-1280)
>>>    > >>
>>>    > >> The following are items that are nice to have in 2.0, but I
>> don’t
>>> feel
>>>    > >> are
>>>    > >> absolutely necessary for an initial 2.0 release:
>>>    > >>
>>>    > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because
>>> it’s
>>>    > >> relevant) — Initially there seemed to be a lot of interest in
>>> this, but
>>>    > >> that seems to have trailed off. I spoke with some Beam
>> developers
>>> and
>>>    > >> there
>>>    > >> seems to be interest from that community as well. Do we want to
>>> move
>>>    > >> that
>>>    > >> effort to the Beam community, or keep it here? Moving it to the
>>> Beam
>>>    > >> community might lead to better collaboration between projects.
>>>    > >>
>>>    > >> * Bounded Spouts (needed for Beam Runner implementation) —
>>> Currently
>>>    > >> spouts are unbounded, there no end to the stream. Beam has the
>>> concept
>>>    > >> of
>>>    > >> bounded sources (roughly analogous to batch processing). To
>>> support
>>>    > >> that,
>>>    > >> we would need to implement a similar concept in Storm. One
>>> benefit of
>>>    > >> such
>>>    > >> a feature would be the ability to handle both bounded and
>>> unbounded
>>>    > >> workflows in Storm.
>>>    > >>
>>>    > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
>>> behind
>>>    > >> this
>>>    > >> effort. What improvements do you envision for 2.0?
>>>    > >>
>>>    > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
>>> targeting this
>>>    > >> for 1.2.0, but it’s designed to be easily portable to
>> master/2.0.
>>>    > >>
>>>    > >> * JStorm Migration — Original outline can be found here [1].
>> Note
>>> a lot
>>>    > >> of
>>>    > >> the associated JIRAs below are assigned, but there hasn’t been
>> any
>>>    > >> recent
>>>    > >> activity or pull requests, we should probably consider them
>>> unassigned
>>>    > >> and
>>>    > >> up for grabs.:
>>>    > >>
>>>    > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has
>>> been the
>>>    > >> bane of a lot of Storm users almost since day one. We have
>> largely
>>>    > >> addressed it by shading/relocating dependencies. It would be
>>> great to
>>>    > >> see
>>>    > >> this addressed once and for all.
>>>    > >>
>>>    > >> * JStorm back pressure implementation (STORM-1324) — The current
>>> back
>>>    > >> pressure implementation leaves a bit to be desired, and the
>> JStorm
>>>    > >> approach
>>>    > >> looks promising, though it also depends on the JStorm concept of
>>>    > >> “topology
>>>    > >> master” (STORM-1323), which may have some implications regarding
>>>    > >> security.
>>>    > >>
>>>    > >> * Dynamic Topology Updates (STORM-1335) — This would provide a
>>> command
>>>    > >> to
>>>    > >> update topology jars and configuration without stopping the
>>> topology,
>>>    > >> and
>>>    > >> is well suited to leverage the blobstore. The restart command
>>> (that can
>>>    > >> also update the topology configuration) also looks compelling
>>>    > >> (STORM-1334).
>>>    > >>
>>>    > >> * Additional Scheduler Implementations (STORM-1320)
>>>    > >>
>>>    > >> * Additional Grouping Implementations (STORM-1328)
>>>    > >>
>>>    > >>
>>>    > >> As always I’m open to any opinions and suggestions.
>>>    > >>
>>>    > >> -Taylor
>>>    > >>
>>>    > >> [1]
>>>    > >>
>>>    > >> https://cwiki.apache.org/confluence/pages/viewpage.
>>>    > >> action?pageId=61328109
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>
>>>    > >>
>>>    >
>>> 
>>> 
>>> 

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Bobby/Jungtaek,

Are you saying you want to forego the 1.2 “metrics_v2” release and include it only in 2.0? (I ask because that work is already based on 1.x-branch, and forward-porting it to master is relatively simple.) I’d kind of like that work go out soon. 

If we go with option 1, I would want to see a 2.0 release (even if it’s a “beta” or “preview) before putting the 1.x line into maintenance mode. 

-Taylor

> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> I see 2 ways to address this.
> 1) We put the 1.x line into maintenance mode like with 0.10.  We don't backport anything except bug fixes.2) We backport a lot of the backwards compatible changes from 2.x to 1.x.
> My personal preference is 1.  It makes it clear the direction we want to go in.  The biggest issue is that we probably would want to do a 2.x release sooner rather then later.  Even if we don't get all of the features that people want, if we just get a release out we can add in new features if they are backwards compatible, or we can create a 3.x line that would have the breaking changes in it.
> 
> - Bobby
> 
> 
> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> I'd like to bump this again instead of initiating new discussion thread.
> 
> I had having hard time to create and apply pull requests for both master
> and 1.x-branch and that's really painful and sometimes blocker for me to do
> merge step.
> Two branches are heavily diverged more than between 0.10 and 1.0.0, even
> IDE can't switch between the branch smoothly. We didn't even address
> checkstyle issue yet, but after addressing, it could be "completely"
> diverged. JDK version is another major issue, since the pull requests
> targeted for master branch are not checked against JDK 7, and some of them
> make some issues regarding JDK version while porting back.
> 
> So personally I really would like to see the plan for 1.x version line
> changed - skipping any minor releases including 1.2.0 - and have epic issue
> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> proposed plan was having 2.0.0 directly after 1.0.0)
> 
> Would like to hear everyone's opinions. If we have consensus to not having
> any minor releases for 1.x version line, I will not port back non-bugfix
> pull requests to 1.x-branch, and guide contributors to create pull requests
> against master branch, not 1.x version line.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <av...@gmail.com>님이
> 작성:
> 
>> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
>> we're very interested anything that can provide better throughput.
>> 
>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
>> 
>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
>>> delivered sooner and my in-progress implementation suggests that it will
>>> yield substantial latency improvements. The 2.0 beta phase would really
>>> help kick the tires on the revised messaging system and the performance
>>> improvements will also be a good incentive for trying out the 2.0 line.
>>> 
>>> I notice multiple other bottlenecks that are holding back throughput a
>>> lot, which can be addressed in a subsequent 2.x minor release.
>>> -roshan
>>> 
>>> 
>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>> 
>>>     I also would love to see metrics V2 code sooner or later too. If we
>>> can get
>>>     it before releasing 2.0.0 that will be great, and then maybe we could
>>> just
>>>     move toward to 2.0.0, not adding any improvements to 1.x version
>> line.
>>>     (And that's what I would want to.)
>>> 
>>>     If we would really want to have 1.2.0, I suggest that we make the
>> 1.1.1
>>>     version correct right now rather than after releasing 1.1.1. We also
>>> merged
>>>     non-bugfix things to 1.x-branch but that's not what users expect. I
>>> agree
>>>     that work may be painful, but anyway need to do it.
>>> 
>>>     - Jungtaek Lim (HeartSaVioR)
>>> 
>>>     2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
>>> 작성:
>>> 
>>>     > I would love to see the metrics V2 code come out sooner rather than
>>>     > later.  +1.
>>>     > My biggest blocker for a 1.x release is
>>>     > https://github.com/apache/storm/pull/2142  Even though the pull
>>> request
>>>     > says it is minor it showed that we messed up pushing back some
>>> changes for
>>>     > pacemaker to open source (the code does not run at all which for me
>>> is a
>>>     > blocker) and I really want to get that fully fixed/tested before
>>> another
>>>     > release.
>>>     > As for 2.x I think we are very close to being able to so a 2.x
>> alpha
>>>     > release.  I would like to see metrics V2 merged in simply because
>> it
>>> is a
>>>     > big change for user facing APIs.  But after that I would love to
>> see
>>> us
>>>     > starting to push forward on getting that out.
>>>     >
>>>     >
>>>     > - Bobby
>>>     >
>>>     >
>>>     > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>>>     > ptgoetz@gmail.com> wrote:
>>>     >
>>>     > I’d like to bump this thread and start a discussion around our next
>>>     > release. Here are my thoughts.
>>>     >
>>>     > There are a number of important fixes in 1.x-branch so I’d like to
>>>     > consider releasing 1.1.1 soon. I’d appreciate input on any open
>>> issues that
>>>     > should be resolved for that release.
>>>     >
>>>     > I’d like us to consider releasing the metrics improvements in
>>> STORM-2153
>>>     > [1] as version 1.2.0. That work is in the metrics_v2 feature branch
>>> right
>>>     > now and would need to be reviewed and merged. That work is against
>>> the
>>>     > 1.x-branch right now. I would recommend porting it to master
>> *after*
>>> the
>>>     > review/merge since there will likely be changes as a result of the
>>> review.
>>>     >
>>>     > > Maybe related to or not, but would we want to create a new branch
>>>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>     >
>>>     >
>>>     >
>>>     > If wee agree to the above, I would say yes. After the 1.1.1
>> release,
>>> we
>>>     > could create a 1.1.x-branch that would be the maintenance/release
>>> branch
>>>     > for that version line. 1.x-branch would then become the target for
>>> the
>>>     > 1.2.0 release.
>>>     >
>>>     > There are a few fixes in the 0.10.x branch that probably warrant a
>>>     > release. After that we may want to back away from that version line
>>> a bit
>>>     > so we can focus more on newer versions.
>>>     >
>>>     > In the past, we’ve shied away form doing “beta” releases, but I’m
>>>     > wondering if we might want to revisit that for the 2.0 release —
>> the
>>> idea
>>>     > being that it would give early adopter users a chance to kick the
>>> tires on
>>>     > what’s coming in 2.0 and provide feedback, find bugs, etc. to help
>>> make the
>>>     > final release more solid. I’m on the fence here and could go either
>>> way.
>>>     >
>>>     > I’d appreciate any input others may have.
>>>     >
>>>     >
>>>     > Thanks,
>>>     >
>>>     > -Taylor
>>>     >
>>>     >
>>>     > [1] https://issues.apache.org/jira/browse/STORM-2153
>>>     >
>>>     >
>>>     >
>>>     > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
>>> wrote:
>>>     > >
>>>     > > Maybe related to or not, but would we want to create a new branch
>>>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>>>     > >
>>>     > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so
>>> hence
>>>     > the
>>>     > > question.
>>>     > >
>>>     > > - Jungtaek Lim (HeartSaVioR)
>>>     > >
>>>     > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
>>> hlouro@hortonworks.com>님이
>>>     > 작성:
>>>     > >
>>>     > >> +1 for finishing the porting to Java ahead of anything else - it
>>> will
>>>     > be a
>>>     > >> significant milestone. I have a JIRA assigned concerning to the
>>>     > porting. I
>>>     > >> will work on it for the 2.0 release.
>>>     > >>
>>>     > >> it’s a priority to guarantee no performance regressions. As part
>>> of this
>>>     > >> endeavor, explore an automated (or easy) way to run and assert
>>> major
>>>     > >> performance benchmarks. Ideally any contributor should be able
>> to
>>> fairly
>>>     > >> easily test the impact of changes under certain performance test
>>>     > scenarios.
>>>     > >>
>>>     > >> Beam Runner work should take into account the impact of
>>> incorporating
>>>     > new
>>>     > >> JStorm features and Storm Worker Redesign<
>>>     > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very
>>> efficient
>>>     > to
>>>     > >> start doing it, to  find out that it will have to chance in face
>>> of
>>>     > Storm
>>>     > >> and worker redesign. That is, it should be done after it’s
>>> building
>>>     > blocks
>>>     > >> are stable.
>>>     > >>
>>>     > >> Thanks,
>>>     > >> Hugo
>>>     > >>
>>>     > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org
>>> <mailto:
>>>     > >> arunm@apache.org>> wrote:
>>>     > >>
>>>     > >> +1 to release with the porting completed. I think its mainly the
>>> UI
>>>     > server
>>>     > >> and log viewer that’s pending.
>>>     > >>
>>>     > >> We can start doing the regression and performance tests for
>>> whatever is
>>>     > >> already ported.
>>>     > >>
>>>     > >> If anyone is running the master branch in their pre-prod / prod
>>>     > >> environments, it will be good to know and give us more
>> confidence.
>>>     > >>
>>>     > >> The other features can be added in follow up releases.
>>>     > >>
>>>     > >> Regards,
>>>     > >> Arun
>>>     > >>
>>>     > >>
>>>     > >> On 3/24/17, 11:47 AM, "Satish Duggana" <
>> satish.duggana@gmail.com
>>>     > <mailto:
>>>     > >> satish.duggana@gmail.com>> wrote:
>>>     > >>
>>>     > >> +1 to have 2.0 with porting and performance(it should be at
>> least
>>> as
>>>     > good
>>>     > >> as 1.x release) issues addressed
>>>     > >>
>>>     > >> We can target other tasks(mentioned by Taylor and Jungtaek) for
>>>     > 2.x-branch.
>>>     > >>
>>>     > >>
>>>     > >> Exactly-once support:
>>>     > >> While thinking through the exactlyonce support design, it is
>>> realized
>>>     > >> better to avoid acking tuples and implement exactly once by
>>> snapshotting
>>>     > >> barriers. It seems JStorm folks followed similar design, they
>>> claim it
>>>     > >> gives better performance. This feature is essential for beam
>>> runner and
>>>     > we
>>>     > >> can decide on respective approaches though.
>>>     > >>
>>>     > >> Beam Runner
>>>     > >> Lets hold on this for now and keep it in Storm till 2.x. We
>>> should avoid
>>>     > >> having a minimal beam runner in haste. It is better to address
>>>     > STORM-2284,
>>>     > >> exactly-once and other windowing enhancements to enable beam
>>> runner.
>>>     > >>
>>>     > >> JStorm
>>>     > >> Agree with Jungtaek on looking at the latest JStorm and
>>> align/scope with
>>>     > >> the features for 2.x.
>>>     > >>
>>>     > >> STORM-2284
>>>     > >> We may want to look at JStorm worker before working on
>> respective
>>>     > >> components in this epic to pull appropriate enhancements.
>>>     > >>
>>>     > >> YARN/MESOS
>>>     > >> Supporting Storm on YARN/Mesos for 2.x.
>>>     > >>
>>>     > >> Thanks,
>>>     > >> Satish.
>>>     > >>
>>>     > >>
>>>     > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
>> kabhwan@gmail.com
>>>     > <mailto:
>>>     > >> kabhwan@gmail.com>> wrote:
>>>     > >>
>>>     > >> First of all, +1 to complete only port work and do sanity check
>>>     > (including
>>>     > >> performance regression), and release.
>>>     > >>
>>>     > >> If we can get STORM-2284 within deterministic time frame (say
>> 2~3
>>>     > months)
>>>     > >> that should be great, but if not I'd in favor of postponing that
>>> to
>>>     > later
>>>     > >> 2.x release.
>>>     > >>
>>>     > >> JStorm released their new versions after code donation. So
>>> there're more
>>>     > >> things we could get ideas from, or even adopt from.
>>>     > >> https://github.com/alibaba/jstorm/blob/master/history.md
>>>     > >> As you noticed from release note link, we also need to update
>>> phase 2
>>>     > since
>>>     > >> they already changed what we're planning to do in phase 2. For
>>> example,
>>>     > >> they changed backpressure to end-to-end, and changed to use
>>> snapshot
>>>     > rather
>>>     > >> than acker.
>>>     > >> May be sure, JStorm pulled many features from today's Storm,
>> like
>>> Flux,
>>>     > >> Windowing, more shuffle groupings, log search, log level change,
>>> and so
>>>     > on.
>>>     > >>
>>>     > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
>> is
>>> due to
>>>     > >> the
>>>     > >> limitation of Spout lifecycle (all the things are done in single
>>>     > thread),
>>>     > >> and STORM-1358 <
>> https://issues.apache.org/jira/browse/STORM-1358
>>>     > >(JStorm's
>>>     > >> multi-thread Spout) can remedy this (despite that Spout
>>> implementation
>>>     > may
>>>     > >> need to guarantee thread-safety later). It's not a just
>>> improvement but
>>>     > >> close to design concern so would like to address sooner than
>> other
>>>     > things
>>>     > >> in phase 2.
>>>     > >>
>>>     > >> For Storm SQL side, I've lost progress but major work would be
>>> adopting
>>>     > >> group by with windowing. It was not available from Calcite but
>>> will be
>>>     > >> available at next release (1.12.0).
>>>     > >> I've filed this to STORM-2405
>>>     > >> <https://issues.apache.org/jira/browse/STORM-2405>, but
>>> windowing &
>>>     > micro
>>>     > >> batch is not intuitive, so I would like to change the underlying
>>> API to
>>>     > >> stream API in SQL. Also filed this to STORM-2406
>>>     > >> <https://issues.apache.org/jira/browse/STORM-2406>.
>>>     > >>
>>>     > >> Just 2 cents btw, hopefully I would like to see metrics V2
>> sooner
>>> since
>>>     > we
>>>     > >> lost metrics even when doing normal operation like restarting
>>> worker,
>>>     > >> rebalancing, and so on. Eventually we need to fight with dynamic
>>>     > scaling,
>>>     > >> and then metrics will be broken often.
>>>     > >>
>>>     > >> Thanks,
>>>     > >> Jungtaek Lim (HeartSaVioR)
>>>     > >>
>>>     > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <storm@harsha.io
>>> 님이
>>> 작성:
>>>     > >>
>>>     > >> Storm 2.0 migration to java in itself is a big win and would
>>> attract
>>>     > >> wider
>>>     > >> community and adoption. So my vote would be to resolve the first
>>> 3 items
>>>     > >> to
>>>     > >> get a release out.
>>>     > >> All the other featured mentioned are great to have but shouldn't
>>> be
>>>     > >> blockers for 2.0 release.
>>>     > >>
>>>     > >> -Harsha
>>>     > >>
>>>     > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
>>> ptgoetz@gmail.com>
>>>     > >> wrote:
>>>     > >>
>>>     > >> With the 1.1.0 release nearing completion, I’d like to turn our
>>>     > >> attention
>>>     > >> to 2.0 and develop a plan for what features, etc. to include.
>>>     > >>
>>>     > >> The following 3 are what I feel are the minimum for a 2.0
>> release.
>>>     > >> These
>>>     > >> could likely be resolved relatively quickly:
>>>     > >>
>>>     > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x
>>> or
>>>     > >> 1.1.x
>>>     > >> in a while, but I feel it will be important to make sure there
>>> are no
>>>     > >> performance regressions, and would hope that we actually have a
>>>     > >> performance
>>>     > >> improvement over previous versions. To that end (e.g. if there
>> is
>>> in
>>>     > >> fact a
>>>     > >> performance regression), the proposals that Roshan Naik put
>>> together
>>>     > >> for
>>>     > >> revising the threading and execution model (STORM-2307) and
>>> replacing
>>>     > >> Disruptor with JCTools (STORM-2306) warrant review and
>>> consideration.
>>>     > >> See
>>>     > >> also STORM-2284 which is the parent JIRA.
>>>     > >>
>>>     > >> * Finish porting Storm UI to java (STORM-1311)
>>>     > >>
>>>     > >> * Finish porting log viewer to java (STORM-1280)
>>>     > >>
>>>     > >> The following are items that are nice to have in 2.0, but I
>> don’t
>>> feel
>>>     > >> are
>>>     > >> absolutely necessary for an initial 2.0 release:
>>>     > >>
>>>     > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because
>>> it’s
>>>     > >> relevant) — Initially there seemed to be a lot of interest in
>>> this, but
>>>     > >> that seems to have trailed off. I spoke with some Beam
>> developers
>>> and
>>>     > >> there
>>>     > >> seems to be interest from that community as well. Do we want to
>>> move
>>>     > >> that
>>>     > >> effort to the Beam community, or keep it here? Moving it to the
>>> Beam
>>>     > >> community might lead to better collaboration between projects.
>>>     > >>
>>>     > >> * Bounded Spouts (needed for Beam Runner implementation) —
>>> Currently
>>>     > >> spouts are unbounded, there no end to the stream. Beam has the
>>> concept
>>>     > >> of
>>>     > >> bounded sources (roughly analogous to batch processing). To
>>> support
>>>     > >> that,
>>>     > >> we would need to implement a similar concept in Storm. One
>>> benefit of
>>>     > >> such
>>>     > >> a feature would be the ability to handle both bounded and
>>> unbounded
>>>     > >> workflows in Storm.
>>>     > >>
>>>     > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
>>> behind
>>>     > >> this
>>>     > >> effort. What improvements do you envision for 2.0?
>>>     > >>
>>>     > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
>>> targeting this
>>>     > >> for 1.2.0, but it’s designed to be easily portable to
>> master/2.0.
>>>     > >>
>>>     > >> * JStorm Migration — Original outline can be found here [1].
>> Note
>>> a lot
>>>     > >> of
>>>     > >> the associated JIRAs below are assigned, but there hasn’t been
>> any
>>>     > >> recent
>>>     > >> activity or pull requests, we should probably consider them
>>> unassigned
>>>     > >> and
>>>     > >> up for grabs.:
>>>     > >>
>>>     > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has
>>> been the
>>>     > >> bane of a lot of Storm users almost since day one. We have
>> largely
>>>     > >> addressed it by shading/relocating dependencies. It would be
>>> great to
>>>     > >> see
>>>     > >> this addressed once and for all.
>>>     > >>
>>>     > >> * JStorm back pressure implementation (STORM-1324) — The current
>>> back
>>>     > >> pressure implementation leaves a bit to be desired, and the
>> JStorm
>>>     > >> approach
>>>     > >> looks promising, though it also depends on the JStorm concept of
>>>     > >> “topology
>>>     > >> master” (STORM-1323), which may have some implications regarding
>>>     > >> security.
>>>     > >>
>>>     > >> * Dynamic Topology Updates (STORM-1335) — This would provide a
>>> command
>>>     > >> to
>>>     > >> update topology jars and configuration without stopping the
>>> topology,
>>>     > >> and
>>>     > >> is well suited to leverage the blobstore. The restart command
>>> (that can
>>>     > >> also update the topology configuration) also looks compelling
>>>     > >> (STORM-1334).
>>>     > >>
>>>     > >> * Additional Scheduler Implementations (STORM-1320)
>>>     > >>
>>>     > >> * Additional Grouping Implementations (STORM-1328)
>>>     > >>
>>>     > >>
>>>     > >> As always I’m open to any opinions and suggestions.
>>>     > >>
>>>     > >> -Taylor
>>>     > >>
>>>     > >> [1]
>>>     > >>
>>>     > >> https://cwiki.apache.org/confluence/pages/viewpage.
>>>     > >> action?pageId=61328109
>>>     > >>
>>>     > >>
>>>     > >>
>>>     > >>
>>>     > >>
>>>     > >>
>>>     > >>
>>>     > >>
>>>     > >>
>>>     > >>
>>>     >
>>> 
>>> 
>>> 


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
I see 2 ways to address this.
1) We put the 1.x line into maintenance mode like with 0.10.  We don't backport anything except bug fixes.2) We backport a lot of the backwards compatible changes from 2.x to 1.x.
My personal preference is 1.  It makes it clear the direction we want to go in.  The biggest issue is that we probably would want to do a 2.x release sooner rather then later.  Even if we don't get all of the features that people want, if we just get a release out we can add in new features if they are backwards compatible, or we can create a 3.x line that would have the breaking changes in it.

- Bobby


On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <ka...@gmail.com> wrote:

I'd like to bump this again instead of initiating new discussion thread.

I had having hard time to create and apply pull requests for both master
and 1.x-branch and that's really painful and sometimes blocker for me to do
merge step.
Two branches are heavily diverged more than between 0.10 and 1.0.0, even
IDE can't switch between the branch smoothly. We didn't even address
checkstyle issue yet, but after addressing, it could be "completely"
diverged. JDK version is another major issue, since the pull requests
targeted for master branch are not checked against JDK 7, and some of them
make some issues regarding JDK version while porting back.

So personally I really would like to see the plan for 1.x version line
changed - skipping any minor releases including 1.2.0 - and have epic issue
for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
proposed plan was having 2.0.0 directly after 1.0.0)

Would like to hear everyone's opinions. If we have consensus to not having
any minor releases for 1.x version line, I will not port back non-bugfix
pull requests to 1.x-branch, and guide contributors to create pull requests
against master branch, not 1.x version line.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <av...@gmail.com>님이
작성:

> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
> we're very interested anything that can provide better throughput.
>
> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
>
> > For 2.0 beta … it would be good to incorporate some of the Worker
> > improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
> > delivered sooner and my in-progress implementation suggests that it will
> > yield substantial latency improvements. The 2.0 beta phase would really
> > help kick the tires on the revised messaging system and the performance
> > improvements will also be a good incentive for trying out the 2.0 line.
> >
> > I notice multiple other bottlenecks that are holding back throughput a
> > lot, which can be addressed in a subsequent 2.x minor release.
> > -roshan
> >
> >
> > On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >
> >    I also would love to see metrics V2 code sooner or later too. If we
> > can get
> >    it before releasing 2.0.0 that will be great, and then maybe we could
> > just
> >    move toward to 2.0.0, not adding any improvements to 1.x version
> line.
> >    (And that's what I would want to.)
> >
> >    If we would really want to have 1.2.0, I suggest that we make the
> 1.1.1
> >    version correct right now rather than after releasing 1.1.1. We also
> > merged
> >    non-bugfix things to 1.x-branch but that's not what users expect. I
> > agree
> >    that work may be painful, but anyway need to do it.
> >
> >    - Jungtaek Lim (HeartSaVioR)
> >
> >    2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> > 작성:
> >
> >    > I would love to see the metrics V2 code come out sooner rather than
> >    > later.  +1.
> >    > My biggest blocker for a 1.x release is
> >    > https://github.com/apache/storm/pull/2142  Even though the pull
> > request
> >    > says it is minor it showed that we messed up pushing back some
> > changes for
> >    > pacemaker to open source (the code does not run at all which for me
> > is a
> >    > blocker) and I really want to get that fully fixed/tested before
> > another
> >    > release.
> >    > As for 2.x I think we are very close to being able to so a 2.x
> alpha
> >    > release.  I would like to see metrics V2 merged in simply because
> it
> > is a
> >    > big change for user facing APIs.  But after that I would love to
> see
> > us
> >    > starting to push forward on getting that out.
> >    >
> >    >
> >    > - Bobby
> >    >
> >    >
> >    > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> >    > ptgoetz@gmail.com> wrote:
> >    >
> >    > I’d like to bump this thread and start a discussion around our next
> >    > release. Here are my thoughts.
> >    >
> >    > There are a number of important fixes in 1.x-branch so I’d like to
> >    > consider releasing 1.1.1 soon. I’d appreciate input on any open
> > issues that
> >    > should be resolved for that release.
> >    >
> >    > I’d like us to consider releasing the metrics improvements in
> > STORM-2153
> >    > [1] as version 1.2.0. That work is in the metrics_v2 feature branch
> > right
> >    > now and would need to be reviewed and merged. That work is against
> > the
> >    > 1.x-branch right now. I would recommend porting it to master
> *after*
> > the
> >    > review/merge since there will likely be changes as a result of the
> > review.
> >    >
> >    > > Maybe related to or not, but would we want to create a new branch
> >    > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >    >
> >    >
> >    >
> >    > If wee agree to the above, I would say yes. After the 1.1.1
> release,
> > we
> >    > could create a 1.1.x-branch that would be the maintenance/release
> > branch
> >    > for that version line. 1.x-branch would then become the target for
> > the
> >    > 1.2.0 release.
> >    >
> >    > There are a few fixes in the 0.10.x branch that probably warrant a
> >    > release. After that we may want to back away from that version line
> > a bit
> >    > so we can focus more on newer versions.
> >    >
> >    > In the past, we’ve shied away form doing “beta” releases, but I’m
> >    > wondering if we might want to revisit that for the 2.0 release —
> the
> > idea
> >    > being that it would give early adopter users a chance to kick the
> > tires on
> >    > what’s coming in 2.0 and provide feedback, find bugs, etc. to help
> > make the
> >    > final release more solid. I’m on the fence here and could go either
> > way.
> >    >
> >    > I’d appreciate any input others may have.
> >    >
> >    >
> >    > Thanks,
> >    >
> >    > -Taylor
> >    >
> >    >
> >    > [1] https://issues.apache.org/jira/browse/STORM-2153
> >    >
> >    >
> >    >
> >    > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
> > wrote:
> >    > >
> >    > > Maybe related to or not, but would we want to create a new branch
> >    > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >    > >
> >    > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so
> > hence
> >    > the
> >    > > question.
> >    > >
> >    > > - Jungtaek Lim (HeartSaVioR)
> >    > >
> >    > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > hlouro@hortonworks.com>님이
> >    > 작성:
> >    > >
> >    > >> +1 for finishing the porting to Java ahead of anything else - it
> > will
> >    > be a
> >    > >> significant milestone. I have a JIRA assigned concerning to the
> >    > porting. I
> >    > >> will work on it for the 2.0 release.
> >    > >>
> >    > >> it’s a priority to guarantee no performance regressions. As part
> > of this
> >    > >> endeavor, explore an automated (or easy) way to run and assert
> > major
> >    > >> performance benchmarks. Ideally any contributor should be able
> to
> > fairly
> >    > >> easily test the impact of changes under certain performance test
> >    > scenarios.
> >    > >>
> >    > >> Beam Runner work should take into account the impact of
> > incorporating
> >    > new
> >    > >> JStorm features and Storm Worker Redesign<
> >    > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > efficient
> >    > to
> >    > >> start doing it, to  find out that it will have to chance in face
> > of
> >    > Storm
> >    > >> and worker redesign. That is, it should be done after it’s
> > building
> >    > blocks
> >    > >> are stable.
> >    > >>
> >    > >> Thanks,
> >    > >> Hugo
> >    > >>
> >    > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org
> > <mailto:
> >    > >> arunm@apache.org>> wrote:
> >    > >>
> >    > >> +1 to release with the porting completed. I think its mainly the
> > UI
> >    > server
> >    > >> and log viewer that’s pending.
> >    > >>
> >    > >> We can start doing the regression and performance tests for
> > whatever is
> >    > >> already ported.
> >    > >>
> >    > >> If anyone is running the master branch in their pre-prod / prod
> >    > >> environments, it will be good to know and give us more
> confidence.
> >    > >>
> >    > >> The other features can be added in follow up releases.
> >    > >>
> >    > >> Regards,
> >    > >> Arun
> >    > >>
> >    > >>
> >    > >> On 3/24/17, 11:47 AM, "Satish Duggana" <
> satish.duggana@gmail.com
> >    > <mailto:
> >    > >> satish.duggana@gmail.com>> wrote:
> >    > >>
> >    > >> +1 to have 2.0 with porting and performance(it should be at
> least
> > as
> >    > good
> >    > >> as 1.x release) issues addressed
> >    > >>
> >    > >> We can target other tasks(mentioned by Taylor and Jungtaek) for
> >    > 2.x-branch.
> >    > >>
> >    > >>
> >    > >> Exactly-once support:
> >    > >> While thinking through the exactlyonce support design, it is
> > realized
> >    > >> better to avoid acking tuples and implement exactly once by
> > snapshotting
> >    > >> barriers. It seems JStorm folks followed similar design, they
> > claim it
> >    > >> gives better performance. This feature is essential for beam
> > runner and
> >    > we
> >    > >> can decide on respective approaches though.
> >    > >>
> >    > >> Beam Runner
> >    > >> Lets hold on this for now and keep it in Storm till 2.x. We
> > should avoid
> >    > >> having a minimal beam runner in haste. It is better to address
> >    > STORM-2284,
> >    > >> exactly-once and other windowing enhancements to enable beam
> > runner.
> >    > >>
> >    > >> JStorm
> >    > >> Agree with Jungtaek on looking at the latest JStorm and
> > align/scope with
> >    > >> the features for 2.x.
> >    > >>
> >    > >> STORM-2284
> >    > >> We may want to look at JStorm worker before working on
> respective
> >    > >> components in this epic to pull appropriate enhancements.
> >    > >>
> >    > >> YARN/MESOS
> >    > >> Supporting Storm on YARN/Mesos for 2.x.
> >    > >>
> >    > >> Thanks,
> >    > >> Satish.
> >    > >>
> >    > >>
> >    > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> kabhwan@gmail.com
> >    > <mailto:
> >    > >> kabhwan@gmail.com>> wrote:
> >    > >>
> >    > >> First of all, +1 to complete only port work and do sanity check
> >    > (including
> >    > >> performance regression), and release.
> >    > >>
> >    > >> If we can get STORM-2284 within deterministic time frame (say
> 2~3
> >    > months)
> >    > >> that should be great, but if not I'd in favor of postponing that
> > to
> >    > later
> >    > >> 2.x release.
> >    > >>
> >    > >> JStorm released their new versions after code donation. So
> > there're more
> >    > >> things we could get ideas from, or even adopt from.
> >    > >> https://github.com/alibaba/jstorm/blob/master/history.md
> >    > >> As you noticed from release note link, we also need to update
> > phase 2
> >    > since
> >    > >> they already changed what we're planning to do in phase 2. For
> > example,
> >    > >> they changed backpressure to end-to-end, and changed to use
> > snapshot
> >    > rather
> >    > >> than acker.
> >    > >> May be sure, JStorm pulled many features from today's Storm,
> like
> > Flux,
> >    > >> Windowing, more shuffle groupings, log search, log level change,
> > and so
> >    > on.
> >    > >>
> >    > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> is
> > due to
> >    > >> the
> >    > >> limitation of Spout lifecycle (all the things are done in single
> >    > thread),
> >    > >> and STORM-1358 <
> https://issues.apache.org/jira/browse/STORM-1358
> >    > >(JStorm's
> >    > >> multi-thread Spout) can remedy this (despite that Spout
> > implementation
> >    > may
> >    > >> need to guarantee thread-safety later). It's not a just
> > improvement but
> >    > >> close to design concern so would like to address sooner than
> other
> >    > things
> >    > >> in phase 2.
> >    > >>
> >    > >> For Storm SQL side, I've lost progress but major work would be
> > adopting
> >    > >> group by with windowing. It was not available from Calcite but
> > will be
> >    > >> available at next release (1.12.0).
> >    > >> I've filed this to STORM-2405
> >    > >> <https://issues.apache.org/jira/browse/STORM-2405>, but
> > windowing &
> >    > micro
> >    > >> batch is not intuitive, so I would like to change the underlying
> > API to
> >    > >> stream API in SQL. Also filed this to STORM-2406
> >    > >> <https://issues.apache.org/jira/browse/STORM-2406>.
> >    > >>
> >    > >> Just 2 cents btw, hopefully I would like to see metrics V2
> sooner
> > since
> >    > we
> >    > >> lost metrics even when doing normal operation like restarting
> > worker,
> >    > >> rebalancing, and so on. Eventually we need to fight with dynamic
> >    > scaling,
> >    > >> and then metrics will be broken often.
> >    > >>
> >    > >> Thanks,
> >    > >> Jungtaek Lim (HeartSaVioR)
> >    > >>
> >    > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <storm@harsha.io
> >님이
> > 작성:
> >    > >>
> >    > >> Storm 2.0 migration to java in itself is a big win and would
> > attract
> >    > >> wider
> >    > >> community and adoption. So my vote would be to resolve the first
> > 3 items
> >    > >> to
> >    > >> get a release out.
> >    > >> All the other featured mentioned are great to have but shouldn't
> > be
> >    > >> blockers for 2.0 release.
> >    > >>
> >    > >> -Harsha
> >    > >>
> >    > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > ptgoetz@gmail.com>
> >    > >> wrote:
> >    > >>
> >    > >> With the 1.1.0 release nearing completion, I’d like to turn our
> >    > >> attention
> >    > >> to 2.0 and develop a plan for what features, etc. to include.
> >    > >>
> >    > >> The following 3 are what I feel are the minimum for a 2.0
> release.
> >    > >> These
> >    > >> could likely be resolved relatively quickly:
> >    > >>
> >    > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x
> > or
> >    > >> 1.1.x
> >    > >> in a while, but I feel it will be important to make sure there
> > are no
> >    > >> performance regressions, and would hope that we actually have a
> >    > >> performance
> >    > >> improvement over previous versions. To that end (e.g. if there
> is
> > in
> >    > >> fact a
> >    > >> performance regression), the proposals that Roshan Naik put
> > together
> >    > >> for
> >    > >> revising the threading and execution model (STORM-2307) and
> > replacing
> >    > >> Disruptor with JCTools (STORM-2306) warrant review and
> > consideration.
> >    > >> See
> >    > >> also STORM-2284 which is the parent JIRA.
> >    > >>
> >    > >> * Finish porting Storm UI to java (STORM-1311)
> >    > >>
> >    > >> * Finish porting log viewer to java (STORM-1280)
> >    > >>
> >    > >> The following are items that are nice to have in 2.0, but I
> don’t
> > feel
> >    > >> are
> >    > >> absolutely necessary for an initial 2.0 release:
> >    > >>
> >    > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because
> > it’s
> >    > >> relevant) — Initially there seemed to be a lot of interest in
> > this, but
> >    > >> that seems to have trailed off. I spoke with some Beam
> developers
> > and
> >    > >> there
> >    > >> seems to be interest from that community as well. Do we want to
> > move
> >    > >> that
> >    > >> effort to the Beam community, or keep it here? Moving it to the
> > Beam
> >    > >> community might lead to better collaboration between projects.
> >    > >>
> >    > >> * Bounded Spouts (needed for Beam Runner implementation) —
> > Currently
> >    > >> spouts are unbounded, there no end to the stream. Beam has the
> > concept
> >    > >> of
> >    > >> bounded sources (roughly analogous to batch processing). To
> > support
> >    > >> that,
> >    > >> we would need to implement a similar concept in Storm. One
> > benefit of
> >    > >> such
> >    > >> a feature would be the ability to handle both bounded and
> > unbounded
> >    > >> workflows in Storm.
> >    > >>
> >    > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > behind
> >    > >> this
> >    > >> effort. What improvements do you envision for 2.0?
> >    > >>
> >    > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > targeting this
> >    > >> for 1.2.0, but it’s designed to be easily portable to
> master/2.0.
> >    > >>
> >    > >> * JStorm Migration — Original outline can be found here [1].
> Note
> > a lot
> >    > >> of
> >    > >> the associated JIRAs below are assigned, but there hasn’t been
> any
> >    > >> recent
> >    > >> activity or pull requests, we should probably consider them
> > unassigned
> >    > >> and
> >    > >> up for grabs.:
> >    > >>
> >    > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > been the
> >    > >> bane of a lot of Storm users almost since day one. We have
> largely
> >    > >> addressed it by shading/relocating dependencies. It would be
> > great to
> >    > >> see
> >    > >> this addressed once and for all.
> >    > >>
> >    > >> * JStorm back pressure implementation (STORM-1324) — The current
> > back
> >    > >> pressure implementation leaves a bit to be desired, and the
> JStorm
> >    > >> approach
> >    > >> looks promising, though it also depends on the JStorm concept of
> >    > >> “topology
> >    > >> master” (STORM-1323), which may have some implications regarding
> >    > >> security.
> >    > >>
> >    > >> * Dynamic Topology Updates (STORM-1335) — This would provide a
> > command
> >    > >> to
> >    > >> update topology jars and configuration without stopping the
> > topology,
> >    > >> and
> >    > >> is well suited to leverage the blobstore. The restart command
> > (that can
> >    > >> also update the topology configuration) also looks compelling
> >    > >> (STORM-1334).
> >    > >>
> >    > >> * Additional Scheduler Implementations (STORM-1320)
> >    > >>
> >    > >> * Additional Grouping Implementations (STORM-1328)
> >    > >>
> >    > >>
> >    > >> As always I’m open to any opinions and suggestions.
> >    > >>
> >    > >> -Taylor
> >    > >>
> >    > >> [1]
> >    > >>
> >    > >> https://cwiki.apache.org/confluence/pages/viewpage.
> >    > >> action?pageId=61328109
> >    > >>
> >    > >>
> >    > >>
> >    > >>
> >    > >>
> >    > >>
> >    > >>
> >    > >>
> >    > >>
> >    > >>
> >    >
> >
> >
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Jungtaek Lim <ka...@gmail.com>.
I'd like to bump this again instead of initiating new discussion thread.

I had having hard time to create and apply pull requests for both master
and 1.x-branch and that's really painful and sometimes blocker for me to do
merge step.
Two branches are heavily diverged more than between 0.10 and 1.0.0, even
IDE can't switch between the branch smoothly. We didn't even address
checkstyle issue yet, but after addressing, it could be "completely"
diverged. JDK version is another major issue, since the pull requests
targeted for master branch are not checked against JDK 7, and some of them
make some issues regarding JDK version while porting back.

So personally I really would like to see the plan for 1.x version line
changed - skipping any minor releases including 1.2.0 - and have epic issue
for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
proposed plan was having 2.0.0 directly after 1.0.0)

Would like to hear everyone's opinions. If we have consensus to not having
any minor releases for 1.x version line, I will not port back non-bugfix
pull requests to 1.x-branch, and guide contributors to create pull requests
against master branch, not 1.x version line.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <av...@gmail.com>님이
작성:

> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
> we're very interested anything that can provide better throughput.
>
> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:
>
> > For 2.0 beta … it would be good to incorporate some of the Worker
> > improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
> > delivered sooner and my in-progress implementation suggests that it will
> > yield substantial latency improvements. The 2.0 beta phase would really
> > help kick the tires on the revised messaging system and the performance
> > improvements will also be a good incentive for trying out the 2.0 line.
> >
> > I notice multiple other bottlenecks that are holding back throughput a
> > lot, which can be addressed in a subsequent 2.x minor release.
> > -roshan
> >
> >
> > On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
> >
> >     I also would love to see metrics V2 code sooner or later too. If we
> > can get
> >     it before releasing 2.0.0 that will be great, and then maybe we could
> > just
> >     move toward to 2.0.0, not adding any improvements to 1.x version
> line.
> >     (And that's what I would want to.)
> >
> >     If we would really want to have 1.2.0, I suggest that we make the
> 1.1.1
> >     version correct right now rather than after releasing 1.1.1. We also
> > merged
> >     non-bugfix things to 1.x-branch but that's not what users expect. I
> > agree
> >     that work may be painful, but anyway need to do it.
> >
> >     - Jungtaek Lim (HeartSaVioR)
> >
> >     2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> > 작성:
> >
> >     > I would love to see the metrics V2 code come out sooner rather than
> >     > later.  +1.
> >     > My biggest blocker for a 1.x release is
> >     > https://github.com/apache/storm/pull/2142  Even though the pull
> > request
> >     > says it is minor it showed that we messed up pushing back some
> > changes for
> >     > pacemaker to open source (the code does not run at all which for me
> > is a
> >     > blocker) and I really want to get that fully fixed/tested before
> > another
> >     > release.
> >     > As for 2.x I think we are very close to being able to so a 2.x
> alpha
> >     > release.  I would like to see metrics V2 merged in simply because
> it
> > is a
> >     > big change for user facing APIs.  But after that I would love to
> see
> > us
> >     > starting to push forward on getting that out.
> >     >
> >     >
> >     > - Bobby
> >     >
> >     >
> >     > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> >     > ptgoetz@gmail.com> wrote:
> >     >
> >     > I’d like to bump this thread and start a discussion around our next
> >     > release. Here are my thoughts.
> >     >
> >     > There are a number of important fixes in 1.x-branch so I’d like to
> >     > consider releasing 1.1.1 soon. I’d appreciate input on any open
> > issues that
> >     > should be resolved for that release.
> >     >
> >     > I’d like us to consider releasing the metrics improvements in
> > STORM-2153
> >     > [1] as version 1.2.0. That work is in the metrics_v2 feature branch
> > right
> >     > now and would need to be reviewed and merged. That work is against
> > the
> >     > 1.x-branch right now. I would recommend porting it to master
> *after*
> > the
> >     > review/merge since there will likely be changes as a result of the
> > review.
> >     >
> >     > > Maybe related to or not, but would we want to create a new branch
> >     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >     >
> >     >
> >     >
> >     > If wee agree to the above, I would say yes. After the 1.1.1
> release,
> > we
> >     > could create a 1.1.x-branch that would be the maintenance/release
> > branch
> >     > for that version line. 1.x-branch would then become the target for
> > the
> >     > 1.2.0 release.
> >     >
> >     > There are a few fixes in the 0.10.x branch that probably warrant a
> >     > release. After that we may want to back away from that version line
> > a bit
> >     > so we can focus more on newer versions.
> >     >
> >     > In the past, we’ve shied away form doing “beta” releases, but I’m
> >     > wondering if we might want to revisit that for the 2.0 release —
> the
> > idea
> >     > being that it would give early adopter users a chance to kick the
> > tires on
> >     > what’s coming in 2.0 and provide feedback, find bugs, etc. to help
> > make the
> >     > final release more solid. I’m on the fence here and could go either
> > way.
> >     >
> >     > I’d appreciate any input others may have.
> >     >
> >     >
> >     > Thanks,
> >     >
> >     > -Taylor
> >     >
> >     >
> >     > [1] https://issues.apache.org/jira/browse/STORM-2153
> >     >
> >     >
> >     >
> >     > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
> > wrote:
> >     > >
> >     > > Maybe related to or not, but would we want to create a new branch
> >     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >     > >
> >     > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so
> > hence
> >     > the
> >     > > question.
> >     > >
> >     > > - Jungtaek Lim (HeartSaVioR)
> >     > >
> >     > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> > hlouro@hortonworks.com>님이
> >     > 작성:
> >     > >
> >     > >> +1 for finishing the porting to Java ahead of anything else - it
> > will
> >     > be a
> >     > >> significant milestone. I have a JIRA assigned concerning to the
> >     > porting. I
> >     > >> will work on it for the 2.0 release.
> >     > >>
> >     > >> it’s a priority to guarantee no performance regressions. As part
> > of this
> >     > >> endeavor, explore an automated (or easy) way to run and assert
> > major
> >     > >> performance benchmarks. Ideally any contributor should be able
> to
> > fairly
> >     > >> easily test the impact of changes under certain performance test
> >     > scenarios.
> >     > >>
> >     > >> Beam Runner work should take into account the impact of
> > incorporating
> >     > new
> >     > >> JStorm features and Storm Worker Redesign<
> >     > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> > efficient
> >     > to
> >     > >> start doing it, to  find out that it will have to chance in face
> > of
> >     > Storm
> >     > >> and worker redesign. That is, it should be done after it’s
> > building
> >     > blocks
> >     > >> are stable.
> >     > >>
> >     > >> Thanks,
> >     > >> Hugo
> >     > >>
> >     > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org
> > <mailto:
> >     > >> arunm@apache.org>> wrote:
> >     > >>
> >     > >> +1 to release with the porting completed. I think its mainly the
> > UI
> >     > server
> >     > >> and log viewer that’s pending.
> >     > >>
> >     > >> We can start doing the regression and performance tests for
> > whatever is
> >     > >> already ported.
> >     > >>
> >     > >> If anyone is running the master branch in their pre-prod / prod
> >     > >> environments, it will be good to know and give us more
> confidence.
> >     > >>
> >     > >> The other features can be added in follow up releases.
> >     > >>
> >     > >> Regards,
> >     > >> Arun
> >     > >>
> >     > >>
> >     > >> On 3/24/17, 11:47 AM, "Satish Duggana" <
> satish.duggana@gmail.com
> >     > <mailto:
> >     > >> satish.duggana@gmail.com>> wrote:
> >     > >>
> >     > >> +1 to have 2.0 with porting and performance(it should be at
> least
> > as
> >     > good
> >     > >> as 1.x release) issues addressed
> >     > >>
> >     > >> We can target other tasks(mentioned by Taylor and Jungtaek) for
> >     > 2.x-branch.
> >     > >>
> >     > >>
> >     > >> Exactly-once support:
> >     > >> While thinking through the exactlyonce support design, it is
> > realized
> >     > >> better to avoid acking tuples and implement exactly once by
> > snapshotting
> >     > >> barriers. It seems JStorm folks followed similar design, they
> > claim it
> >     > >> gives better performance. This feature is essential for beam
> > runner and
> >     > we
> >     > >> can decide on respective approaches though.
> >     > >>
> >     > >> Beam Runner
> >     > >> Lets hold on this for now and keep it in Storm till 2.x. We
> > should avoid
> >     > >> having a minimal beam runner in haste. It is better to address
> >     > STORM-2284,
> >     > >> exactly-once and other windowing enhancements to enable beam
> > runner.
> >     > >>
> >     > >> JStorm
> >     > >> Agree with Jungtaek on looking at the latest JStorm and
> > align/scope with
> >     > >> the features for 2.x.
> >     > >>
> >     > >> STORM-2284
> >     > >> We may want to look at JStorm worker before working on
> respective
> >     > >> components in this epic to pull appropriate enhancements.
> >     > >>
> >     > >> YARN/MESOS
> >     > >> Supporting Storm on YARN/Mesos for 2.x.
> >     > >>
> >     > >> Thanks,
> >     > >> Satish.
> >     > >>
> >     > >>
> >     > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <
> kabhwan@gmail.com
> >     > <mailto:
> >     > >> kabhwan@gmail.com>> wrote:
> >     > >>
> >     > >> First of all, +1 to complete only port work and do sanity check
> >     > (including
> >     > >> performance regression), and release.
> >     > >>
> >     > >> If we can get STORM-2284 within deterministic time frame (say
> 2~3
> >     > months)
> >     > >> that should be great, but if not I'd in favor of postponing that
> > to
> >     > later
> >     > >> 2.x release.
> >     > >>
> >     > >> JStorm released their new versions after code donation. So
> > there're more
> >     > >> things we could get ideas from, or even adopt from.
> >     > >> https://github.com/alibaba/jstorm/blob/master/history.md
> >     > >> As you noticed from release note link, we also need to update
> > phase 2
> >     > since
> >     > >> they already changed what we're planning to do in phase 2. For
> > example,
> >     > >> they changed backpressure to end-to-end, and changed to use
> > snapshot
> >     > rather
> >     > >> than acker.
> >     > >> May be sure, JStorm pulled many features from today's Storm,
> like
> > Flux,
> >     > >> Windowing, more shuffle groupings, log search, log level change,
> > and so
> >     > on.
> >     > >>
> >     > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426>
> is
> > due to
> >     > >> the
> >     > >> limitation of Spout lifecycle (all the things are done in single
> >     > thread),
> >     > >> and STORM-1358 <
> https://issues.apache.org/jira/browse/STORM-1358
> >     > >(JStorm's
> >     > >> multi-thread Spout) can remedy this (despite that Spout
> > implementation
> >     > may
> >     > >> need to guarantee thread-safety later). It's not a just
> > improvement but
> >     > >> close to design concern so would like to address sooner than
> other
> >     > things
> >     > >> in phase 2.
> >     > >>
> >     > >> For Storm SQL side, I've lost progress but major work would be
> > adopting
> >     > >> group by with windowing. It was not available from Calcite but
> > will be
> >     > >> available at next release (1.12.0).
> >     > >> I've filed this to STORM-2405
> >     > >> <https://issues.apache.org/jira/browse/STORM-2405>, but
> > windowing &
> >     > micro
> >     > >> batch is not intuitive, so I would like to change the underlying
> > API to
> >     > >> stream API in SQL. Also filed this to STORM-2406
> >     > >> <https://issues.apache.org/jira/browse/STORM-2406>.
> >     > >>
> >     > >> Just 2 cents btw, hopefully I would like to see metrics V2
> sooner
> > since
> >     > we
> >     > >> lost metrics even when doing normal operation like restarting
> > worker,
> >     > >> rebalancing, and so on. Eventually we need to fight with dynamic
> >     > scaling,
> >     > >> and then metrics will be broken often.
> >     > >>
> >     > >> Thanks,
> >     > >> Jungtaek Lim (HeartSaVioR)
> >     > >>
> >     > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <storm@harsha.io
> >님이
> > 작성:
> >     > >>
> >     > >> Storm 2.0 migration to java in itself is a big win and would
> > attract
> >     > >> wider
> >     > >> community and adoption. So my vote would be to resolve the first
> > 3 items
> >     > >> to
> >     > >> get a release out.
> >     > >> All the other featured mentioned are great to have but shouldn't
> > be
> >     > >> blockers for 2.0 release.
> >     > >>
> >     > >> -Harsha
> >     > >>
> >     > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> > ptgoetz@gmail.com>
> >     > >> wrote:
> >     > >>
> >     > >> With the 1.1.0 release nearing completion, I’d like to turn our
> >     > >> attention
> >     > >> to 2.0 and develop a plan for what features, etc. to include.
> >     > >>
> >     > >> The following 3 are what I feel are the minimum for a 2.0
> release.
> >     > >> These
> >     > >> could likely be resolved relatively quickly:
> >     > >>
> >     > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x
> > or
> >     > >> 1.1.x
> >     > >> in a while, but I feel it will be important to make sure there
> > are no
> >     > >> performance regressions, and would hope that we actually have a
> >     > >> performance
> >     > >> improvement over previous versions. To that end (e.g. if there
> is
> > in
> >     > >> fact a
> >     > >> performance regression), the proposals that Roshan Naik put
> > together
> >     > >> for
> >     > >> revising the threading and execution model (STORM-2307) and
> > replacing
> >     > >> Disruptor with JCTools (STORM-2306) warrant review and
> > consideration.
> >     > >> See
> >     > >> also STORM-2284 which is the parent JIRA.
> >     > >>
> >     > >> * Finish porting Storm UI to java (STORM-1311)
> >     > >>
> >     > >> * Finish porting log viewer to java (STORM-1280)
> >     > >>
> >     > >> The following are items that are nice to have in 2.0, but I
> don’t
> > feel
> >     > >> are
> >     > >> absolutely necessary for an initial 2.0 release:
> >     > >>
> >     > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because
> > it’s
> >     > >> relevant) — Initially there seemed to be a lot of interest in
> > this, but
> >     > >> that seems to have trailed off. I spoke with some Beam
> developers
> > and
> >     > >> there
> >     > >> seems to be interest from that community as well. Do we want to
> > move
> >     > >> that
> >     > >> effort to the Beam community, or keep it here? Moving it to the
> > Beam
> >     > >> community might lead to better collaboration between projects.
> >     > >>
> >     > >> * Bounded Spouts (needed for Beam Runner implementation) —
> > Currently
> >     > >> spouts are unbounded, there no end to the stream. Beam has the
> > concept
> >     > >> of
> >     > >> bounded sources (roughly analogous to batch processing). To
> > support
> >     > >> that,
> >     > >> we would need to implement a similar concept in Storm. One
> > benefit of
> >     > >> such
> >     > >> a feature would be the ability to handle both bounded and
> > unbounded
> >     > >> workflows in Storm.
> >     > >>
> >     > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> > behind
> >     > >> this
> >     > >> effort. What improvements do you envision for 2.0?
> >     > >>
> >     > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> > targeting this
> >     > >> for 1.2.0, but it’s designed to be easily portable to
> master/2.0.
> >     > >>
> >     > >> * JStorm Migration — Original outline can be found here [1].
> Note
> > a lot
> >     > >> of
> >     > >> the associated JIRAs below are assigned, but there hasn’t been
> any
> >     > >> recent
> >     > >> activity or pull requests, we should probably consider them
> > unassigned
> >     > >> and
> >     > >> up for grabs.:
> >     > >>
> >     > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> > been the
> >     > >> bane of a lot of Storm users almost since day one. We have
> largely
> >     > >> addressed it by shading/relocating dependencies. It would be
> > great to
> >     > >> see
> >     > >> this addressed once and for all.
> >     > >>
> >     > >> * JStorm back pressure implementation (STORM-1324) — The current
> > back
> >     > >> pressure implementation leaves a bit to be desired, and the
> JStorm
> >     > >> approach
> >     > >> looks promising, though it also depends on the JStorm concept of
> >     > >> “topology
> >     > >> master” (STORM-1323), which may have some implications regarding
> >     > >> security.
> >     > >>
> >     > >> * Dynamic Topology Updates (STORM-1335) — This would provide a
> > command
> >     > >> to
> >     > >> update topology jars and configuration without stopping the
> > topology,
> >     > >> and
> >     > >> is well suited to leverage the blobstore. The restart command
> > (that can
> >     > >> also update the topology configuration) also looks compelling
> >     > >> (STORM-1334).
> >     > >>
> >     > >> * Additional Scheduler Implementations (STORM-1320)
> >     > >>
> >     > >> * Additional Grouping Implementations (STORM-1328)
> >     > >>
> >     > >>
> >     > >> As always I’m open to any opinions and suggestions.
> >     > >>
> >     > >> -Taylor
> >     > >>
> >     > >> [1]
> >     > >>
> >     > >> https://cwiki.apache.org/confluence/pages/viewpage.
> >     > >> action?pageId=61328109
> >     > >>
> >     > >>
> >     > >>
> >     > >>
> >     > >>
> >     > >>
> >     > >>
> >     > >>
> >     > >>
> >     > >>
> >     >
> >
> >
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Alexandre Vermeerbergen <av...@gmail.com>.
+1 for Roshan's suggestion : in our Storm 1.x based supervision system,
we're very interested anything that can provide better throughput.

2017-06-03 18:12 GMT+02:00 Roshan Naik <ro...@hortonworks.com>:

> For 2.0 beta … it would be good to incorporate some of the Worker
> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
> delivered sooner and my in-progress implementation suggests that it will
> yield substantial latency improvements. The 2.0 beta phase would really
> help kick the tires on the revised messaging system and the performance
> improvements will also be a good incentive for trying out the 2.0 line.
>
> I notice multiple other bottlenecks that are holding back throughput a
> lot, which can be addressed in a subsequent 2.x minor release.
> -roshan
>
>
> On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>
>     I also would love to see metrics V2 code sooner or later too. If we
> can get
>     it before releasing 2.0.0 that will be great, and then maybe we could
> just
>     move toward to 2.0.0, not adding any improvements to 1.x version line.
>     (And that's what I would want to.)
>
>     If we would really want to have 1.2.0, I suggest that we make the 1.1.1
>     version correct right now rather than after releasing 1.1.1. We also
> merged
>     non-bugfix things to 1.x-branch but that's not what users expect. I
> agree
>     that work may be painful, but anyway need to do it.
>
>     - Jungtaek Lim (HeartSaVioR)
>
>     2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> 작성:
>
>     > I would love to see the metrics V2 code come out sooner rather than
>     > later.  +1.
>     > My biggest blocker for a 1.x release is
>     > https://github.com/apache/storm/pull/2142  Even though the pull
> request
>     > says it is minor it showed that we messed up pushing back some
> changes for
>     > pacemaker to open source (the code does not run at all which for me
> is a
>     > blocker) and I really want to get that fully fixed/tested before
> another
>     > release.
>     > As for 2.x I think we are very close to being able to so a 2.x alpha
>     > release.  I would like to see metrics V2 merged in simply because it
> is a
>     > big change for user facing APIs.  But after that I would love to see
> us
>     > starting to push forward on getting that out.
>     >
>     >
>     > - Bobby
>     >
>     >
>     > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
>     > ptgoetz@gmail.com> wrote:
>     >
>     > I’d like to bump this thread and start a discussion around our next
>     > release. Here are my thoughts.
>     >
>     > There are a number of important fixes in 1.x-branch so I’d like to
>     > consider releasing 1.1.1 soon. I’d appreciate input on any open
> issues that
>     > should be resolved for that release.
>     >
>     > I’d like us to consider releasing the metrics improvements in
> STORM-2153
>     > [1] as version 1.2.0. That work is in the metrics_v2 feature branch
> right
>     > now and would need to be reviewed and merged. That work is against
> the
>     > 1.x-branch right now. I would recommend porting it to master *after*
> the
>     > review/merge since there will likely be changes as a result of the
> review.
>     >
>     > > Maybe related to or not, but would we want to create a new branch
>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>     >
>     >
>     >
>     > If wee agree to the above, I would say yes. After the 1.1.1 release,
> we
>     > could create a 1.1.x-branch that would be the maintenance/release
> branch
>     > for that version line. 1.x-branch would then become the target for
> the
>     > 1.2.0 release.
>     >
>     > There are a few fixes in the 0.10.x branch that probably warrant a
>     > release. After that we may want to back away from that version line
> a bit
>     > so we can focus more on newer versions.
>     >
>     > In the past, we’ve shied away form doing “beta” releases, but I’m
>     > wondering if we might want to revisit that for the 2.0 release — the
> idea
>     > being that it would give early adopter users a chance to kick the
> tires on
>     > what’s coming in 2.0 and provide feedback, find bugs, etc. to help
> make the
>     > final release more solid. I’m on the fence here and could go either
> way.
>     >
>     > I’d appreciate any input others may have.
>     >
>     >
>     > Thanks,
>     >
>     > -Taylor
>     >
>     >
>     > [1] https://issues.apache.org/jira/browse/STORM-2153
>     >
>     >
>     >
>     > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com>
> wrote:
>     > >
>     > > Maybe related to or not, but would we want to create a new branch
>     > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>     > >
>     > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so
> hence
>     > the
>     > > question.
>     > >
>     > > - Jungtaek Lim (HeartSaVioR)
>     > >
>     > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <
> hlouro@hortonworks.com>님이
>     > 작성:
>     > >
>     > >> +1 for finishing the porting to Java ahead of anything else - it
> will
>     > be a
>     > >> significant milestone. I have a JIRA assigned concerning to the
>     > porting. I
>     > >> will work on it for the 2.0 release.
>     > >>
>     > >> it’s a priority to guarantee no performance regressions. As part
> of this
>     > >> endeavor, explore an automated (or easy) way to run and assert
> major
>     > >> performance benchmarks. Ideally any contributor should be able to
> fairly
>     > >> easily test the impact of changes under certain performance test
>     > scenarios.
>     > >>
>     > >> Beam Runner work should take into account the impact of
> incorporating
>     > new
>     > >> JStorm features and Storm Worker Redesign<
>     > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very
> efficient
>     > to
>     > >> start doing it, to  find out that it will have to chance in face
> of
>     > Storm
>     > >> and worker redesign. That is, it should be done after it’s
> building
>     > blocks
>     > >> are stable.
>     > >>
>     > >> Thanks,
>     > >> Hugo
>     > >>
>     > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org
> <mailto:
>     > >> arunm@apache.org>> wrote:
>     > >>
>     > >> +1 to release with the porting completed. I think its mainly the
> UI
>     > server
>     > >> and log viewer that’s pending.
>     > >>
>     > >> We can start doing the regression and performance tests for
> whatever is
>     > >> already ported.
>     > >>
>     > >> If anyone is running the master branch in their pre-prod / prod
>     > >> environments, it will be good to know and give us more confidence.
>     > >>
>     > >> The other features can be added in follow up releases.
>     > >>
>     > >> Regards,
>     > >> Arun
>     > >>
>     > >>
>     > >> On 3/24/17, 11:47 AM, "Satish Duggana" <satish.duggana@gmail.com
>     > <mailto:
>     > >> satish.duggana@gmail.com>> wrote:
>     > >>
>     > >> +1 to have 2.0 with porting and performance(it should be at least
> as
>     > good
>     > >> as 1.x release) issues addressed
>     > >>
>     > >> We can target other tasks(mentioned by Taylor and Jungtaek) for
>     > 2.x-branch.
>     > >>
>     > >>
>     > >> Exactly-once support:
>     > >> While thinking through the exactlyonce support design, it is
> realized
>     > >> better to avoid acking tuples and implement exactly once by
> snapshotting
>     > >> barriers. It seems JStorm folks followed similar design, they
> claim it
>     > >> gives better performance. This feature is essential for beam
> runner and
>     > we
>     > >> can decide on respective approaches though.
>     > >>
>     > >> Beam Runner
>     > >> Lets hold on this for now and keep it in Storm till 2.x. We
> should avoid
>     > >> having a minimal beam runner in haste. It is better to address
>     > STORM-2284,
>     > >> exactly-once and other windowing enhancements to enable beam
> runner.
>     > >>
>     > >> JStorm
>     > >> Agree with Jungtaek on looking at the latest JStorm and
> align/scope with
>     > >> the features for 2.x.
>     > >>
>     > >> STORM-2284
>     > >> We may want to look at JStorm worker before working on respective
>     > >> components in this epic to pull appropriate enhancements.
>     > >>
>     > >> YARN/MESOS
>     > >> Supporting Storm on YARN/Mesos for 2.x.
>     > >>
>     > >> Thanks,
>     > >> Satish.
>     > >>
>     > >>
>     > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <kabhwan@gmail.com
>     > <mailto:
>     > >> kabhwan@gmail.com>> wrote:
>     > >>
>     > >> First of all, +1 to complete only port work and do sanity check
>     > (including
>     > >> performance regression), and release.
>     > >>
>     > >> If we can get STORM-2284 within deterministic time frame (say 2~3
>     > months)
>     > >> that should be great, but if not I'd in favor of postponing that
> to
>     > later
>     > >> 2.x release.
>     > >>
>     > >> JStorm released their new versions after code donation. So
> there're more
>     > >> things we could get ideas from, or even adopt from.
>     > >> https://github.com/alibaba/jstorm/blob/master/history.md
>     > >> As you noticed from release note link, we also need to update
> phase 2
>     > since
>     > >> they already changed what we're planning to do in phase 2. For
> example,
>     > >> they changed backpressure to end-to-end, and changed to use
> snapshot
>     > rather
>     > >> than acker.
>     > >> May be sure, JStorm pulled many features from today's Storm, like
> Flux,
>     > >> Windowing, more shuffle groupings, log search, log level change,
> and so
>     > on.
>     > >>
>     > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> is
> due to
>     > >> the
>     > >> limitation of Spout lifecycle (all the things are done in single
>     > thread),
>     > >> and STORM-1358 <https://issues.apache.org/jira/browse/STORM-1358
>     > >(JStorm's
>     > >> multi-thread Spout) can remedy this (despite that Spout
> implementation
>     > may
>     > >> need to guarantee thread-safety later). It's not a just
> improvement but
>     > >> close to design concern so would like to address sooner than other
>     > things
>     > >> in phase 2.
>     > >>
>     > >> For Storm SQL side, I've lost progress but major work would be
> adopting
>     > >> group by with windowing. It was not available from Calcite but
> will be
>     > >> available at next release (1.12.0).
>     > >> I've filed this to STORM-2405
>     > >> <https://issues.apache.org/jira/browse/STORM-2405>, but
> windowing &
>     > micro
>     > >> batch is not intuitive, so I would like to change the underlying
> API to
>     > >> stream API in SQL. Also filed this to STORM-2406
>     > >> <https://issues.apache.org/jira/browse/STORM-2406>.
>     > >>
>     > >> Just 2 cents btw, hopefully I would like to see metrics V2 sooner
> since
>     > we
>     > >> lost metrics even when doing normal operation like restarting
> worker,
>     > >> rebalancing, and so on. Eventually we need to fight with dynamic
>     > scaling,
>     > >> and then metrics will be broken often.
>     > >>
>     > >> Thanks,
>     > >> Jungtaek Lim (HeartSaVioR)
>     > >>
>     > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <st...@harsha.io>님이
> 작성:
>     > >>
>     > >> Storm 2.0 migration to java in itself is a big win and would
> attract
>     > >> wider
>     > >> community and adoption. So my vote would be to resolve the first
> 3 items
>     > >> to
>     > >> get a release out.
>     > >> All the other featured mentioned are great to have but shouldn't
> be
>     > >> blockers for 2.0 release.
>     > >>
>     > >> -Harsha
>     > >>
>     > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <
> ptgoetz@gmail.com>
>     > >> wrote:
>     > >>
>     > >> With the 1.1.0 release nearing completion, I’d like to turn our
>     > >> attention
>     > >> to 2.0 and develop a plan for what features, etc. to include.
>     > >>
>     > >> The following 3 are what I feel are the minimum for a 2.0 release.
>     > >> These
>     > >> could likely be resolved relatively quickly:
>     > >>
>     > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x
> or
>     > >> 1.1.x
>     > >> in a while, but I feel it will be important to make sure there
> are no
>     > >> performance regressions, and would hope that we actually have a
>     > >> performance
>     > >> improvement over previous versions. To that end (e.g. if there is
> in
>     > >> fact a
>     > >> performance regression), the proposals that Roshan Naik put
> together
>     > >> for
>     > >> revising the threading and execution model (STORM-2307) and
> replacing
>     > >> Disruptor with JCTools (STORM-2306) warrant review and
> consideration.
>     > >> See
>     > >> also STORM-2284 which is the parent JIRA.
>     > >>
>     > >> * Finish porting Storm UI to java (STORM-1311)
>     > >>
>     > >> * Finish porting log viewer to java (STORM-1280)
>     > >>
>     > >> The following are items that are nice to have in 2.0, but I don’t
> feel
>     > >> are
>     > >> absolutely necessary for an initial 2.0 release:
>     > >>
>     > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because
> it’s
>     > >> relevant) — Initially there seemed to be a lot of interest in
> this, but
>     > >> that seems to have trailed off. I spoke with some Beam developers
> and
>     > >> there
>     > >> seems to be interest from that community as well. Do we want to
> move
>     > >> that
>     > >> effort to the Beam community, or keep it here? Moving it to the
> Beam
>     > >> community might lead to better collaboration between projects.
>     > >>
>     > >> * Bounded Spouts (needed for Beam Runner implementation) —
> Currently
>     > >> spouts are unbounded, there no end to the stream. Beam has the
> concept
>     > >> of
>     > >> bounded sources (roughly analogous to batch processing). To
> support
>     > >> that,
>     > >> we would need to implement a similar concept in Storm. One
> benefit of
>     > >> such
>     > >> a feature would be the ability to handle both bounded and
> unbounded
>     > >> workflows in Storm.
>     > >>
>     > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers
> behind
>     > >> this
>     > >> effort. What improvements do you envision for 2.0?
>     > >>
>     > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been
> targeting this
>     > >> for 1.2.0, but it’s designed to be easily portable to master/2.0.
>     > >>
>     > >> * JStorm Migration — Original outline can be found here [1]. Note
> a lot
>     > >> of
>     > >> the associated JIRAs below are assigned, but there hasn’t been any
>     > >> recent
>     > >> activity or pull requests, we should probably consider them
> unassigned
>     > >> and
>     > >> up for grabs.:
>     > >>
>     > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has
> been the
>     > >> bane of a lot of Storm users almost since day one. We have largely
>     > >> addressed it by shading/relocating dependencies. It would be
> great to
>     > >> see
>     > >> this addressed once and for all.
>     > >>
>     > >> * JStorm back pressure implementation (STORM-1324) — The current
> back
>     > >> pressure implementation leaves a bit to be desired, and the JStorm
>     > >> approach
>     > >> looks promising, though it also depends on the JStorm concept of
>     > >> “topology
>     > >> master” (STORM-1323), which may have some implications regarding
>     > >> security.
>     > >>
>     > >> * Dynamic Topology Updates (STORM-1335) — This would provide a
> command
>     > >> to
>     > >> update topology jars and configuration without stopping the
> topology,
>     > >> and
>     > >> is well suited to leverage the blobstore. The restart command
> (that can
>     > >> also update the topology configuration) also looks compelling
>     > >> (STORM-1334).
>     > >>
>     > >> * Additional Scheduler Implementations (STORM-1320)
>     > >>
>     > >> * Additional Grouping Implementations (STORM-1328)
>     > >>
>     > >>
>     > >> As always I’m open to any opinions and suggestions.
>     > >>
>     > >> -Taylor
>     > >>
>     > >> [1]
>     > >>
>     > >> https://cwiki.apache.org/confluence/pages/viewpage.
>     > >> action?pageId=61328109
>     > >>
>     > >>
>     > >>
>     > >>
>     > >>
>     > >>
>     > >>
>     > >>
>     > >>
>     > >>
>     >
>
>
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Roshan Naik <ro...@hortonworks.com>.
For 2.0 beta … it would be good to incorporate some of the Worker improvements (STORM-2284)  IMO. Changes to messaging subsystem can be delivered sooner and my in-progress implementation suggests that it will yield substantial latency improvements. The 2.0 beta phase would really help kick the tires on the revised messaging system and the performance improvements will also be a good incentive for trying out the 2.0 line. 

I notice multiple other bottlenecks that are holding back throughput a lot, which can be addressed in a subsequent 2.x minor release. 
-roshan


On 6/3/17, 7:20 AM, "Jungtaek Lim" <ka...@gmail.com> wrote:

    I also would love to see metrics V2 code sooner or later too. If we can get
    it before releasing 2.0.0 that will be great, and then maybe we could just
    move toward to 2.0.0, not adding any improvements to 1.x version line.
    (And that's what I would want to.)
    
    If we would really want to have 1.2.0, I suggest that we make the 1.1.1
    version correct right now rather than after releasing 1.1.1. We also merged
    non-bugfix things to 1.x-branch but that's not what users expect. I agree
    that work may be painful, but anyway need to do it.
    
    - Jungtaek Lim (HeartSaVioR)
    
    2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:
    
    > I would love to see the metrics V2 code come out sooner rather than
    > later.  +1.
    > My biggest blocker for a 1.x release is
    > https://github.com/apache/storm/pull/2142  Even though the pull request
    > says it is minor it showed that we messed up pushing back some changes for
    > pacemaker to open source (the code does not run at all which for me is a
    > blocker) and I really want to get that fully fixed/tested before another
    > release.
    > As for 2.x I think we are very close to being able to so a 2.x alpha
    > release.  I would like to see metrics V2 merged in simply because it is a
    > big change for user facing APIs.  But after that I would love to see us
    > starting to push forward on getting that out.
    >
    >
    > - Bobby
    >
    >
    > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
    > ptgoetz@gmail.com> wrote:
    >
    > I’d like to bump this thread and start a discussion around our next
    > release. Here are my thoughts.
    >
    > There are a number of important fixes in 1.x-branch so I’d like to
    > consider releasing 1.1.1 soon. I’d appreciate input on any open issues that
    > should be resolved for that release.
    >
    > I’d like us to consider releasing the metrics improvements in STORM-2153
    > [1] as version 1.2.0. That work is in the metrics_v2 feature branch right
    > now and would need to be reviewed and merged. That work is against the
    > 1.x-branch right now. I would recommend porting it to master *after* the
    > review/merge since there will likely be changes as a result of the review.
    >
    > > Maybe related to or not, but would we want to create a new branch
    > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
    >
    >
    >
    > If wee agree to the above, I would say yes. After the 1.1.1 release, we
    > could create a 1.1.x-branch that would be the maintenance/release branch
    > for that version line. 1.x-branch would then become the target for the
    > 1.2.0 release.
    >
    > There are a few fixes in the 0.10.x branch that probably warrant a
    > release. After that we may want to back away from that version line a bit
    > so we can focus more on newer versions.
    >
    > In the past, we’ve shied away form doing “beta” releases, but I’m
    > wondering if we might want to revisit that for the 2.0 release — the idea
    > being that it would give early adopter users a chance to kick the tires on
    > what’s coming in 2.0 and provide feedback, find bugs, etc. to help make the
    > final release more solid. I’m on the fence here and could go either way.
    >
    > I’d appreciate any input others may have.
    >
    >
    > Thanks,
    >
    > -Taylor
    >
    >
    > [1] https://issues.apache.org/jira/browse/STORM-2153
    >
    >
    >
    > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com> wrote:
    > >
    > > Maybe related to or not, but would we want to create a new branch
    > > "1.1.x-branch", and make "1.x-branch" target for 1.2?
    > >
    > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so hence
    > the
    > > question.
    > >
    > > - Jungtaek Lim (HeartSaVioR)
    > >
    > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <hl...@hortonworks.com>님이
    > 작성:
    > >
    > >> +1 for finishing the porting to Java ahead of anything else - it will
    > be a
    > >> significant milestone. I have a JIRA assigned concerning to the
    > porting. I
    > >> will work on it for the 2.0 release.
    > >>
    > >> it’s a priority to guarantee no performance regressions. As part of this
    > >> endeavor, explore an automated (or easy) way to run and assert major
    > >> performance benchmarks. Ideally any contributor should be able to fairly
    > >> easily test the impact of changes under certain performance test
    > scenarios.
    > >>
    > >> Beam Runner work should take into account the impact of incorporating
    > new
    > >> JStorm features and Storm Worker Redesign<
    > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very efficient
    > to
    > >> start doing it, to  find out that it will have to chance in face of
    > Storm
    > >> and worker redesign. That is, it should be done after it’s building
    > blocks
    > >> are stable.
    > >>
    > >> Thanks,
    > >> Hugo
    > >>
    > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org<mailto:
    > >> arunm@apache.org>> wrote:
    > >>
    > >> +1 to release with the porting completed. I think its mainly the UI
    > server
    > >> and log viewer that’s pending.
    > >>
    > >> We can start doing the regression and performance tests for whatever is
    > >> already ported.
    > >>
    > >> If anyone is running the master branch in their pre-prod / prod
    > >> environments, it will be good to know and give us more confidence.
    > >>
    > >> The other features can be added in follow up releases.
    > >>
    > >> Regards,
    > >> Arun
    > >>
    > >>
    > >> On 3/24/17, 11:47 AM, "Satish Duggana" <satish.duggana@gmail.com
    > <mailto:
    > >> satish.duggana@gmail.com>> wrote:
    > >>
    > >> +1 to have 2.0 with porting and performance(it should be at least as
    > good
    > >> as 1.x release) issues addressed
    > >>
    > >> We can target other tasks(mentioned by Taylor and Jungtaek) for
    > 2.x-branch.
    > >>
    > >>
    > >> Exactly-once support:
    > >> While thinking through the exactlyonce support design, it is realized
    > >> better to avoid acking tuples and implement exactly once by snapshotting
    > >> barriers. It seems JStorm folks followed similar design, they claim it
    > >> gives better performance. This feature is essential for beam runner and
    > we
    > >> can decide on respective approaches though.
    > >>
    > >> Beam Runner
    > >> Lets hold on this for now and keep it in Storm till 2.x. We should avoid
    > >> having a minimal beam runner in haste. It is better to address
    > STORM-2284,
    > >> exactly-once and other windowing enhancements to enable beam runner.
    > >>
    > >> JStorm
    > >> Agree with Jungtaek on looking at the latest JStorm and align/scope with
    > >> the features for 2.x.
    > >>
    > >> STORM-2284
    > >> We may want to look at JStorm worker before working on respective
    > >> components in this epic to pull appropriate enhancements.
    > >>
    > >> YARN/MESOS
    > >> Supporting Storm on YARN/Mesos for 2.x.
    > >>
    > >> Thanks,
    > >> Satish.
    > >>
    > >>
    > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <kabhwan@gmail.com
    > <mailto:
    > >> kabhwan@gmail.com>> wrote:
    > >>
    > >> First of all, +1 to complete only port work and do sanity check
    > (including
    > >> performance regression), and release.
    > >>
    > >> If we can get STORM-2284 within deterministic time frame (say 2~3
    > months)
    > >> that should be great, but if not I'd in favor of postponing that to
    > later
    > >> 2.x release.
    > >>
    > >> JStorm released their new versions after code donation. So there're more
    > >> things we could get ideas from, or even adopt from.
    > >> https://github.com/alibaba/jstorm/blob/master/history.md
    > >> As you noticed from release note link, we also need to update phase 2
    > since
    > >> they already changed what we're planning to do in phase 2. For example,
    > >> they changed backpressure to end-to-end, and changed to use snapshot
    > rather
    > >> than acker.
    > >> May be sure, JStorm pulled many features from today's Storm, like Flux,
    > >> Windowing, more shuffle groupings, log search, log level change, and so
    > on.
    > >>
    > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> is due to
    > >> the
    > >> limitation of Spout lifecycle (all the things are done in single
    > thread),
    > >> and STORM-1358 <https://issues.apache.org/jira/browse/STORM-1358
    > >(JStorm's
    > >> multi-thread Spout) can remedy this (despite that Spout implementation
    > may
    > >> need to guarantee thread-safety later). It's not a just improvement but
    > >> close to design concern so would like to address sooner than other
    > things
    > >> in phase 2.
    > >>
    > >> For Storm SQL side, I've lost progress but major work would be adopting
    > >> group by with windowing. It was not available from Calcite but will be
    > >> available at next release (1.12.0).
    > >> I've filed this to STORM-2405
    > >> <https://issues.apache.org/jira/browse/STORM-2405>, but windowing &
    > micro
    > >> batch is not intuitive, so I would like to change the underlying API to
    > >> stream API in SQL. Also filed this to STORM-2406
    > >> <https://issues.apache.org/jira/browse/STORM-2406>.
    > >>
    > >> Just 2 cents btw, hopefully I would like to see metrics V2 sooner since
    > we
    > >> lost metrics even when doing normal operation like restarting worker,
    > >> rebalancing, and so on. Eventually we need to fight with dynamic
    > scaling,
    > >> and then metrics will be broken often.
    > >>
    > >> Thanks,
    > >> Jungtaek Lim (HeartSaVioR)
    > >>
    > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <st...@harsha.io>님이 작성:
    > >>
    > >> Storm 2.0 migration to java in itself is a big win and would attract
    > >> wider
    > >> community and adoption. So my vote would be to resolve the first 3 items
    > >> to
    > >> get a release out.
    > >> All the other featured mentioned are great to have but shouldn't be
    > >> blockers for 2.0 release.
    > >>
    > >> -Harsha
    > >>
    > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <pt...@gmail.com>
    > >> wrote:
    > >>
    > >> With the 1.1.0 release nearing completion, I’d like to turn our
    > >> attention
    > >> to 2.0 and develop a plan for what features, etc. to include.
    > >>
    > >> The following 3 are what I feel are the minimum for a 2.0 release.
    > >> These
    > >> could likely be resolved relatively quickly:
    > >>
    > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x or
    > >> 1.1.x
    > >> in a while, but I feel it will be important to make sure there are no
    > >> performance regressions, and would hope that we actually have a
    > >> performance
    > >> improvement over previous versions. To that end (e.g. if there is in
    > >> fact a
    > >> performance regression), the proposals that Roshan Naik put together
    > >> for
    > >> revising the threading and execution model (STORM-2307) and replacing
    > >> Disruptor with JCTools (STORM-2306) warrant review and consideration.
    > >> See
    > >> also STORM-2284 which is the parent JIRA.
    > >>
    > >> * Finish porting Storm UI to java (STORM-1311)
    > >>
    > >> * Finish porting log viewer to java (STORM-1280)
    > >>
    > >> The following are items that are nice to have in 2.0, but I don’t feel
    > >> are
    > >> absolutely necessary for an initial 2.0 release:
    > >>
    > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because it’s
    > >> relevant) — Initially there seemed to be a lot of interest in this, but
    > >> that seems to have trailed off. I spoke with some Beam developers and
    > >> there
    > >> seems to be interest from that community as well. Do we want to move
    > >> that
    > >> effort to the Beam community, or keep it here? Moving it to the Beam
    > >> community might lead to better collaboration between projects.
    > >>
    > >> * Bounded Spouts (needed for Beam Runner implementation) — Currently
    > >> spouts are unbounded, there no end to the stream. Beam has the concept
    > >> of
    > >> bounded sources (roughly analogous to batch processing). To support
    > >> that,
    > >> we would need to implement a similar concept in Storm. One benefit of
    > >> such
    > >> a feature would be the ability to handle both bounded and unbounded
    > >> workflows in Storm.
    > >>
    > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers behind
    > >> this
    > >> effort. What improvements do you envision for 2.0?
    > >>
    > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been targeting this
    > >> for 1.2.0, but it’s designed to be easily portable to master/2.0.
    > >>
    > >> * JStorm Migration — Original outline can be found here [1]. Note a lot
    > >> of
    > >> the associated JIRAs below are assigned, but there hasn’t been any
    > >> recent
    > >> activity or pull requests, we should probably consider them unassigned
    > >> and
    > >> up for grabs.:
    > >>
    > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has been the
    > >> bane of a lot of Storm users almost since day one. We have largely
    > >> addressed it by shading/relocating dependencies. It would be great to
    > >> see
    > >> this addressed once and for all.
    > >>
    > >> * JStorm back pressure implementation (STORM-1324) — The current back
    > >> pressure implementation leaves a bit to be desired, and the JStorm
    > >> approach
    > >> looks promising, though it also depends on the JStorm concept of
    > >> “topology
    > >> master” (STORM-1323), which may have some implications regarding
    > >> security.
    > >>
    > >> * Dynamic Topology Updates (STORM-1335) — This would provide a command
    > >> to
    > >> update topology jars and configuration without stopping the topology,
    > >> and
    > >> is well suited to leverage the blobstore. The restart command (that can
    > >> also update the topology configuration) also looks compelling
    > >> (STORM-1334).
    > >>
    > >> * Additional Scheduler Implementations (STORM-1320)
    > >>
    > >> * Additional Grouping Implementations (STORM-1328)
    > >>
    > >>
    > >> As always I’m open to any opinions and suggestions.
    > >>
    > >> -Taylor
    > >>
    > >> [1]
    > >>
    > >> https://cwiki.apache.org/confluence/pages/viewpage.
    > >> action?pageId=61328109
    > >>
    > >>
    > >>
    > >>
    > >>
    > >>
    > >>
    > >>
    > >>
    > >>
    >
    


Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Jungtaek Lim <ka...@gmail.com>.
I also would love to see metrics V2 code sooner or later too. If we can get
it before releasing 2.0.0 that will be great, and then maybe we could just
move toward to 2.0.0, not adding any improvements to 1.x version line.
(And that's what I would want to.)

If we would really want to have 1.2.0, I suggest that we make the 1.1.1
version correct right now rather than after releasing 1.1.1. We also merged
non-bugfix things to 1.x-branch but that's not what users expect. I agree
that work may be painful, but anyway need to do it.

- Jungtaek Lim (HeartSaVioR)

2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:

> I would love to see the metrics V2 code come out sooner rather than
> later.  +1.
> My biggest blocker for a 1.x release is
> https://github.com/apache/storm/pull/2142  Even though the pull request
> says it is minor it showed that we messed up pushing back some changes for
> pacemaker to open source (the code does not run at all which for me is a
> blocker) and I really want to get that fully fixed/tested before another
> release.
> As for 2.x I think we are very close to being able to so a 2.x alpha
> release.  I would like to see metrics V2 merged in simply because it is a
> big change for user facing APIs.  But after that I would love to see us
> starting to push forward on getting that out.
>
>
> - Bobby
>
>
> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <
> ptgoetz@gmail.com> wrote:
>
> I’d like to bump this thread and start a discussion around our next
> release. Here are my thoughts.
>
> There are a number of important fixes in 1.x-branch so I’d like to
> consider releasing 1.1.1 soon. I’d appreciate input on any open issues that
> should be resolved for that release.
>
> I’d like us to consider releasing the metrics improvements in STORM-2153
> [1] as version 1.2.0. That work is in the metrics_v2 feature branch right
> now and would need to be reviewed and merged. That work is against the
> 1.x-branch right now. I would recommend porting it to master *after* the
> review/merge since there will likely be changes as a result of the review.
>
> > Maybe related to or not, but would we want to create a new branch
> > "1.1.x-branch", and make "1.x-branch" target for 1.2?
>
>
>
> If wee agree to the above, I would say yes. After the 1.1.1 release, we
> could create a 1.1.x-branch that would be the maintenance/release branch
> for that version line. 1.x-branch would then become the target for the
> 1.2.0 release.
>
> There are a few fixes in the 0.10.x branch that probably warrant a
> release. After that we may want to back away from that version line a bit
> so we can focus more on newer versions.
>
> In the past, we’ve shied away form doing “beta” releases, but I’m
> wondering if we might want to revisit that for the 2.0 release — the idea
> being that it would give early adopter users a chance to kick the tires on
> what’s coming in 2.0 and provide feedback, find bugs, etc. to help make the
> final release more solid. I’m on the fence here and could go either way.
>
> I’d appreciate any input others may have.
>
>
> Thanks,
>
> -Taylor
>
>
> [1] https://issues.apache.org/jira/browse/STORM-2153
>
>
>
> > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> >
> > Maybe related to or not, but would we want to create a new branch
> > "1.1.x-branch", and make "1.x-branch" target for 1.2?
> >
> > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so hence
> the
> > question.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <hl...@hortonworks.com>님이
> 작성:
> >
> >> +1 for finishing the porting to Java ahead of anything else - it will
> be a
> >> significant milestone. I have a JIRA assigned concerning to the
> porting. I
> >> will work on it for the 2.0 release.
> >>
> >> it’s a priority to guarantee no performance regressions. As part of this
> >> endeavor, explore an automated (or easy) way to run and assert major
> >> performance benchmarks. Ideally any contributor should be able to fairly
> >> easily test the impact of changes under certain performance test
> scenarios.
> >>
> >> Beam Runner work should take into account the impact of incorporating
> new
> >> JStorm features and Storm Worker Redesign<
> >> https://issues.apache.org/jira/browse/STORM-2284>. Not very efficient
> to
> >> start doing it, to  find out that it will have to chance in face of
> Storm
> >> and worker redesign. That is, it should be done after it’s building
> blocks
> >> are stable.
> >>
> >> Thanks,
> >> Hugo
> >>
> >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org<mailto:
> >> arunm@apache.org>> wrote:
> >>
> >> +1 to release with the porting completed. I think its mainly the UI
> server
> >> and log viewer that’s pending.
> >>
> >> We can start doing the regression and performance tests for whatever is
> >> already ported.
> >>
> >> If anyone is running the master branch in their pre-prod / prod
> >> environments, it will be good to know and give us more confidence.
> >>
> >> The other features can be added in follow up releases.
> >>
> >> Regards,
> >> Arun
> >>
> >>
> >> On 3/24/17, 11:47 AM, "Satish Duggana" <satish.duggana@gmail.com
> <mailto:
> >> satish.duggana@gmail.com>> wrote:
> >>
> >> +1 to have 2.0 with porting and performance(it should be at least as
> good
> >> as 1.x release) issues addressed
> >>
> >> We can target other tasks(mentioned by Taylor and Jungtaek) for
> 2.x-branch.
> >>
> >>
> >> Exactly-once support:
> >> While thinking through the exactlyonce support design, it is realized
> >> better to avoid acking tuples and implement exactly once by snapshotting
> >> barriers. It seems JStorm folks followed similar design, they claim it
> >> gives better performance. This feature is essential for beam runner and
> we
> >> can decide on respective approaches though.
> >>
> >> Beam Runner
> >> Lets hold on this for now and keep it in Storm till 2.x. We should avoid
> >> having a minimal beam runner in haste. It is better to address
> STORM-2284,
> >> exactly-once and other windowing enhancements to enable beam runner.
> >>
> >> JStorm
> >> Agree with Jungtaek on looking at the latest JStorm and align/scope with
> >> the features for 2.x.
> >>
> >> STORM-2284
> >> We may want to look at JStorm worker before working on respective
> >> components in this epic to pull appropriate enhancements.
> >>
> >> YARN/MESOS
> >> Supporting Storm on YARN/Mesos for 2.x.
> >>
> >> Thanks,
> >> Satish.
> >>
> >>
> >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <kabhwan@gmail.com
> <mailto:
> >> kabhwan@gmail.com>> wrote:
> >>
> >> First of all, +1 to complete only port work and do sanity check
> (including
> >> performance regression), and release.
> >>
> >> If we can get STORM-2284 within deterministic time frame (say 2~3
> months)
> >> that should be great, but if not I'd in favor of postponing that to
> later
> >> 2.x release.
> >>
> >> JStorm released their new versions after code donation. So there're more
> >> things we could get ideas from, or even adopt from.
> >> https://github.com/alibaba/jstorm/blob/master/history.md
> >> As you noticed from release note link, we also need to update phase 2
> since
> >> they already changed what we're planning to do in phase 2. For example,
> >> they changed backpressure to end-to-end, and changed to use snapshot
> rather
> >> than acker.
> >> May be sure, JStorm pulled many features from today's Storm, like Flux,
> >> Windowing, more shuffle groupings, log search, log level change, and so
> on.
> >>
> >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> is due to
> >> the
> >> limitation of Spout lifecycle (all the things are done in single
> thread),
> >> and STORM-1358 <https://issues.apache.org/jira/browse/STORM-1358
> >(JStorm's
> >> multi-thread Spout) can remedy this (despite that Spout implementation
> may
> >> need to guarantee thread-safety later). It's not a just improvement but
> >> close to design concern so would like to address sooner than other
> things
> >> in phase 2.
> >>
> >> For Storm SQL side, I've lost progress but major work would be adopting
> >> group by with windowing. It was not available from Calcite but will be
> >> available at next release (1.12.0).
> >> I've filed this to STORM-2405
> >> <https://issues.apache.org/jira/browse/STORM-2405>, but windowing &
> micro
> >> batch is not intuitive, so I would like to change the underlying API to
> >> stream API in SQL. Also filed this to STORM-2406
> >> <https://issues.apache.org/jira/browse/STORM-2406>.
> >>
> >> Just 2 cents btw, hopefully I would like to see metrics V2 sooner since
> we
> >> lost metrics even when doing normal operation like restarting worker,
> >> rebalancing, and so on. Eventually we need to fight with dynamic
> scaling,
> >> and then metrics will be broken often.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <st...@harsha.io>님이 작성:
> >>
> >> Storm 2.0 migration to java in itself is a big win and would attract
> >> wider
> >> community and adoption. So my vote would be to resolve the first 3 items
> >> to
> >> get a release out.
> >> All the other featured mentioned are great to have but shouldn't be
> >> blockers for 2.0 release.
> >>
> >> -Harsha
> >>
> >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <pt...@gmail.com>
> >> wrote:
> >>
> >> With the 1.1.0 release nearing completion, I’d like to turn our
> >> attention
> >> to 2.0 and develop a plan for what features, etc. to include.
> >>
> >> The following 3 are what I feel are the minimum for a 2.0 release.
> >> These
> >> could likely be resolved relatively quickly:
> >>
> >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x or
> >> 1.1.x
> >> in a while, but I feel it will be important to make sure there are no
> >> performance regressions, and would hope that we actually have a
> >> performance
> >> improvement over previous versions. To that end (e.g. if there is in
> >> fact a
> >> performance regression), the proposals that Roshan Naik put together
> >> for
> >> revising the threading and execution model (STORM-2307) and replacing
> >> Disruptor with JCTools (STORM-2306) warrant review and consideration.
> >> See
> >> also STORM-2284 which is the parent JIRA.
> >>
> >> * Finish porting Storm UI to java (STORM-1311)
> >>
> >> * Finish porting log viewer to java (STORM-1280)
> >>
> >> The following are items that are nice to have in 2.0, but I don’t feel
> >> are
> >> absolutely necessary for an initial 2.0 release:
> >>
> >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because it’s
> >> relevant) — Initially there seemed to be a lot of interest in this, but
> >> that seems to have trailed off. I spoke with some Beam developers and
> >> there
> >> seems to be interest from that community as well. Do we want to move
> >> that
> >> effort to the Beam community, or keep it here? Moving it to the Beam
> >> community might lead to better collaboration between projects.
> >>
> >> * Bounded Spouts (needed for Beam Runner implementation) — Currently
> >> spouts are unbounded, there no end to the stream. Beam has the concept
> >> of
> >> bounded sources (roughly analogous to batch processing). To support
> >> that,
> >> we would need to implement a similar concept in Storm. One benefit of
> >> such
> >> a feature would be the ability to handle both bounded and unbounded
> >> workflows in Storm.
> >>
> >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers behind
> >> this
> >> effort. What improvements do you envision for 2.0?
> >>
> >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been targeting this
> >> for 1.2.0, but it’s designed to be easily portable to master/2.0.
> >>
> >> * JStorm Migration — Original outline can be found here [1]. Note a lot
> >> of
> >> the associated JIRAs below are assigned, but there hasn’t been any
> >> recent
> >> activity or pull requests, we should probably consider them unassigned
> >> and
> >> up for grabs.:
> >>
> >> * Worker Classloader Isolation (STORM-1338) — Lack of this has been the
> >> bane of a lot of Storm users almost since day one. We have largely
> >> addressed it by shading/relocating dependencies. It would be great to
> >> see
> >> this addressed once and for all.
> >>
> >> * JStorm back pressure implementation (STORM-1324) — The current back
> >> pressure implementation leaves a bit to be desired, and the JStorm
> >> approach
> >> looks promising, though it also depends on the JStorm concept of
> >> “topology
> >> master” (STORM-1323), which may have some implications regarding
> >> security.
> >>
> >> * Dynamic Topology Updates (STORM-1335) — This would provide a command
> >> to
> >> update topology jars and configuration without stopping the topology,
> >> and
> >> is well suited to leverage the blobstore. The restart command (that can
> >> also update the topology configuration) also looks compelling
> >> (STORM-1334).
> >>
> >> * Additional Scheduler Implementations (STORM-1320)
> >>
> >> * Additional Grouping Implementations (STORM-1328)
> >>
> >>
> >> As always I’m open to any opinions and suggestions.
> >>
> >> -Taylor
> >>
> >> [1]
> >>
> >> https://cwiki.apache.org/confluence/pages/viewpage.
> >> action?pageId=61328109
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>

Re: [DISCUSS] Storm 2.0 Roadmap

Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
I would love to see the metrics V2 code come out sooner rather than later.  +1.
My biggest blocker for a 1.x release is https://github.com/apache/storm/pull/2142  Even though the pull request says it is minor it showed that we messed up pushing back some changes for pacemaker to open source (the code does not run at all which for me is a blocker) and I really want to get that fully fixed/tested before another release.
As for 2.x I think we are very close to being able to so a 2.x alpha release.  I would like to see metrics V2 merged in simply because it is a big change for user facing APIs.  But after that I would love to see us starting to push forward on getting that out.


- Bobby


On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz <pt...@gmail.com> wrote:

I’d like to bump this thread and start a discussion around our next release. Here are my thoughts.

There are a number of important fixes in 1.x-branch so I’d like to consider releasing 1.1.1 soon. I’d appreciate input on any open issues that should be resolved for that release.

I’d like us to consider releasing the metrics improvements in STORM-2153 [1] as version 1.2.0. That work is in the metrics_v2 feature branch right now and would need to be reviewed and merged. That work is against the 1.x-branch right now. I would recommend porting it to master *after* the review/merge since there will likely be changes as a result of the review.

> Maybe related to or not, but would we want to create a new branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?



If wee agree to the above, I would say yes. After the 1.1.1 release, we could create a 1.1.x-branch that would be the maintenance/release branch for that version line. 1.x-branch would then become the target for the 1.2.0 release.

There are a few fixes in the 0.10.x branch that probably warrant a release. After that we may want to back away from that version line a bit so we can focus more on newer versions.

In the past, we’ve shied away form doing “beta” releases, but I’m wondering if we might want to revisit that for the 2.0 release — the idea being that it would give early adopter users a chance to kick the tires on what’s coming in 2.0 and provide feedback, find bugs, etc. to help make the final release more solid. I’m on the fence here and could go either way.

I’d appreciate any input others may have.


Thanks,

-Taylor


[1] https://issues.apache.org/jira/browse/STORM-2153



> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> 
> Maybe related to or not, but would we want to create a new branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> 
> I'm not clear we don't release 1.2 for moving toward to 2.0.0, so hence the
> question.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro <hl...@hortonworks.com>님이 작성:
> 
>> +1 for finishing the porting to Java ahead of anything else - it will be a
>> significant milestone. I have a JIRA assigned concerning to the porting. I
>> will work on it for the 2.0 release.
>> 
>> it’s a priority to guarantee no performance regressions. As part of this
>> endeavor, explore an automated (or easy) way to run and assert major
>> performance benchmarks. Ideally any contributor should be able to fairly
>> easily test the impact of changes under certain performance test scenarios.
>> 
>> Beam Runner work should take into account the impact of incorporating new
>> JStorm features and Storm Worker Redesign<
>> https://issues.apache.org/jira/browse/STORM-2284>. Not very efficient to
>> start doing it, to  find out that it will have to chance in face of Storm
>> and worker redesign. That is, it should be done after it’s building blocks
>> are stable.
>> 
>> Thanks,
>> Hugo
>> 
>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <arunm@apache.org<mailto:
>> arunm@apache.org>> wrote:
>> 
>> +1 to release with the porting completed. I think its mainly the UI server
>> and log viewer that’s pending.
>> 
>> We can start doing the regression and performance tests for whatever is
>> already ported.
>> 
>> If anyone is running the master branch in their pre-prod / prod
>> environments, it will be good to know and give us more confidence.
>> 
>> The other features can be added in follow up releases.
>> 
>> Regards,
>> Arun
>> 
>> 
>> On 3/24/17, 11:47 AM, "Satish Duggana" <satish.duggana@gmail.com<mailto:
>> satish.duggana@gmail.com>> wrote:
>> 
>> +1 to have 2.0 with porting and performance(it should be at least as good
>> as 1.x release) issues addressed
>> 
>> We can target other tasks(mentioned by Taylor and Jungtaek) for 2.x-branch.
>> 
>> 
>> Exactly-once support:
>> While thinking through the exactlyonce support design, it is realized
>> better to avoid acking tuples and implement exactly once by snapshotting
>> barriers. It seems JStorm folks followed similar design, they claim it
>> gives better performance. This feature is essential for beam runner and we
>> can decide on respective approaches though.
>> 
>> Beam Runner
>> Lets hold on this for now and keep it in Storm till 2.x. We should avoid
>> having a minimal beam runner in haste. It is better to address STORM-2284,
>> exactly-once and other windowing enhancements to enable beam runner.
>> 
>> JStorm
>> Agree with Jungtaek on looking at the latest JStorm and align/scope with
>> the features for 2.x.
>> 
>> STORM-2284
>> We may want to look at JStorm worker before working on respective
>> components in this epic to pull appropriate enhancements.
>> 
>> YARN/MESOS
>> Supporting Storm on YARN/Mesos for 2.x.
>> 
>> Thanks,
>> Satish.
>> 
>> 
>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <kabhwan@gmail.com<mailto:
>> kabhwan@gmail.com>> wrote:
>> 
>> First of all, +1 to complete only port work and do sanity check (including
>> performance regression), and release.
>> 
>> If we can get STORM-2284 within deterministic time frame (say 2~3 months)
>> that should be great, but if not I'd in favor of postponing that to later
>> 2.x release.
>> 
>> JStorm released their new versions after code donation. So there're more
>> things we could get ideas from, or even adopt from.
>> https://github.com/alibaba/jstorm/blob/master/history.md
>> As you noticed from release note link, we also need to update phase 2 since
>> they already changed what we're planning to do in phase 2. For example,
>> they changed backpressure to end-to-end, and changed to use snapshot rather
>> than acker.
>> May be sure, JStorm pulled many features from today's Storm, like Flux,
>> Windowing, more shuffle groupings, log search, log level change, and so on.
>> 
>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> is due to
>> the
>> limitation of Spout lifecycle (all the things are done in single thread),
>> and STORM-1358 <https://issues.apache.org/jira/browse/STORM-1358>(JStorm's
>> multi-thread Spout) can remedy this (despite that Spout implementation may
>> need to guarantee thread-safety later). It's not a just improvement but
>> close to design concern so would like to address sooner than other things
>> in phase 2.
>> 
>> For Storm SQL side, I've lost progress but major work would be adopting
>> group by with windowing. It was not available from Calcite but will be
>> available at next release (1.12.0).
>> I've filed this to STORM-2405
>> <https://issues.apache.org/jira/browse/STORM-2405>, but windowing & micro
>> batch is not intuitive, so I would like to change the underlying API to
>> stream API in SQL. Also filed this to STORM-2406
>> <https://issues.apache.org/jira/browse/STORM-2406>.
>> 
>> Just 2 cents btw, hopefully I would like to see metrics V2 sooner since we
>> lost metrics even when doing normal operation like restarting worker,
>> rebalancing, and so on. Eventually we need to fight with dynamic scaling,
>> and then metrics will be broken often.
>> 
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>> 
>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <st...@harsha.io>님이 작성:
>> 
>> Storm 2.0 migration to java in itself is a big win and would attract
>> wider
>> community and adoption. So my vote would be to resolve the first 3 items
>> to
>> get a release out.
>> All the other featured mentioned are great to have but shouldn't be
>> blockers for 2.0 release.
>> 
>> -Harsha
>> 
>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz <pt...@gmail.com>
>> wrote:
>> 
>> With the 1.1.0 release nearing completion, I’d like to turn our
>> attention
>> to 2.0 and develop a plan for what features, etc. to include.
>> 
>> The following 3 are what I feel are the minimum for a 2.0 release.
>> These
>> could likely be resolved relatively quickly:
>> 
>> * Performance — I’ve not benchmarked the master branch vs. 1.0.x or
>> 1.1.x
>> in a while, but I feel it will be important to make sure there are no
>> performance regressions, and would hope that we actually have a
>> performance
>> improvement over previous versions. To that end (e.g. if there is in
>> fact a
>> performance regression), the proposals that Roshan Naik put together
>> for
>> revising the threading and execution model (STORM-2307) and replacing
>> Disruptor with JCTools (STORM-2306) warrant review and consideration.
>> See
>> also STORM-2284 which is the parent JIRA.
>> 
>> * Finish porting Storm UI to java (STORM-1311)
>> 
>> * Finish porting log viewer to java (STORM-1280)
>> 
>> The following are items that are nice to have in 2.0, but I don’t feel
>> are
>> absolutely necessary for an initial 2.0 release:
>> 
>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because it’s
>> relevant) — Initially there seemed to be a lot of interest in this, but
>> that seems to have trailed off. I spoke with some Beam developers and
>> there
>> seems to be interest from that community as well. Do we want to move
>> that
>> effort to the Beam community, or keep it here? Moving it to the Beam
>> community might lead to better collaboration between projects.
>> 
>> * Bounded Spouts (needed for Beam Runner implementation) — Currently
>> spouts are unbounded, there no end to the stream. Beam has the concept
>> of
>> bounded sources (roughly analogous to batch processing). To support
>> that,
>> we would need to implement a similar concept in Storm. One benefit of
>> such
>> a feature would be the ability to handle both bounded and unbounded
>> workflows in Storm.
>> 
>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers behind
>> this
>> effort. What improvements do you envision for 2.0?
>> 
>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been targeting this
>> for 1.2.0, but it’s designed to be easily portable to master/2.0.
>> 
>> * JStorm Migration — Original outline can be found here [1]. Note a lot
>> of
>> the associated JIRAs below are assigned, but there hasn’t been any
>> recent
>> activity or pull requests, we should probably consider them unassigned
>> and
>> up for grabs.:
>> 
>> * Worker Classloader Isolation (STORM-1338) — Lack of this has been the
>> bane of a lot of Storm users almost since day one. We have largely
>> addressed it by shading/relocating dependencies. It would be great to
>> see
>> this addressed once and for all.
>> 
>> * JStorm back pressure implementation (STORM-1324) — The current back
>> pressure implementation leaves a bit to be desired, and the JStorm
>> approach
>> looks promising, though it also depends on the JStorm concept of
>> “topology
>> master” (STORM-1323), which may have some implications regarding
>> security.
>> 
>> * Dynamic Topology Updates (STORM-1335) — This would provide a command
>> to
>> update topology jars and configuration without stopping the topology,
>> and
>> is well suited to leverage the blobstore. The restart command (that can
>> also update the topology configuration) also looks compelling
>> (STORM-1334).
>> 
>> * Additional Scheduler Implementations (STORM-1320)
>> 
>> * Additional Grouping Implementations (STORM-1328)
>> 
>> 
>> As always I’m open to any opinions and suggestions.
>> 
>> -Taylor
>> 
>> [1]
>> 
>> https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=61328109
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>