You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by John Fang <xi...@alibaba-inc.com> on 2015/11/24 17:29:56 UTC
答复: [DISCUSS] Plan for Merging JStorm Code
Hi Taylor,
I'd like to help too, could you add me in? my id is: hustfxj
-----邮件原件-----
发件人: P. Taylor Goetz [mailto:ptgoetz@gmail.com]
发送时间: 2015年11月24日 23:54
收件人: dev@storm.apache.org
主题: Re: [DISCUSS] Plan for Merging JStorm Code
Hi Cody,
I wasn’t able to find your username. Are you sure you have an account on cwiki.apache.org?
-Taylor
> On Nov 22, 2015, at 8:46 AM, Cody Innowhere <e....@gmail.com> wrote:
>
> Hi Taylor,
> I'd like to help too, could you add me in? my id is: Cody
>
> On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu)
> <ba...@alibaba-inc.com>
> wrote:
>
>> Hi Taylor，
>>
>> Sorry for the late response.
>> I'd like to help on this. Could you please help to give me the permission?
>> Thanks.
>> UserName: basti.lj
>>
>> Regards
>> Basti
>>
>> -----Original Message-----
>> From: P. Taylor Goetz [mailto:ptgoetz@gmail.com]
>> Sent: Thursday, November 19, 2015 6:24 AM
>> To: dev@storm.apache.org; Bobby Evans
>> Subject: Re: [DISCUSS] Plan for Merging JStorm Code
>>
>> All I have at this point is a placeholder wiki entry [1], and a lot
>> of local notes that likely would only make sense to me.
>>
>> Let me know your wiki username and I’ll give you permissions. The
>> same goes for anyone else who wants to help.
>>
>> -Taylor
>>
>> [1]
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6132
>> 8109
>>
>>> On Nov 18, 2015, at 2:08 PM, Bobby Evans
>>> <ev...@yahoo-inc.com.INVALID>
>> wrote:
>>>
>>> Taylor and others I was hoping to get started filing JIRA and
>>> planning on how we are going to do the java migration + JStorm
>>> merger. Is anyone else starting to do this? If not would anyone
>>> object to me starting on it? - Bobby
>>>
>>>
>>> On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz <
>> ptgoetz@gmail.com> wrote:
>>>
>>>
>>> Thanks for putting this together Basti, that comparison helps a lot.
>>>
>>> And thanks Bobby for converting it into markdown. I was going to
>>> just
>> attach the spreadsheet to JIRA, but markdown is a much better solution.
>>>
>>> -Taylor
>>>
>>>> On Nov 12, 2015, at 12:03 PM, Bobby Evans
>>>> <ev...@yahoo-inc.com.INVALID>
>> wrote:
>>>>
>>>> I translated the excel spreadsheet into a markdown file and put up
>>>> a
>> pull request for it.
>>>> https://github.com/apache/storm/pull/877
>>>> I did a few edits to it to make it work with Markdown, and to add
>>>> in a
>> few of my own comments. I also put in a field for JIRAs to be able
>> to track the migration.
>>>> Overall I think your evaluation was very good. We have a fair
>>>> amount
>> of work ahead of us to decide what version of various features we
>> want to go forward with.
>>>> - Bobby
>>>>
>>>>
>>>> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) <
>> basti.lj@alibaba-inc.com> wrote:
>>>>
>>>>
>>>> Hi Bobby & Jungtaek,
>>>>
>>>> Thanks for your replay.
>>>> I totally agree that compatibility is the most important thing.
>> Actually, JStorm has been compatible with the user API of Storm.
>>>> As you mentioned below, we indeed still have some features
>>>> different
>> between Storm and JStorm. I have tried to list them (minor update or
>> improvements are not included).
>>>> Please refer to attachment for details. If any missing, please help
>>>> to point out. (The current working features are probably missing
>>>> here.)
>> Just have a look at these differences. For the missing features in
>> JStorm, I did not see any obstacle which will block the merge to JStorm.
>>>> For the features which has different solution between Storm and
>>>> JStorm,
>> we can evaluate the solution one by one to decision which one is
>> appropriate.
>>>> After the finalization of evaluation, I think JStorm team can take
>>>> the
>> merging job and publish a stable release in 2 months.
>>>> But anyway, the detailed implementation for these features with
>> different solution is transparent to user. So, from user's point of
>> view, there is not any compatibility problem.
>>>>
>>>> Besides compatibility, by our experience, stability is also
>>>> important
>> and is not an easy job. 4 people in JStorm team took almost one year
>> to finish the porting from "clojure core"
>>>> to "java core", and to make it stable. Of course, we have many devs
>>>> in
>> community to make the porting job faster. But it still needs a long
>> time to run many online complex topologys to find bugs and fix them.
>> So, that is the reason why I proposed to do merging and build on a stable "java core".
>>>>
>>>> -----Original Message-----
>>>> From: Bobby Evans [mailto:evans@yahoo-inc.com.INVALID]
>>>> Sent: Wednesday, November 11, 2015 10:51 PM
>>>> To: dev@storm.apache.org
>>>> Subject: Re: [DISCUSS] Plan for Merging JStorm Code
>>>>
>>>> +1 for doing a 1.0 release based off of the clojure 0.11.x code.
>> Migrating the APIs to org.apache.storm is a big non-backwards
>> compatible move, and a major version bump to 2.x seems like a good move there.
>>>> +1 for the release plan
>>>>
>>>> I would like the move for user facing APIs to org.apache to be one
>>>> of
>> the last things we do. Translating clojure code into java and moving
>> it to org.apache I am not too concerned about.
>>>>
>>>> Basti,
>>>> We have two code bases that have diverged significantly from one
>> another in terms of functionality. The storm code now or soon will
>> have A Heartbeat Server, Nimbus HA (Different Implementation),
>> Resource Aware Scheduling, a distributed cache like API, log
>> searching, security, massive performance improvements, shaded almost
>> all of our dependencies, a REST API for programtically accessing
>> everything on the UI, and I am sure I am missing a few other things.
>> JStorm also has many changes including cgroup isolation, restructured zookeeper layout, classpath isolation, and more too.
>>>> No matter what we do it will be a large effort to port changes from
>>>> one code base to another, and from clojure to java. I proposed
>>>> this initially because it can be broken up into incremental
>>>> changes. It may take a little longer, but we will always have a
>>>> working codebase that is testable and compatible with the current
>>>> storm release, at least until we move the user facing APIs to be under org.apache.
>>>> This lets the community continue to build and test the master
>>>> branch and report problems that they find, which is incredibly
>>>> valuable. I personally don't think it will be much easier,
>>>> especially if we are intent on always maintaining compatibility
>>>> with storm. - Bobby
>>>>
>>>>
>>>> On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) <
>> basti.lj@alibaba-inc.com> wrote:
>>>>
>>>>
>>>> Hi Taylor,
>>>>
>>>>
>>>>
>>>> Thanks for the merge plan. I have a question about “Phase 2.2”.
>>>>
>>>> Do you mean community plan to create a fresh new “java core” based
>>>> on
>> current “clojure core” firstly, and then migrate the features from JStorm?
>>>>
>>>> If so, it confused me. It is really a huge job which might require
>>>> a
>> long developing time to make it stable, while JStorm is already a
>> stable version.
>>>>
>>>> The release planned to be release after Nov 11th has already run
>>>> online
>> stably several month in Alibaba.
>>>>
>>>> Besides this, there are many valuable internal requirements in
>>>> Alibaba,
>> the fast evolution of JStorm is forseeable in next few months.
>>>>
>>>> If the “java core” is totally fresh new, it might bring many
>>>> problems
>> for the coming merge.
>>>>
>>>> So, from the point of this view, I think it is much better and
>>>> easier
>> to migrate the features of “clojure core” basing on JStorm for the
>> “java core”.
>>>>
>>>> Please correct me, if any misunderstanding.
>>>>
>>>>
>>>>
>>>> Regards
>>>>
>>>> Basti
>>>>
>>>>
>>>>
>>>> 发件人: P. Taylor Goetz [mailto:ptgoetz@gmail.com]
>>>> 发送时间: 2015年11月11日 5:32
>>>> 收件人: dev@storm.apache.org
>>>> 主题: [DISCUSS] Plan for Merging JStorm Code
>>>>
>>>>
>>>>
>>>> Based on a number of discussions regarding merging the JStorm code,
>> I’ve tried to distill the ideas presented and inserted some of my
>> own. The result is below.
>>>>
>>>>
>>>>
>>>> I’ve divided the plan into three phases, though they are not
>> necessarily sequential — obviously some tasks can take place in parallel.
>>>>
>>>>
>>>>
>>>> None of this is set in stone, just presented for discussion. Any
>>>> and
>> all comments are welcome.
>>>>
>>>>
>>>>
>>>> -------
>>>>
>>>>
>>>>
>>>> Phase 1 - Plan for 0.11.x Release
>>>>
>>>> 1. Determine feature set for 0.11.x and publish to wiki [1].
>>>>
>>>> 2. Announce feature-freeze for 0.11.x
>>>>
>>>> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
>>>>
>>>> 4. Release 0.11.0 (or whatever version # we want to use)
>>>>
>>>> 5. Bug fixes and subsequent releases from 0.11.x-branch
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
>>>>
>>>> 1. Determine/document unique features in JStorm (e.g. classpath
>> isolation, cgroups, etc.) and create JIRA for migrating the feature.
>>>>
>>>> 2. Create JIRA for migrating each clojure component (or logical
>>>> group
>> of components) to Java. Assumes tests will be ported as well.
>>>>
>>>> 3. Discuss/establish style guide for Java coding conventions.
>>>> Consider
>> using Oracle’s or Google’s Java conventions as a base — they are both
>> pretty solid.
>>>>
>>>> 4. align package names (e.g backtype.storm --> org.apache.storm /
>>>> com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Phase 3 - Migrate Clojure --> Java
>>>>
>>>> 1. Port code/tests to Java, leveraging existing JStorm code
>>>> wherever
>> possible (core functionality only, features distinct to JStorm
>> migrated separately).
>>>>
>>>> 2. Port JStorm-specific features.
>>>>
>>>> 3. Begin releasing preview/beta versions.
>>>>
>>>> 4. Code cleanup (across the board) and refactoring using
>>>> established coding conventions, and leveraging PMD/Checkstyle
>>>> reports for reference. (Note: good oportunity for new
>>>> contributors.)
>>>>
>>>> 5. Release 0.12.0 (or whatever version # we want to use) and lift
>> feature freeze.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Notes:
>>>>
>>>> We should consider bumping up to version 1.0 sometime soon and then
>> switching to semantic versioning [3] from then on.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> With the exception of package name alignment, the "jstorm-import"
>> branch will largely be read-only throughout the process.
>>>>
>>>>
>>>>
>>>> During migration, it's probably easiest to operate with two local
>> clones of the Apache Storm repo: one for working (i.e. checked out to
>> working branch) and one for reference/copying (i.e. checked out to
>> "jstorm-import").
>>>>
>>>>
>>>>
>>>> Feature-freeze probably only needs to be enforced against core
>> functionality. Components under "external" can likely be exempt, but
>> we should figure out a process for accepting and releasing new
>> features during the migration.
>>>>
>>>>
>>>>
>>>> Performance testing should be continuous throughout the process.
>>>> Since
>> we don't really have ASF infrastructure for performance testing, we
>> will need a volunteer(s) to host and run the performance tests.
>> Performance test results can be posted to the wiki [2]. It would
>> probably be a good idea to establish a baseline with the 0.10.0 release.
>>>>
>>>>
>>>>
>>>> I’ve attached an analysis document Sean Zhong put together a while
>>>> back
>> to the JStorm merge JIRA [4]. The analysis was against the 0.9.3
>> release but is still relevant and has a lot of good information.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Fe
>>>> at
>>>> ure+Set
>>>>
>>>> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
>>>>
>>>> [3] http://semver.org
>>>>
>>>> [4] https://issues.apache.org/jira/browse/STORM-717
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -Taylor
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>