You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "刘键(夏蒅)" <ba...@alibaba-inc.com> on 2014/10/24 05:19:49 UTC
Re: Could JStorm project collaborate with Storm?
Hi Andy,
Thanks for your quick response. What you concern is reasonable and
understandable.
But maybe there is some misunderstanding on our proposal. According to the
current development situation in China, java core is indeed to bring more
contributors here.
That is reason why we begin to think if it is also a way for us to provide
our contribution to Storm project. It is true that it is a risk to switch
the core in a short period.
Actually, if possible, maybe we can consider to take JStorm as a subproject
of Storm(A branch or some other ways…). We will be responsible for the
maintain of it and try to propagate the features from Storm to JStorm, and
continue to develop our features and improvements on this. You guys can try
the new core and take a long time to estimate if it is worth doing the
switch in the future or taking JStorm as official optional core of Storm for
user.
Regards
Basti
发件人: Andrew Feng [mailto:afeng@yahoo-inc.com]
发送时间: 2014年10月24日 0:24
收件人: 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
ddas@hortonworks.com; m.ben.franklin@gmail.com; benjamin.hindman@gmail.com
抄送: aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
mlchen@iflytek.com; zeus@tencent.com
主题: Re: 答复: Could JStorm project collaborate with Storm?
Zhongyan:
Yahoo almost took your path 2 years ago. After some discussion with Nathan,
we decided to work with community on Storm, and some of us learned Clojure
quickly. We are very glad that we did that.
Under the assumption that Storm provides all features of JStorm, my personal
vote will be NO for your proposal.
* Storm has enjoyed its current success before Nathan built a very
solid core in Clojure. We should not replace that core until we are 100%
sure that alternative implementation is at least as good as our Clojure
implementation.
* Clojure has not prevent Storm attract contributors (currently 108).
We have contributors who write code in Clojure and Java. It doesn’t take
much time for one to be able to understand Clojure code.
* The convergence of streaming processing and batch processing will
occur at higher level of abstractions. Trident API, for example, is very
much similar to batch API such as Pig or Cascading.
* Including Jstorm under Storm will only create confusion to our user
community at this stage.
Why don’t you have discussion with 徐明明 to figure out an alternative path
for JStorm?
Thanks,
Andy
From: "封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
Reply-To: "封仲淹(纪君祥)" <zh...@alibaba-inc.com>
Date: Wednesday, October 22, 2014 at 11:37 PM
To: Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com"
<na...@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>,
"jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com" <fl...@infochimps.com>,
"davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
<pt...@gmail.com>, Doug Cutting <cu...@apache.org>,
"tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org"
<ar...@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>,
"m.ben.franklin@gmail.com" <m....@gmail.com>,
"benjamin.hindman@gmail.com" <be...@gmail.com>
Cc: aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com" <ya...@tencent.com>,
"mlchen@iflytek.com" <ml...@iflytek.com>, "zeus@tencent.com" <zeus@tencent.
com>
Subject: 答复: Could JStorm project collaborate with Storm?
Andrew, Thanks for suggest.
The problem is that the core of Storm is implemented by Clojure, if it is
java, we are glad to merge all our commit into the Strom Trunk.
If the core of Storm is implemented by Java, I think the contributor of
Storm would be double.
May I put forward 2 proposal:
1. Could Jstorm be the son project of Storm?
2. Task a long time to change Storm-core as Jstorm-core.
a) If the core of Storm is implemented by Java, I think the
contributor of Storm would be double. Once one user found one bug, maybe he
is able to fix it by himself, in this method, Storm resolve problem will be
double
b) In the next one or two years, batch handling and stream handling
will merge into one solution, Spark/Flink are doing this job. If we still
use clojure, I am a little afraid that we can’t follow the steps of other
community and this direction.
Thanks
Longda
发件人: Andrew Feng [mailto:afeng@yahoo-inc.com]
发送时间: 2014年10月23日 13:31
收件人: 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
ddas@hortonworks.com; m.ben.franklin@gmail.com; benjamin.hindman@gmail.com
抄送: aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
mlchen@iflytek.com; zeus@tencent.com
主题: Re: Could JStorm project collaborate with Storm?
Zhongyan:
I would suggest you identify a collection of key features in Jstorm, and
then propose pull requests to Storm.
Once these pull requests are accepted, you could then simply adopt Storm.
Andy
From: "封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
Reply-To: "封仲淹(纪君祥)" <zh...@alibaba-inc.com>
Date: Wednesday, October 22, 2014 at 8:23 PM
To: "nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)"
<mi...@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy Feng
<af...@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>,
"davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
<pt...@gmail.com>, Doug Cutting <cu...@apache.org>,
"tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org"
<ar...@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>,
"m.ben.franklin@gmail.com" <m....@gmail.com>,
"benjamin.hindman@gmail.com" <be...@gmail.com>
Cc: aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com" <ya...@tencent.com>,
"mlchen@iflytek.com" <ml...@iflytek.com>, "zeus@tencent.com" <zeus@tencent.
com>
Subject: Could JStorm project collaborate with Storm?
Hi all,
Firstly, congratulate Storm becoming the top project of Apache. Since the
first day of Storm opened source, it has become one of hottest technologies
in the BigData field and brought a new perspective to the world, which has
helped many companies resolve hundreds of requirements.
Since 2012/4, we have forked a project named "JStorm". At the beginning
time, we just wanted to implement Storm with Java. As time goes on, more and
more features have been added to JStorm. In 2013/9, JStorm changed to be an
open source project; from then on, more and more people and companies join
this project.
As JStorm is derived from Storm, and born in a great company named Alibaba,
which just successfully listed on the New York Stock Exchange on Sept. 19,
is a very open company. We also want to share our technology with more
people, and favor rapid evolution of streaming technology.
So, could JStorm project collaborate with Storm?
A little introduction for JStorm as follows:
JStorm is a distributed and fault-tolerant real-time computation system.
Inspired by Apache Storm, JStorm has been completely rewritten in Java and
provides more enhanced features. JStorm has been widely used in many
enterprise environments and proved robust and stable.
Begun from 2012/2, we have released 11 versions till 2014/09/28. More than
10 companies are using it, including several leading companies such as
Alibaba, Tencent, Baidu, Huawei, ZTE in China.
Please refer to https://github.com/alibaba/jstorm for more details
Regards
Longda
Re: 答复: Could JStorm project collaborate with Storm?
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
+1 for the plan here, but I want to be sure that everyone in the community has the opportunity to participate in any off email meeting, a summary of it really needs to be posted to the mailing lists and any decisions that require votes must be done on the propper lists, not in an outside meeting.
We want people who only have access to the mailing lists to be able to comment and participate fully in the community even if it is slower than would be otherwise.
-Bobby
答复: Could JStorm project collaborate with Storm?
Posted by "封仲淹(纪君祥)" <zh...@alibaba-inc.com>.
Sean, Thanks for your response.
Could you please leave a single phone call number to me in a single email, I will contact to you later.
Regards
Longda
发件人: Sean Zhong [mailto:clockfly@gmail.com]
发送时间: 2014年10月30日 13:27
收件人: dev@storm.apache.org
抄送: 封仲淹(纪君祥); jstorm_dev@googlegroups.com
主题: Re: Could JStorm project collaborate with Storm?
Resend this, I missed some recipients in last mail
I love this discussion.
Let's list the facts first:
1. JStorm and Storm have different feature set, but we both target on the same goals.
2. We have two community and doing things in duplicated fashion.
3. Both realize that It will take huge efforts to absort the common and good things from the other with different source repo.
4. Storm doesn't preclude java, Clojure and Java works pretty well together in storm. As Talyor said 66% of code is java.
5. Both want to do thing in a open way in apache that can everyone around the world is comfortable, and be benefited from this process.
So, IMHO, based on the above facts, we should merge these two together under the name of storm apache project.
For the difference of these two projects, here is my suggestions:
1. Review the change at the feature level.
Instead of saying whether we should replace the core from clojure to java as a whole, let's think in the view of single feature. What is the best decision for this feature. In my opinion, storm are open to the changes, including changing some part of clojure to java, but only when necessary.
2. Make the new feature plugable, and general.
If we can make some plug point in storm clojure core, then the implementation can be either in java or clojure.
For example, like the JStorm scheduler feature, it is more powerful than storm. Can we stick to the java code JStorm scheduler, and make a plug point in storm?
They are good examples for this practice in storm already. Like the storm messaging layer. The netty implementation is in java, the plugin point is in clojure. By following this process, it can largely reduce the refactory efforts we need to make.
3. Colloborate to solve the gap of clojure language, across Jstorm and Storm community.
Though I personally think Clojure is not that hard to learn, but it do requires a steep learning curve. Insteading of letting JStorm to program with a new language from zero, I suggest we stick to our most productive language in the begining, and do things colloborately in community. Java guys and Clojure guys from two community can decide the interface(plug point) first, works seperately in their own favorable language.
4. More communication channels beyond mail to negotiate the difference and make sure we have same understanding.
We also need some online chat, face to face meetup to do a more effiicient and fast loop communication and negotiation. It is important to establish common understanding.
Since JStorm guys are mostly in China. There are two guys of storm commitor in China, Xu Mingming and me(located @Intel Shanghai). I would love to volunteer to be one additional channel for early stage discussion, so that we can minimize the differences faster. By then, we will have a further considerate proposal that are acceptable for both community.
On Thu, Oct 30, 2014 at 4:17 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
After taking some time to review the JStorm code and history of both projects, here are my thoughts:
Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The disparity in LOC is primarily due to the conciseness of Clojure compared to Java. Apache Storm's Clojure code consists mainly of implementations of interfaces defined in Java. This fact makes it possible to provide an implementation in Java itself, which is exactly what the JStorm project seems to have done. JStorm is a copy of the Apache Storm codebase, with the Clojure code being replaced with a Java implementation, with some additional changes.
The justification of the project centers around a distaste for Clojure, and a seeming lack of developers with Clojure skills [1]:
"Storm is wonderful product, but it is implemented with Clojure. We don't like clojure. it isn't a populate language, so in Alibaba, few people can fix the bug of storm. We are the first users of Storm since it has been open source. During using, we found several problem such as zeromq, zookeeper, performance.”
While I agree that to someone experienced with Java development, the first exposure to Clojure can be a bit jarring, I don’t think it is much of a barrier to entry. In my opinion, learning Clojure is a valuable investment in making oneself a more well-rounded developer. The technology ecosystem to which Storm belongs is becoming increasingly polyglot with respect to JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to that fact. As an example (perhaps not the best example), lets look at Kafka. If I didn’t like or want to learn Scala, would it make sense to write a clone in Java and attempt to keep feature parity and compatibility, or would it be better to bite the bullet and learn Scala? (Yes, I know it’s not the best example since Storm’s Clojure code is a subset of the codebase, and the way it’s designed makes it possible for alternative implementations). And in reality, someone from Alibaba had to at least learn enough Clojure to be able to replicate Storm’s Clojure implementation in java.
Looking at the history of JStorm [2][3], it appears a lot of work has gone into keeping feature parity with Apache Storm by copying in new code or reimplementing it. For example a kafka spout was added about two weeks ago [4], and Yahoo’s storm-on-yarn project was also recently copied in [5].
To me, this kind of work is rather counter-productive. If the Apache Storm community feels strongly that Storm should support multiple “core” implementations (i.e. one implemented in Clojure, and one implemented in Java), then it could be accommodated by making them pluggable and rearranging the codebase a bit. That being said, the Apache Storm community has invested a significant amount of time and effort maintaining and improving Storm’s Clojure “core.” One place this is most apparent, as Bobby mentioned, is the upcoming security release. To maintain feature parity with that release a “JStorm core” would have to be updated to implement that functionality, which is not a trivial undertaking. At some point as developers we need to ask ourselves “what's the point of that work?” There are a lot more people using Storm than developing internals. We need to make sure that both groups gain value from whatever we decide, and that we don’t sacrifice any value in the process.
So to answer the question in the subject line of the email thread: Could the JStorm project collaborate with Apache Storm? My answer would be a resounding “yes.”
Though the JStorm community is small in comparison to that of Apache Storm, it’s clear that the JStorm contributors have a solid grasp of Storm’s internals and would be welcomed by the Storm community. Reimplementing Storm’s core in Java no doubt provided insight into areas that might be improved.
That being said, replacing Storm’s Clojure-based core is a decision that should not be taken lightly, and would require the support of the Apache Storm PMC and entire community. A “dual-core” setup could be an option, but again could become a maintenance nightmare with little value added for end users. I’m open to discussing options further, and welcome input from other community members.
In short, I would rather see us work together than apart.
-Taylor
[1] https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a
[2] https://github.com/alibaba/jstorm/commits/master
[3] https://github.com/alibaba/jstorm/blob/master/history.md
[4] https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka
[5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn
On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
Sorry I am coming late to the conversation on this. Looking at the code for jstorm I am willing to entertain the idea of merging the two projects. I understand that by using java we can make the project more accessible to a larger number of developers, and that does seem to fit with the apache model of community over code. But I don't really want change just for change's sake. Look at kill_topology in both java:https://github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
and in clojure:https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
The java version is 61 lines of code and is not actually compatible with the clojure version because it just counts argument for the wait parameter instead of using a -w. The clojure version is 29, but the 15 of which are the apache header, so it is closer to 14 lines. Granted, to someone who does not know you have to work a bit to read it, but not that much.
My concerns would be two fold.
First is the effort involved in maintaining two separate code bases, sub project or otherwise, is a lot. I don't want to do it, and I assume you don't want to either, that is why you are talking about merging. We would need to decide which code to keep and which code to drop, and in all likelihood some of it will still be clojure. I also would not want to restrict people from using clojure for new work if they want to, just like we don't restrict java usage for new work.
Second is true feature parity. Storm is a moving target, I would not want to replace any code, even if it is faster, without verifying that it has feature parity, or it is a conscious decision that the missing features are not needed. Especially around the security branch, which is scheduled for the 0.10.0 release, but is not a part of jstorm at all.
- Bobby
From: 封仲淹(纪君祥) <zh...@alibaba-inc.com>
To: dev@storm.apache.org; tdunning@maprtech.com
Cc: jstorm_dev@googlegroups.com
Sent: Wednesday, October 29, 2014 4:27 AM
Subject: 答复: Could JStorm project collaborate with Storm?
Thanks for your suggestion.
@dev, Does anyone has proposal? please feel free to raise.
Thanks
Longda
-----邮件原件-----
发件人: Ted Dunning [mailto:tdunning@maprtech.com]
发送时间: 2014年10月25日 7:27
收件人: dev@storm.apache.org
主题: Re: Could JStorm project collaborate with Storm?
I just moved all of the non-Storm-dev addresses on this thread to bcc. If anybody is not on the Storm dev list and would like to continue to follow this conversation, you should subscribe to the storm dev list. We should not be spamming everybody else.
There is a standard Apache process to deal with this. JStorm can become an incubator project. Once JStorm demonstrates the ability to function as an Apache project, it can graduate either to a top-level project or to become part of a project like Storm.
[Longda]This will be a feasible solution. From the Apache side, it had better Jstorm can collaborate with Storm.
A second mechanism is that if the Storm group is interested, they can directly adopt the code as it is, possibly asking one or more of the contributors to become committers to help support this new code. That is a much harder thing to do than incubation since it is unusual for a project to be enthusiastic about supporting a large new codebase.
[Longda]I agree. It is a hard job.
On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
Hi Andy,
Thanks for your quick response. What you concern is reasonable and
understandable.
But maybe there is some misunderstanding on our proposal. According to
the current development situation in China, java core is indeed to
bring more contributors here.
That is reason why we begin to think if it is also a way for us to
provide our contribution to Storm project. It is true that it is a
risk to switch the core in a short period.
Actually, if possible, maybe we can consider to take JStorm as a
subproject of Storm(A branch or some other ways…). We will be
responsible for the maintain of it and try to propagate the features
from Storm to JStorm, and continue to develop our features and
improvements on this. You guys can try the new core and take a long
time to estimate if it is worth doing the switch in the future or
taking JStorm as official optional core of Storm for user.
Regards
Basti
*发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
*发送时间:* 2014年10月24日 0:24
*收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
ddas@hortonworks.com; m.ben.franklin@gmail.com;
benjamin.hindman@gmail.com
*抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
mlchen@iflytek.com; zeus@tencent.com
*主题:* Re: 答复: Could JStorm project collaborate with Storm?
Zhongyan:
Yahoo almost took your path 2 years ago. After some discussion with
Nathan, we decided to work with community on Storm, and some of us
learned Clojure quickly. We are very glad that we did that.
Under the assumption that Storm provides all features of JStorm, my
personal vote will be NO for your proposal.
- Storm has enjoyed its current success before Nathan built a very
solid core in Clojure. We should not replace that core until we are 100%
sure that alternative implementation is at least as good as our Clojure
implementation.
- Clojure has not prevent Storm attract contributors (currently 108).
We have contributors who write code in Clojure and Java. It doesn’t take
much time for one to be able to understand Clojure code.
- The convergence of streaming processing and batch processing will
occur at higher level of abstractions. Trident API, for example, is very
much similar to batch API such as Pig or Cascading.
- Including Jstorm under Storm will only create confusion to our user
community at this stage.
Why don’t you have discussion with 徐明明 to figure out an alternative
path for JStorm?
Thanks,
Andy
*From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
*Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
*Date: *Wednesday, October 22, 2014 at 11:37 PM
*To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
<fl...@infochimps.com>, "davidlao@microsoft.com"
<da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
m.ben.franklin@gmail.com" <m....@gmail.com>, "
benjamin.hindman@gmail.com" <be...@gmail.com>
*Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com"
<ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
"zeus@tencent.com" < zeus@tencent.com>
*Subject: *答复: Could JStorm project collaborate with Storm?
Andrew, Thanks for suggest.
The problem is that the core of Storm is implemented by Clojure, if it
is java, we are glad to merge all our commit into the Strom Trunk.
If the core of Storm is implemented by Java, I think the contributor
of Storm would be double.
May I put forward 2 proposal:
1. Could Jstorm be the son project of Storm?
2. Task a long time to change Storm-core as Jstorm-core.
a) If the core of Storm is implemented by Java, I think the
contributor of Storm would be double. Once one user found one bug,
maybe he is able to fix it by himself, in this method, Storm resolve
problem will be double
b) In the next one or two years, batch handling and stream handling
will merge into one solution, Spark/Flink are doing this job. If we
still use clojure, I am a little afraid that we can’t follow the steps
of other community and this direction.
Thanks
Longda
*发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
*发送时间:* 2014年10月23日 13:31
*收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
ddas@hortonworks.com; m.ben.franklin@gmail.com;
benjamin.hindman@gmail.com
*抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
mlchen@iflytek.com; zeus@tencent.com
*主题:* Re: Could JStorm project collaborate with Storm?
Zhongyan:
I would suggest you identify a collection of key features in Jstorm,
and then propose pull requests to Storm.
Once these pull requests are accepted, you could then simply adopt Storm.
Andy
*From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
*Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
*Date: *Wednesday, October 22, 2014 at 8:23 PM
*To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>, "
davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
< ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
m.ben.franklin@gmail.com" <m....@gmail.com>, "
benjamin.hindman@gmail.com" <be...@gmail.com>
*Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com"
<ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
"zeus@tencent.com" < zeus@tencent.com>
*Subject: *Could JStorm project collaborate with Storm?
Hi all,
Firstly, congratulate Storm becoming the top project of Apache. Since
the first day of Storm opened source, it has become one of hottest
technologies in the BigData field and brought a new perspective to the
world, which has helped many companies resolve hundreds of requirements.
Since 2012/4, we have forked a project named "JStorm". At the
beginning time, we just wanted to implement Storm with Java. As time
goes on, more and more features have been added to JStorm. In 2013/9,
JStorm changed to be an open source project; from then on, more and
more people and companies join this project.
As JStorm is derived from Storm, and born in a great company named
Alibaba, which just successfully listed on the New York Stock Exchange
on Sept. 19, is a very open company. We also want to share our
technology with more people, and favor rapid evolution of streaming technology.
*So, could JStorm project collaborate with Storm? *
A little introduction for JStorm as follows:
JStorm is a distributed and fault-tolerant real-time computation system.
Inspired by Apache Storm, JStorm has been completely rewritten in Java
and provides more enhanced features. JStorm has been widely used in
many enterprise environments and proved robust and stable.
Begun from 2012/2, we have released 11 versions till 2014/09/28. More
than
10 companies are using it, including several leading companies such as
Alibaba, Tencent, Baidu, Huawei, ZTE in China.
Please refer to https://github.com/alibaba/jstorm for more details
Regards
Longda
Re: Could JStorm project collaborate with Storm?
Posted by Sean Zhong <cl...@gmail.com>.
Resend this, I missed some recipients in last mail
I love this discussion.
Let's list the facts first:
1. JStorm and Storm have different feature set, but we both target on the
same goals.
2. We have two community and doing things in duplicated fashion.
3. Both realize that It will take huge efforts to absort the common and
good things from the other with different source repo.
4. Storm doesn't preclude java, Clojure and Java works pretty well together
in storm. As Talyor said 66% of code is java.
5. Both want to do thing in a open way in apache that can everyone around
the world is comfortable, and be benefited from this process.
So, IMHO, based on the above facts, we should merge these two together
under the name of storm apache project.
For the difference of these two projects, here is my suggestions:
1. Review the change at the feature level.
Instead of saying whether we should replace the core from clojure to
java as a whole, let's think in the view of single feature. What is the
best decision for this feature. In my opinion, storm are open to the
changes, including changing some part of clojure to java, but only when
necessary.
2. Make the new feature plugable, and general.
If we can make some plug point in storm clojure core, then the
implementation can be either in java or clojure.
For example, like the JStorm scheduler feature, it is more powerful than
storm. Can we stick to the java code JStorm scheduler, and make a plug
point in storm?
They are good examples for this practice in storm already. Like the storm
messaging layer. The netty implementation is in java, the plugin point is
in clojure. By following this process, it can largely reduce the refactory
efforts we need to make.
3. Colloborate to solve the gap of clojure language, across Jstorm and
Storm community.
Though I personally think Clojure is not that hard to learn, but it do
requires a steep learning curve. Insteading of letting JStorm to program
with a new language from zero, I suggest we stick to our most productive
language in the begining, and do things colloborately in community. Java
guys and Clojure guys from two community can decide the interface(plug
point) first, works seperately in their own favorable language.
4. More communication channels beyond mail to negotiate the difference and
make sure we have same understanding.
We also need some online chat, face to face meetup to do a more effiicient
and fast loop communication and negotiation. It is important to establish
common understanding.
Since JStorm guys are mostly in China. There are two guys of storm
commitor in China, Xu Mingming and me(located @Intel Shanghai). I would
love to volunteer to be one additional channel for early stage discussion,
so that we can minimize the differences faster. By then, we will have a
further considerate proposal that are acceptable for both community.
On Thu, Oct 30, 2014 at 4:17 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> After taking some time to review the JStorm code and history of both
> projects, here are my thoughts:
>
> Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The
> disparity in LOC is primarily due to the conciseness of Clojure compared to
> Java. Apache Storm's Clojure code consists mainly of implementations of
> interfaces defined in Java. This fact makes it possible to provide an
> implementation in Java itself, which is exactly what the JStorm project
> seems to have done. JStorm is a copy of the Apache Storm codebase, with the
> Clojure code being replaced with a Java implementation, with some
> additional changes.
>
> The justification of the project centers around a distaste for Clojure,
> and a seeming lack of developers with Clojure skills [1]:
>
> "Storm is wonderful product, but it is implemented with Clojure. We don't
> like clojure. it isn't a populate language, so in Alibaba, few people can
> fix the bug of storm. We are the first users of Storm since it has been
> open source. During using, we found several problem such as zeromq,
> zookeeper, performance.”
>
>
> While I agree that to someone experienced with Java development, the first
> exposure to Clojure can be a bit jarring, I don’t think it is much of a
> barrier to entry. In my opinion, learning Clojure is a valuable investment
> in making oneself a more well-rounded developer. The technology ecosystem
> to which Storm belongs is becoming increasingly polyglot with respect to
> JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to
> that fact. As an example (perhaps not the best example), lets look at
> Kafka. If I didn’t like or want to learn Scala, would it make sense to
> write a clone in Java and attempt to keep feature parity and compatibility,
> or would it be better to bite the bullet and learn Scala? (Yes, I know it’s
> not the best example since Storm’s Clojure code is a subset of the
> codebase, and the way it’s designed makes it possible for alternative
> implementations). And in reality, someone from Alibaba had to at least
> learn enough Clojure to be able to replicate Storm’s Clojure implementation
> in java.
>
> Looking at the history of JStorm [2][3], it appears a lot of work has gone
> into keeping feature parity with Apache Storm by copying in new code or
> reimplementing it. For example a kafka spout was added about two weeks ago
> [4], and Yahoo’s storm-on-yarn project was also recently copied in [5].
>
> To me, this kind of work is rather counter-productive. If the Apache Storm
> community feels strongly that Storm should support multiple “core”
> implementations (i.e. one implemented in Clojure, and one implemented in
> Java), then it could be accommodated by making them pluggable and
> rearranging the codebase a bit. That being said, the Apache Storm community
> has invested a significant amount of time and effort maintaining and
> improving Storm’s Clojure “core.” One place this is most apparent, as Bobby
> mentioned, is the upcoming security release. To maintain feature parity
> with that release a “JStorm core” would have to be updated to implement
> that functionality, which is not a trivial undertaking. At some point as
> developers we need to ask ourselves “what's the point of that work?” There
> are a lot more people using Storm than developing internals. We need to
> make sure that both groups gain value from whatever we decide, and that we
> don’t sacrifice any value in the process.
>
> So to answer the question in the subject line of the email thread: Could
> the JStorm project collaborate with Apache Storm? My answer would be a
> resounding “yes.”
>
> Though the JStorm community is small in comparison to that of Apache
> Storm, it’s clear that the JStorm contributors have a solid grasp of
> Storm’s internals and would be welcomed by the Storm community.
> Reimplementing Storm’s core in Java no doubt provided insight into areas
> that might be improved.
>
> That being said, replacing Storm’s Clojure-based core is a decision that
> should not be taken lightly, and would require the support of the Apache
> Storm PMC and entire community. A “dual-core” setup could be an option, but
> again could become a maintenance nightmare with little value added for end
> users. I’m open to discussing options further, and welcome input from other
> community members.
>
> In short, I would rather see us work together than apart.
>
> -Taylor
>
>
> [1]
> https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a
> [2] https://github.com/alibaba/jstorm/commits/master
> [3] https://github.com/alibaba/jstorm/blob/master/history.md
> [4]
> https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka
> [5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn
>
> On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> wrote:
>
> Sorry I am coming late to the conversation on this. Looking at the code
> for jstorm I am willing to entertain the idea of merging the two projects.
> I understand that by using java we can make the project more accessible to
> a larger number of developers, and that does seem to fit with the apache
> model of community over code. But I don't really want change just for
> change's sake. Look at kill_topology in both java:https://
> github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
>
> and in clojure:https://
> github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
>
> The java version is 61 lines of code and is not actually compatible with
> the clojure version because it just counts argument for the wait parameter
> instead of using a -w. The clojure version is 29, but the 15 of which are
> the apache header, so it is closer to 14 lines. Granted, to someone who
> does not know you have to work a bit to read it, but not that much.
> My concerns would be two fold.
> First is the effort involved in maintaining two separate code bases, sub
> project or otherwise, is a lot. I don't want to do it, and I assume you
> don't want to either, that is why you are talking about merging. We would
> need to decide which code to keep and which code to drop, and in all
> likelihood some of it will still be clojure. I also would not want to
> restrict people from using clojure for new work if they want to, just like
> we don't restrict java usage for new work.
> Second is true feature parity. Storm is a moving target, I would not want
> to replace any code, even if it is faster, without verifying that it has
> feature parity, or it is a conscious decision that the missing features are
> not needed. Especially around the security branch, which is scheduled for
> the 0.10.0 release, but is not a part of jstorm at all.
> - Bobby
>
> From: 封仲淹(纪君祥) <zh...@alibaba-inc.com>
> To: dev@storm.apache.org; tdunning@maprtech.com
> Cc: jstorm_dev@googlegroups.com
> Sent: Wednesday, October 29, 2014 4:27 AM
> Subject: 答复: Could JStorm project collaborate with Storm?
>
> Thanks for your suggestion.
>
> @dev, Does anyone has proposal? please feel free to raise.
>
>
> Thanks
> Longda
>
> -----邮件原件-----
> 发件人: Ted Dunning [mailto:tdunning@maprtech.com <td...@maprtech.com>]
> 发送时间: 2014年10月25日 7:27
> 收件人: dev@storm.apache.org
> 主题: Re: Could JStorm project collaborate with Storm?
>
> I just moved all of the non-Storm-dev addresses on this thread to bcc. If
> anybody is not on the Storm dev list and would like to continue to follow
> this conversation, you should subscribe to the storm dev list. We should
> not be spamming everybody else.
>
>
> There is a standard Apache process to deal with this. JStorm can become
> an incubator project. Once JStorm demonstrates the ability to function as
> an Apache project, it can graduate either to a top-level project or to
> become part of a project like Storm.
> [Longda]This will be a feasible solution. From the Apache side, it had
> better Jstorm can collaborate with Storm.
>
> A second mechanism is that if the Storm group is interested, they can
> directly adopt the code as it is, possibly asking one or more of the
> contributors to become committers to help support this new code. That is a
> much harder thing to do than incubation since it is unusual for a project
> to be enthusiastic about supporting a large new codebase.
> [Longda]I agree. It is a hard job.
>
>
>
>
>
>
> On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
>
> Hi Andy,
>
>
>
> Thanks for your quick response. What you concern is reasonable and
> understandable.
>
>
>
> But maybe there is some misunderstanding on our proposal. According to
> the current development situation in China, java core is indeed to
> bring more contributors here.
>
> That is reason why we begin to think if it is also a way for us to
> provide our contribution to Storm project. It is true that it is a
> risk to switch the core in a short period.
>
> Actually, if possible, maybe we can consider to take JStorm as a
> subproject of Storm(A branch or some other ways…). We will be
> responsible for the maintain of it and try to propagate the features
> from Storm to JStorm, and continue to develop our features and
> improvements on this. You guys can try the new core and take a long
> time to estimate if it is worth doing the switch in the future or
> taking JStorm as official optional core of Storm for user.
>
>
>
> Regards
>
> Basti
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com> <
> afeng@yahoo-inc.com>]
> *发送时间:* 2014年10月24日 0:24
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: 答复: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> Yahoo almost took your path 2 years ago. After some discussion with
> Nathan, we decided to work with community on Storm, and some of us
> learned Clojure quickly. We are very glad that we did that.
>
>
>
> Under the assumption that Storm provides all features of JStorm, my
> personal vote will be NO for your proposal.
>
> - Storm has enjoyed its current success before Nathan built a very
> solid core in Clojure. We should not replace that core until we are
> 100%
> sure that alternative implementation is at least as good as our Clojure
> implementation.
> - Clojure has not prevent Storm attract contributors (currently 108).
> We have contributors who write code in Clojure and Java. It doesn’t
> take
> much time for one to be able to understand Clojure code.
> - The convergence of streaming processing and batch processing will
> occur at higher level of abstractions. Trident API, for example, is
> very
> much similar to batch API such as Pig or Cascading.
> - Including Jstorm under Storm will only create confusion to our user
> community at this stage.
>
>
>
> Why don’t you have discussion with 徐明明 to figure out an alternative
> path for JStorm?
>
>
>
> Thanks,
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 11:37 PM
> *To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
> nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
> jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
> <fl...@infochimps.com>, "davidlao@microsoft.com"
> <da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug
> Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *答复: Could JStorm project collaborate with Storm?
>
>
>
> Andrew, Thanks for suggest.
>
>
>
> The problem is that the core of Storm is implemented by Clojure, if it
> is java, we are glad to merge all our commit into the Strom Trunk.
>
> If the core of Storm is implemented by Java, I think the contributor
> of Storm would be double.
>
>
>
>
>
> May I put forward 2 proposal:
>
> 1. Could Jstorm be the son project of Storm?
>
> 2. Task a long time to change Storm-core as Jstorm-core.
>
> a) If the core of Storm is implemented by Java, I think the
> contributor of Storm would be double. Once one user found one bug,
> maybe he is able to fix it by himself, in this method, Storm resolve
> problem will be double
>
> b) In the next one or two years, batch handling and stream handling
> will merge into one solution, Spark/Flink are doing this job. If we
> still use clojure, I am a little afraid that we can’t follow the steps
> of other community and this direction.
>
>
>
>
>
> Thanks
>
> Longda
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月23日 13:31
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> I would suggest you identify a collection of key features in Jstorm,
> and then propose pull requests to Storm.
>
> Once these pull requests are accepted, you could then simply adopt Storm.
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 8:23 PM
> *To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
> mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
> Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>,
> "
> davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
> < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *Could JStorm project collaborate with Storm?
>
>
>
> Hi all,
>
>
>
> Firstly, congratulate Storm becoming the top project of Apache. Since
> the first day of Storm opened source, it has become one of hottest
> technologies in the BigData field and brought a new perspective to the
> world, which has helped many companies resolve hundreds of requirements.
>
>
>
> Since 2012/4, we have forked a project named "JStorm". At the
> beginning time, we just wanted to implement Storm with Java. As time
> goes on, more and more features have been added to JStorm. In 2013/9,
> JStorm changed to be an open source project; from then on, more and
> more people and companies join this project.
>
>
>
> As JStorm is derived from Storm, and born in a great company named
> Alibaba, which just successfully listed on the New York Stock Exchange
> on Sept. 19, is a very open company. We also want to share our
> technology with more people, and favor rapid evolution of streaming
> technology.
>
>
>
> *So, could JStorm project collaborate with Storm? *
>
>
>
> A little introduction for JStorm as follows:
>
> JStorm is a distributed and fault-tolerant real-time computation system.
> Inspired by Apache Storm, JStorm has been completely rewritten in Java
> and provides more enhanced features. JStorm has been widely used in
> many enterprise environments and proved robust and stable.
>
>
>
> Begun from 2012/2, we have released 11 versions till 2014/09/28. More
> than
> 10 companies are using it, including several leading companies such as
> Alibaba, Tencent, Baidu, Huawei, ZTE in China.
>
>
>
> Please refer to https://github.com/alibaba/jstorm for more details
>
>
>
>
>
> Regards
>
> Longda
>
>
>
>
>
Re: Could JStorm project collaborate with Storm?
Posted by Sean Zhong <cl...@gmail.com>.
I love this discussion.
Let's list the facts first:
1. JStorm and Storm have different feature set, but we both target on the
same goals.
2. We have two community and doing things in duplicated fashion.
3. Both realize that It will take huge efforts to absort the common and
good things from the other with different source repo.
4. Storm doesn't preclude java, Clojure and Java works pretty well together
in storm. As Talyor said 66% of code is java.
5. Both want to do thing in a open way in apache that can everyone around
the world is comfortable, and be benefited from this process.
So, IMHO, based on the above facts, we should merge these two together
under the name of storm apache project.
For the difference of these two projects, here is my suggestions:
1. Review the change at the feature level.
Instead of saying whether we should replace the core from clojure to
java as a whole, let's think in the view of single feature. What is the
best decision for this feature. In my opinion, storm are open to the
changes, including changing some part of clojure to java, but only when
necessary.
2. Make the new feature plugable, and general.
If we can make some plug point in storm clojure core, then the
implementation can be either in java or clojure.
For example, like the JStorm scheduler feature, it is more powerful than
storm. Can we stick to the java code JStorm scheduler, and make a plug
point in storm?
They are good examples for this practice in storm already. Like the storm
messaging layer. The netty implementation is in java, the plugin point is
in clojure. By following this process, it can largely reduce the refactory
efforts we need to make.
3. Colloborate to solve the gap of clojure language, across Jstorm and
Storm community.
Though I personally think Clojure is not that hard to learn, but it do
requires a steep learning curve. Insteading of letting JStorm to program
with a new language from zero, I suggest we stick to our most productive
language in the begining, and do things colloborately in community. Java
guys and Clojure guys from two community can decide the interface(plug
point) first, works seperately in their own favorable language.
4. More communication channels beyond mail to negotiate the difference and
make sure we have same understanding.
We also need some online chat, face to face meetup to do a more effiicient
and fast loop communication and negotiation. It is important to establish
common understanding.
Since JStorm guys are mostly in China. There are two guys of storm
commitor in China, Xu Mingming and me(located @Intel Shanghai). I would
love to volunteer to be one additional channel for early stage discussion,
so that we can minimize the differences faster. By then, we will have a
further considerate proposal that are acceptable for both community.
Sean
On Thu, Oct 30, 2014 at 9:39 AM, 임정택 <ka...@gmail.com> wrote:
> As a small contributor but willing to contribute more, let me give just 2
> cents to Storm community.
>
> I think it's essential to keep Storm's originality, so merging with JStorm
> should be treated seriously. And let JStorm developers learn clojure and
> contribute to Storm may be faster than merging.
>
> But at this time, we can think about why JStorm is born. It's due to
> learning curve of Clojure.
> I agree Taylor's comment, developers need to be polyglot programmer.
> But I don't agree we can/should learn all of languages whenever we need to.
> It should be a big wall.
>
> Think about a newbie Storm contributors that know only Java.
> Not intended to make a flame between languages, but if he/she need to learn
> a language runs with JVM but not Java, he/she will find which language is
> widely used, with regarding trends.
>
> And it's github trending repositories, clojure and scala.
> https://github.com/trending?l=clojure&since=monthly
> https://github.com/trending?l=scala&since=monthly
>
> Since I don't know Scala well either, but I feel sometimes I need to learn
> Scala because many active big projects are relied on. But Clojure seems
> not.
> (Maybe I cannot find right things, so please correct me if I'm wrong.)
>
> And Scala has similar looks with OOP language, especially Java, so Java
> programmer can implement something like Java with small learning, though it
> would not beautiful and someone blames it's not Scala. But Clojure is not.
> He/She cannot read/write if he/she is familiar with Clojure, at least Lisp.
> It's not that simple hurdle to newbies about to learn new language.
>
> Btw, recently I read yahoo's storm contribution doc, 'The Evolution of
> Storm at Yahoo and Apache'. They (including committers of Storm) learned
> unfamiliar language, Clojure.
> So I have a question regarding it.
> What days/hours did you spent to get familiar with Clojure? What materials
> did you use to learn Clojure?
>
> Actually I'm committer of Jedis project, and I feel Jedis can be improved
> with advanced language but I think we can't move Jedis because Java is
> widely-used language so it's better to collect contributors.
> Seems like Storm project has same question about this. So it's interesting
> to look into discussion.
>
> Just my 2 cents.
>
> Regards.
> Jungtaek Lim (HeartSaVioR)
>
>
> 2014-10-30 5:17 GMT+09:00 P. Taylor Goetz <pt...@gmail.com>:
>
> > After taking some time to review the JStorm code and history of both
> > projects, here are my thoughts:
> >
> > Apache Storm is largely implemented in Java (~68%) and Clojure (~18%).
> The
> > disparity in LOC is primarily due to the conciseness of Clojure compared
> to
> > Java. Apache Storm's Clojure code consists mainly of implementations of
> > interfaces defined in Java. This fact makes it possible to provide an
> > implementation in Java itself, which is exactly what the JStorm project
> > seems to have done. JStorm is a copy of the Apache Storm codebase, with
> the
> > Clojure code being replaced with a Java implementation, with some
> > additional changes.
> >
> > The justification of the project centers around a distaste for Clojure,
> > and a seeming lack of developers with Clojure skills [1]:
> >
> > "Storm is wonderful product, but it is implemented with Clojure. We don't
> > like clojure. it isn't a populate language, so in Alibaba, few people can
> > fix the bug of storm. We are the first users of Storm since it has been
> > open source. During using, we found several problem such as zeromq,
> > zookeeper, performance.”
> >
> >
> > While I agree that to someone experienced with Java development, the
> first
> > exposure to Clojure can be a bit jarring, I don’t think it is much of a
> > barrier to entry. In my opinion, learning Clojure is a valuable
> investment
> > in making oneself a more well-rounded developer. The technology ecosystem
> > to which Storm belongs is becoming increasingly polyglot with respect to
> > JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt
> to
> > that fact. As an example (perhaps not the best example), lets look at
> > Kafka. If I didn’t like or want to learn Scala, would it make sense to
> > write a clone in Java and attempt to keep feature parity and
> compatibility,
> > or would it be better to bite the bullet and learn Scala? (Yes, I know
> it’s
> > not the best example since Storm’s Clojure code is a subset of the
> > codebase, and the way it’s designed makes it possible for alternative
> > implementations). And in reality, someone from Alibaba had to at least
> > learn enough Clojure to be able to replicate Storm’s Clojure
> implementation
> > in java.
> >
> > Looking at the history of JStorm [2][3], it appears a lot of work has
> gone
> > into keeping feature parity with Apache Storm by copying in new code or
> > reimplementing it. For example a kafka spout was added about two weeks
> ago
> > [4], and Yahoo’s storm-on-yarn project was also recently copied in [5].
> >
> > To me, this kind of work is rather counter-productive. If the Apache
> Storm
> > community feels strongly that Storm should support multiple “core”
> > implementations (i.e. one implemented in Clojure, and one implemented in
> > Java), then it could be accommodated by making them pluggable and
> > rearranging the codebase a bit. That being said, the Apache Storm
> community
> > has invested a significant amount of time and effort maintaining and
> > improving Storm’s Clojure “core.” One place this is most apparent, as
> Bobby
> > mentioned, is the upcoming security release. To maintain feature parity
> > with that release a “JStorm core” would have to be updated to implement
> > that functionality, which is not a trivial undertaking. At some point as
> > developers we need to ask ourselves “what's the point of that work?”
> There
> > are a lot more people using Storm than developing internals. We need to
> > make sure that both groups gain value from whatever we decide, and that
> we
> > don’t sacrifice any value in the process.
> >
> > So to answer the question in the subject line of the email thread: Could
> > the JStorm project collaborate with Apache Storm? My answer would be a
> > resounding “yes.”
> >
> > Though the JStorm community is small in comparison to that of Apache
> > Storm, it’s clear that the JStorm contributors have a solid grasp of
> > Storm’s internals and would be welcomed by the Storm community.
> > Reimplementing Storm’s core in Java no doubt provided insight into areas
> > that might be improved.
> >
> > That being said, replacing Storm’s Clojure-based core is a decision that
> > should not be taken lightly, and would require the support of the Apache
> > Storm PMC and entire community. A “dual-core” setup could be an option,
> but
> > again could become a maintenance nightmare with little value added for
> end
> > users. I’m open to discussing options further, and welcome input from
> other
> > community members.
> >
> > In short, I would rather see us work together than apart.
> >
> > -Taylor
> >
> >
> > [1]
> >
> https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a
> > [2] https://github.com/alibaba/jstorm/commits/master
> > [3] https://github.com/alibaba/jstorm/blob/master/history.md
> > [4]
> >
> https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka
> > [5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn
> >
> > On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> > wrote:
> >
> > Sorry I am coming late to the conversation on this. Looking at the code
> > for jstorm I am willing to entertain the idea of merging the two
> projects.
> > I understand that by using java we can make the project more accessible
> to
> > a larger number of developers, and that does seem to fit with the apache
> > model of community over code. But I don't really want change just for
> > change's sake. Look at kill_topology in both java:https://
> >
> github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
> >
> > and in clojure:https://
> >
> github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
> >
> > The java version is 61 lines of code and is not actually compatible with
> > the clojure version because it just counts argument for the wait
> parameter
> > instead of using a -w. The clojure version is 29, but the 15 of which
> are
> > the apache header, so it is closer to 14 lines. Granted, to someone who
> > does not know you have to work a bit to read it, but not that much.
> > My concerns would be two fold.
> > First is the effort involved in maintaining two separate code bases, sub
> > project or otherwise, is a lot. I don't want to do it, and I assume you
> > don't want to either, that is why you are talking about merging. We
> would
> > need to decide which code to keep and which code to drop, and in all
> > likelihood some of it will still be clojure. I also would not want to
> > restrict people from using clojure for new work if they want to, just
> like
> > we don't restrict java usage for new work.
> > Second is true feature parity. Storm is a moving target, I would not
> want
> > to replace any code, even if it is faster, without verifying that it has
> > feature parity, or it is a conscious decision that the missing features
> are
> > not needed. Especially around the security branch, which is scheduled for
> > the 0.10.0 release, but is not a part of jstorm at all.
> > - Bobby
> >
> > From: 封仲淹(纪君祥) <zh...@alibaba-inc.com>
> > To: dev@storm.apache.org; tdunning@maprtech.com
> > Cc: jstorm_dev@googlegroups.com
> > Sent: Wednesday, October 29, 2014 4:27 AM
> > Subject: 答复: Could JStorm project collaborate with Storm?
> >
> > Thanks for your suggestion.
> >
> > @dev, Does anyone has proposal? please feel free to raise.
> >
> >
> > Thanks
> > Longda
> >
> > -----邮件原件-----
> > 发件人: Ted Dunning [mailto:tdunning@maprtech.com <td...@maprtech.com>]
> > 发送时间: 2014年10月25日 7:27
> > 收件人: dev@storm.apache.org
> > 主题: Re: Could JStorm project collaborate with Storm?
> >
> > I just moved all of the non-Storm-dev addresses on this thread to bcc.
> If
> > anybody is not on the Storm dev list and would like to continue to follow
> > this conversation, you should subscribe to the storm dev list. We should
> > not be spamming everybody else.
> >
> >
> > There is a standard Apache process to deal with this. JStorm can become
> > an incubator project. Once JStorm demonstrates the ability to function
> as
> > an Apache project, it can graduate either to a top-level project or to
> > become part of a project like Storm.
> > [Longda]This will be a feasible solution. From the Apache side, it had
> > better Jstorm can collaborate with Storm.
> >
> > A second mechanism is that if the Storm group is interested, they can
> > directly adopt the code as it is, possibly asking one or more of the
> > contributors to become committers to help support this new code. That
> is a
> > much harder thing to do than incubation since it is unusual for a project
> > to be enthusiastic about supporting a large new codebase.
> > [Longda]I agree. It is a hard job.
> >
> >
> >
> >
> >
> >
> > On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com>
> wrote:
> >
> > Hi Andy,
> >
> >
> >
> > Thanks for your quick response. What you concern is reasonable and
> > understandable.
> >
> >
> >
> > But maybe there is some misunderstanding on our proposal. According to
> > the current development situation in China, java core is indeed to
> > bring more contributors here.
> >
> > That is reason why we begin to think if it is also a way for us to
> > provide our contribution to Storm project. It is true that it is a
> > risk to switch the core in a short period.
> >
> > Actually, if possible, maybe we can consider to take JStorm as a
> > subproject of Storm(A branch or some other ways…). We will be
> > responsible for the maintain of it and try to propagate the features
> > from Storm to JStorm, and continue to develop our features and
> > improvements on this. You guys can try the new core and take a long
> > time to estimate if it is worth doing the switch in the future or
> > taking JStorm as official optional core of Storm for user.
> >
> >
> >
> > Regards
> >
> > Basti
> >
> >
> >
> > *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com> <
> > afeng@yahoo-inc.com>]
> > *发送时间:* 2014年10月24日 0:24
> > *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> > flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> > cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> > ddas@hortonworks.com; m.ben.franklin@gmail.com;
> > benjamin.hindman@gmail.com
> > *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> > mlchen@iflytek.com; zeus@tencent.com
> > *主题:* Re: 答复: Could JStorm project collaborate with Storm?
> >
> >
> >
> > Zhongyan:
> >
> >
> >
> > Yahoo almost took your path 2 years ago. After some discussion with
> > Nathan, we decided to work with community on Storm, and some of us
> > learned Clojure quickly. We are very glad that we did that.
> >
> >
> >
> > Under the assumption that Storm provides all features of JStorm, my
> > personal vote will be NO for your proposal.
> >
> > - Storm has enjoyed its current success before Nathan built a very
> > solid core in Clojure. We should not replace that core until we are
> > 100%
> > sure that alternative implementation is at least as good as our
> Clojure
> > implementation.
> > - Clojure has not prevent Storm attract contributors (currently 108).
> > We have contributors who write code in Clojure and Java. It doesn’t
> > take
> > much time for one to be able to understand Clojure code.
> > - The convergence of streaming processing and batch processing will
> > occur at higher level of abstractions. Trident API, for example, is
> > very
> > much similar to batch API such as Pig or Cascading.
> > - Including Jstorm under Storm will only create confusion to our user
> > community at this stage.
> >
> >
> >
> > Why don’t you have discussion with 徐明明 to figure out an alternative
> > path for JStorm?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Andy
> >
> >
> >
> > *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> > *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> > *Date: *Wednesday, October 22, 2014 at 11:37 PM
> > *To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
> > nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
> > jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
> > <fl...@infochimps.com>, "davidlao@microsoft.com"
> > <da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug
> > Cutting <cu...@apache.org>, "
> > tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> > arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> > m.ben.franklin@gmail.com" <m....@gmail.com>, "
> > benjamin.hindman@gmail.com" <be...@gmail.com>
> > *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "
> scott.zhai@vipshop.com"
> > <sc...@vipshop.com>, "yannianmu@tencent.com"
> > <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> > "zeus@tencent.com" < zeus@tencent.com>
> > *Subject: *答复: Could JStorm project collaborate with Storm?
> >
> >
> >
> > Andrew, Thanks for suggest.
> >
> >
> >
> > The problem is that the core of Storm is implemented by Clojure, if it
> > is java, we are glad to merge all our commit into the Strom Trunk.
> >
> > If the core of Storm is implemented by Java, I think the contributor
> > of Storm would be double.
> >
> >
> >
> >
> >
> > May I put forward 2 proposal:
> >
> > 1. Could Jstorm be the son project of Storm?
> >
> > 2. Task a long time to change Storm-core as Jstorm-core.
> >
> > a) If the core of Storm is implemented by Java, I think the
> > contributor of Storm would be double. Once one user found one bug,
> > maybe he is able to fix it by himself, in this method, Storm resolve
> > problem will be double
> >
> > b) In the next one or two years, batch handling and stream handling
> > will merge into one solution, Spark/Flink are doing this job. If we
> > still use clojure, I am a little afraid that we can’t follow the steps
> > of other community and this direction.
> >
> >
> >
> >
> >
> > Thanks
> >
> > Longda
> >
> >
> >
> > *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> > *发送时间:* 2014年10月23日 13:31
> > *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> > flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> > cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> > ddas@hortonworks.com; m.ben.franklin@gmail.com;
> > benjamin.hindman@gmail.com
> > *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> > mlchen@iflytek.com; zeus@tencent.com
> > *主题:* Re: Could JStorm project collaborate with Storm?
> >
> >
> >
> > Zhongyan:
> >
> >
> >
> > I would suggest you identify a collection of key features in Jstorm,
> > and then propose pull requests to Storm.
> >
> > Once these pull requests are accepted, you could then simply adopt Storm.
> >
> >
> >
> > Andy
> >
> >
> >
> > *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> > *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> > *Date: *Wednesday, October 22, 2014 at 8:23 PM
> > *To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
> > mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
> > Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <flip@infochimps.com
> >,
> > "
> > davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
> > < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> > tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> > arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> > m.ben.franklin@gmail.com" <m....@gmail.com>, "
> > benjamin.hindman@gmail.com" <be...@gmail.com>
> > *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "
> scott.zhai@vipshop.com"
> > <sc...@vipshop.com>, "yannianmu@tencent.com"
> > <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> > "zeus@tencent.com" < zeus@tencent.com>
> > *Subject: *Could JStorm project collaborate with Storm?
> >
> >
> >
> > Hi all,
> >
> >
> >
> > Firstly, congratulate Storm becoming the top project of Apache. Since
> > the first day of Storm opened source, it has become one of hottest
> > technologies in the BigData field and brought a new perspective to the
> > world, which has helped many companies resolve hundreds of requirements.
> >
> >
> >
> > Since 2012/4, we have forked a project named "JStorm". At the
> > beginning time, we just wanted to implement Storm with Java. As time
> > goes on, more and more features have been added to JStorm. In 2013/9,
> > JStorm changed to be an open source project; from then on, more and
> > more people and companies join this project.
> >
> >
> >
> > As JStorm is derived from Storm, and born in a great company named
> > Alibaba, which just successfully listed on the New York Stock Exchange
> > on Sept. 19, is a very open company. We also want to share our
> > technology with more people, and favor rapid evolution of streaming
> > technology.
> >
> >
> >
> > *So, could JStorm project collaborate with Storm? *
> >
> >
> >
> > A little introduction for JStorm as follows:
> >
> > JStorm is a distributed and fault-tolerant real-time computation system.
> > Inspired by Apache Storm, JStorm has been completely rewritten in Java
> > and provides more enhanced features. JStorm has been widely used in
> > many enterprise environments and proved robust and stable.
> >
> >
> >
> > Begun from 2012/2, we have released 11 versions till 2014/09/28. More
> > than
> > 10 companies are using it, including several leading companies such as
> > Alibaba, Tencent, Baidu, Huawei, ZTE in China.
> >
> >
> >
> > Please refer to https://github.com/alibaba/jstorm for more details
> >
> >
> >
> >
> >
> > Regards
> >
> > Longda
> >
> >
> >
> >
> >
>
>
> --
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
Re: Could JStorm project collaborate with Storm?
Posted by 임정택 <ka...@gmail.com>.
As a small contributor but willing to contribute more, let me give just 2
cents to Storm community.
I think it's essential to keep Storm's originality, so merging with JStorm
should be treated seriously. And let JStorm developers learn clojure and
contribute to Storm may be faster than merging.
But at this time, we can think about why JStorm is born. It's due to
learning curve of Clojure.
I agree Taylor's comment, developers need to be polyglot programmer.
But I don't agree we can/should learn all of languages whenever we need to.
It should be a big wall.
Think about a newbie Storm contributors that know only Java.
Not intended to make a flame between languages, but if he/she need to learn
a language runs with JVM but not Java, he/she will find which language is
widely used, with regarding trends.
And it's github trending repositories, clojure and scala.
https://github.com/trending?l=clojure&since=monthly
https://github.com/trending?l=scala&since=monthly
Since I don't know Scala well either, but I feel sometimes I need to learn
Scala because many active big projects are relied on. But Clojure seems not.
(Maybe I cannot find right things, so please correct me if I'm wrong.)
And Scala has similar looks with OOP language, especially Java, so Java
programmer can implement something like Java with small learning, though it
would not beautiful and someone blames it's not Scala. But Clojure is not.
He/She cannot read/write if he/she is familiar with Clojure, at least Lisp.
It's not that simple hurdle to newbies about to learn new language.
Btw, recently I read yahoo's storm contribution doc, 'The Evolution of
Storm at Yahoo and Apache'. They (including committers of Storm) learned
unfamiliar language, Clojure.
So I have a question regarding it.
What days/hours did you spent to get familiar with Clojure? What materials
did you use to learn Clojure?
Actually I'm committer of Jedis project, and I feel Jedis can be improved
with advanced language but I think we can't move Jedis because Java is
widely-used language so it's better to collect contributors.
Seems like Storm project has same question about this. So it's interesting
to look into discussion.
Just my 2 cents.
Regards.
Jungtaek Lim (HeartSaVioR)
2014-10-30 5:17 GMT+09:00 P. Taylor Goetz <pt...@gmail.com>:
> After taking some time to review the JStorm code and history of both
> projects, here are my thoughts:
>
> Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The
> disparity in LOC is primarily due to the conciseness of Clojure compared to
> Java. Apache Storm's Clojure code consists mainly of implementations of
> interfaces defined in Java. This fact makes it possible to provide an
> implementation in Java itself, which is exactly what the JStorm project
> seems to have done. JStorm is a copy of the Apache Storm codebase, with the
> Clojure code being replaced with a Java implementation, with some
> additional changes.
>
> The justification of the project centers around a distaste for Clojure,
> and a seeming lack of developers with Clojure skills [1]:
>
> "Storm is wonderful product, but it is implemented with Clojure. We don't
> like clojure. it isn't a populate language, so in Alibaba, few people can
> fix the bug of storm. We are the first users of Storm since it has been
> open source. During using, we found several problem such as zeromq,
> zookeeper, performance.”
>
>
> While I agree that to someone experienced with Java development, the first
> exposure to Clojure can be a bit jarring, I don’t think it is much of a
> barrier to entry. In my opinion, learning Clojure is a valuable investment
> in making oneself a more well-rounded developer. The technology ecosystem
> to which Storm belongs is becoming increasingly polyglot with respect to
> JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to
> that fact. As an example (perhaps not the best example), lets look at
> Kafka. If I didn’t like or want to learn Scala, would it make sense to
> write a clone in Java and attempt to keep feature parity and compatibility,
> or would it be better to bite the bullet and learn Scala? (Yes, I know it’s
> not the best example since Storm’s Clojure code is a subset of the
> codebase, and the way it’s designed makes it possible for alternative
> implementations). And in reality, someone from Alibaba had to at least
> learn enough Clojure to be able to replicate Storm’s Clojure implementation
> in java.
>
> Looking at the history of JStorm [2][3], it appears a lot of work has gone
> into keeping feature parity with Apache Storm by copying in new code or
> reimplementing it. For example a kafka spout was added about two weeks ago
> [4], and Yahoo’s storm-on-yarn project was also recently copied in [5].
>
> To me, this kind of work is rather counter-productive. If the Apache Storm
> community feels strongly that Storm should support multiple “core”
> implementations (i.e. one implemented in Clojure, and one implemented in
> Java), then it could be accommodated by making them pluggable and
> rearranging the codebase a bit. That being said, the Apache Storm community
> has invested a significant amount of time and effort maintaining and
> improving Storm’s Clojure “core.” One place this is most apparent, as Bobby
> mentioned, is the upcoming security release. To maintain feature parity
> with that release a “JStorm core” would have to be updated to implement
> that functionality, which is not a trivial undertaking. At some point as
> developers we need to ask ourselves “what's the point of that work?” There
> are a lot more people using Storm than developing internals. We need to
> make sure that both groups gain value from whatever we decide, and that we
> don’t sacrifice any value in the process.
>
> So to answer the question in the subject line of the email thread: Could
> the JStorm project collaborate with Apache Storm? My answer would be a
> resounding “yes.”
>
> Though the JStorm community is small in comparison to that of Apache
> Storm, it’s clear that the JStorm contributors have a solid grasp of
> Storm’s internals and would be welcomed by the Storm community.
> Reimplementing Storm’s core in Java no doubt provided insight into areas
> that might be improved.
>
> That being said, replacing Storm’s Clojure-based core is a decision that
> should not be taken lightly, and would require the support of the Apache
> Storm PMC and entire community. A “dual-core” setup could be an option, but
> again could become a maintenance nightmare with little value added for end
> users. I’m open to discussing options further, and welcome input from other
> community members.
>
> In short, I would rather see us work together than apart.
>
> -Taylor
>
>
> [1]
> https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a
> [2] https://github.com/alibaba/jstorm/commits/master
> [3] https://github.com/alibaba/jstorm/blob/master/history.md
> [4]
> https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka
> [5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn
>
> On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> wrote:
>
> Sorry I am coming late to the conversation on this. Looking at the code
> for jstorm I am willing to entertain the idea of merging the two projects.
> I understand that by using java we can make the project more accessible to
> a larger number of developers, and that does seem to fit with the apache
> model of community over code. But I don't really want change just for
> change's sake. Look at kill_topology in both java:https://
> github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
>
> and in clojure:https://
> github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
>
> The java version is 61 lines of code and is not actually compatible with
> the clojure version because it just counts argument for the wait parameter
> instead of using a -w. The clojure version is 29, but the 15 of which are
> the apache header, so it is closer to 14 lines. Granted, to someone who
> does not know you have to work a bit to read it, but not that much.
> My concerns would be two fold.
> First is the effort involved in maintaining two separate code bases, sub
> project or otherwise, is a lot. I don't want to do it, and I assume you
> don't want to either, that is why you are talking about merging. We would
> need to decide which code to keep and which code to drop, and in all
> likelihood some of it will still be clojure. I also would not want to
> restrict people from using clojure for new work if they want to, just like
> we don't restrict java usage for new work.
> Second is true feature parity. Storm is a moving target, I would not want
> to replace any code, even if it is faster, without verifying that it has
> feature parity, or it is a conscious decision that the missing features are
> not needed. Especially around the security branch, which is scheduled for
> the 0.10.0 release, but is not a part of jstorm at all.
> - Bobby
>
> From: 封仲淹(纪君祥) <zh...@alibaba-inc.com>
> To: dev@storm.apache.org; tdunning@maprtech.com
> Cc: jstorm_dev@googlegroups.com
> Sent: Wednesday, October 29, 2014 4:27 AM
> Subject: 答复: Could JStorm project collaborate with Storm?
>
> Thanks for your suggestion.
>
> @dev, Does anyone has proposal? please feel free to raise.
>
>
> Thanks
> Longda
>
> -----邮件原件-----
> 发件人: Ted Dunning [mailto:tdunning@maprtech.com <td...@maprtech.com>]
> 发送时间: 2014年10月25日 7:27
> 收件人: dev@storm.apache.org
> 主题: Re: Could JStorm project collaborate with Storm?
>
> I just moved all of the non-Storm-dev addresses on this thread to bcc. If
> anybody is not on the Storm dev list and would like to continue to follow
> this conversation, you should subscribe to the storm dev list. We should
> not be spamming everybody else.
>
>
> There is a standard Apache process to deal with this. JStorm can become
> an incubator project. Once JStorm demonstrates the ability to function as
> an Apache project, it can graduate either to a top-level project or to
> become part of a project like Storm.
> [Longda]This will be a feasible solution. From the Apache side, it had
> better Jstorm can collaborate with Storm.
>
> A second mechanism is that if the Storm group is interested, they can
> directly adopt the code as it is, possibly asking one or more of the
> contributors to become committers to help support this new code. That is a
> much harder thing to do than incubation since it is unusual for a project
> to be enthusiastic about supporting a large new codebase.
> [Longda]I agree. It is a hard job.
>
>
>
>
>
>
> On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
>
> Hi Andy,
>
>
>
> Thanks for your quick response. What you concern is reasonable and
> understandable.
>
>
>
> But maybe there is some misunderstanding on our proposal. According to
> the current development situation in China, java core is indeed to
> bring more contributors here.
>
> That is reason why we begin to think if it is also a way for us to
> provide our contribution to Storm project. It is true that it is a
> risk to switch the core in a short period.
>
> Actually, if possible, maybe we can consider to take JStorm as a
> subproject of Storm(A branch or some other ways…). We will be
> responsible for the maintain of it and try to propagate the features
> from Storm to JStorm, and continue to develop our features and
> improvements on this. You guys can try the new core and take a long
> time to estimate if it is worth doing the switch in the future or
> taking JStorm as official optional core of Storm for user.
>
>
>
> Regards
>
> Basti
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com> <
> afeng@yahoo-inc.com>]
> *发送时间:* 2014年10月24日 0:24
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: 答复: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> Yahoo almost took your path 2 years ago. After some discussion with
> Nathan, we decided to work with community on Storm, and some of us
> learned Clojure quickly. We are very glad that we did that.
>
>
>
> Under the assumption that Storm provides all features of JStorm, my
> personal vote will be NO for your proposal.
>
> - Storm has enjoyed its current success before Nathan built a very
> solid core in Clojure. We should not replace that core until we are
> 100%
> sure that alternative implementation is at least as good as our Clojure
> implementation.
> - Clojure has not prevent Storm attract contributors (currently 108).
> We have contributors who write code in Clojure and Java. It doesn’t
> take
> much time for one to be able to understand Clojure code.
> - The convergence of streaming processing and batch processing will
> occur at higher level of abstractions. Trident API, for example, is
> very
> much similar to batch API such as Pig or Cascading.
> - Including Jstorm under Storm will only create confusion to our user
> community at this stage.
>
>
>
> Why don’t you have discussion with 徐明明 to figure out an alternative
> path for JStorm?
>
>
>
> Thanks,
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 11:37 PM
> *To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
> nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
> jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
> <fl...@infochimps.com>, "davidlao@microsoft.com"
> <da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug
> Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *答复: Could JStorm project collaborate with Storm?
>
>
>
> Andrew, Thanks for suggest.
>
>
>
> The problem is that the core of Storm is implemented by Clojure, if it
> is java, we are glad to merge all our commit into the Strom Trunk.
>
> If the core of Storm is implemented by Java, I think the contributor
> of Storm would be double.
>
>
>
>
>
> May I put forward 2 proposal:
>
> 1. Could Jstorm be the son project of Storm?
>
> 2. Task a long time to change Storm-core as Jstorm-core.
>
> a) If the core of Storm is implemented by Java, I think the
> contributor of Storm would be double. Once one user found one bug,
> maybe he is able to fix it by himself, in this method, Storm resolve
> problem will be double
>
> b) In the next one or two years, batch handling and stream handling
> will merge into one solution, Spark/Flink are doing this job. If we
> still use clojure, I am a little afraid that we can’t follow the steps
> of other community and this direction.
>
>
>
>
>
> Thanks
>
> Longda
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月23日 13:31
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> I would suggest you identify a collection of key features in Jstorm,
> and then propose pull requests to Storm.
>
> Once these pull requests are accepted, you could then simply adopt Storm.
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 8:23 PM
> *To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
> mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
> Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>,
> "
> davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
> < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *Could JStorm project collaborate with Storm?
>
>
>
> Hi all,
>
>
>
> Firstly, congratulate Storm becoming the top project of Apache. Since
> the first day of Storm opened source, it has become one of hottest
> technologies in the BigData field and brought a new perspective to the
> world, which has helped many companies resolve hundreds of requirements.
>
>
>
> Since 2012/4, we have forked a project named "JStorm". At the
> beginning time, we just wanted to implement Storm with Java. As time
> goes on, more and more features have been added to JStorm. In 2013/9,
> JStorm changed to be an open source project; from then on, more and
> more people and companies join this project.
>
>
>
> As JStorm is derived from Storm, and born in a great company named
> Alibaba, which just successfully listed on the New York Stock Exchange
> on Sept. 19, is a very open company. We also want to share our
> technology with more people, and favor rapid evolution of streaming
> technology.
>
>
>
> *So, could JStorm project collaborate with Storm? *
>
>
>
> A little introduction for JStorm as follows:
>
> JStorm is a distributed and fault-tolerant real-time computation system.
> Inspired by Apache Storm, JStorm has been completely rewritten in Java
> and provides more enhanced features. JStorm has been widely used in
> many enterprise environments and proved robust and stable.
>
>
>
> Begun from 2012/2, we have released 11 versions till 2014/09/28. More
> than
> 10 companies are using it, including several leading companies such as
> Alibaba, Tencent, Baidu, Huawei, ZTE in China.
>
>
>
> Please refer to https://github.com/alibaba/jstorm for more details
>
>
>
>
>
> Regards
>
> Longda
>
>
>
>
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
答复: Could JStorm project collaborate with Storm?
Posted by "封仲淹(纪君祥)" <zh...@alibaba-inc.com>.
Sorry for no clarify why we raise the proposal replacing clojre-core with java-core.
The root cause why we start Jstorm is that the speed of Storm evolution is too slow for us. The Alibaba/Tencent’s application require our team/community moving faster and faster. If you take a deep look of Jstorm history, your will found we will release one version every month. Every version of Jstorm will provide new feature comparing last version, every new feature isn’t a simple job, we can also do it with Clojure(Clojure isn’t a hard thing for us), but it will take triple time to finish it comparing with implement it with java.
If we replace Clojure core with java core, I definite think we can move much faster than before. Alibaba and Tencent are one of the most biggest IT companies in the world, thousands of applications run on Jstorm. The fertilizer will drive Jstorm grow more powerful and stronger. In our long term plan, we can do more things than right now, no only just handle online streaming tuples with Java or Clojure, maybe SQL, maybe new Trident, we want to build the ecosystem like Spark.
If we plan to do switch core, we will fork a code branch under Storm, Jstorm-dev team will be <app:ds:responsible> responsible <app:ds:for> for merge all storm’s change to jstorm-core in a long term, after enough tests on the new branch, Jstorm-dev team will merge the branch to trunk and do the switch job.
The second option is we fork one project, named Jstorm(I am a little confused whether it is a normal incubator project under apache or just one subproject under storm). The Jstorm-dev team will take charge of maintain the Jstorm sub-project, merge all Storm’s changes into Jstorm time to time and evolve by himself. And wait user to select which project is more suitable for them. One year later, we check the result and judge whether we plan to merge or not. Once Storm community plan to do switch, we close the Jstorm subproject and merge into Storm.
@dev, what’s your opinion?
Here is a simple history of Jstorm, just list new features.
1. 0.9.0,
a) Support Nimbus HA
b) Redesign the scheduler, assign topology from CPU/Memory/Disk/Networker dimension, support user-define assignment
c) Rewrite Netty framework, supporting batch and sync mode.
2. 0.9.1
a) Support Classloader
b) Support group quota, different group has different quato
3. 0.9.2
a) Support resource isolation, import CGroups
b) Support Local mode
c) Redesign Logview
4. 0.9.3
a) Support run in Hadoop yarn/Aliyun Apsara, fix the bug under storm-on-yarn
5. 0.9.4
a) Rewrite netty code, support async mode and batch, improve performance much.
b) Add a new transaction programming mode
c) Add metrics and statics collection
d) Merge Alimama branch into Jstorm, support submit topology with different parameters
6. 0.9.5.x
a) Redesign scheduler arithmetic, assign topology basing worker slot, but support CGroups
b) Support two Netty mode, sync/async mode
7. 0.9.6.x
a) Redesign metrics/statics, support user-defined metrics and warning, web UI display new metrics
b) Add management of multiple-cluster to Web UI
c) Replace gson with fastjson
d) Support assign topology to user-defined supervisors
e) Add mechanism to wait tasks finish cleanup job.
In fact, there are more than one hundred little improvement, if you take a deep insight of Jstorm and Storm.
Regards
Longda
发件人: P. Taylor Goetz [mailto:ptgoetz@gmail.com]
发送时间: 2014年10月30日 4:17
收件人: dev@storm.apache.org
抄送: "封仲淹(纪君祥)"; jstorm_dev@googlegroups.com
主题: Re: Could JStorm project collaborate with Storm?
After taking some time to review the JStorm code and history of both projects, here are my thoughts:
Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The disparity in LOC is primarily due to the conciseness of Clojure compared to Java. Apache Storm's Clojure code consists mainly of implementations of interfaces defined in Java. This fact makes it possible to provide an implementation in Java itself, which is exactly what the JStorm project seems to have done. JStorm is a copy of the Apache Storm codebase, with the Clojure code being replaced with a Java implementation, with some additional changes.
The justification of the project centers around a distaste for Clojure, and a seeming lack of developers with Clojure skills [1]:
"Storm is wonderful product, but it is implemented with Clojure. We don't like clojure. it isn't a populate language, so in Alibaba, few people can fix the bug of storm. We are the first users of Storm since it has been open source. During using, we found several problem such as zeromq, zookeeper, performance.”
While I agree that to someone experienced with Java development, the first exposure to Clojure can be a bit jarring, I don’t think it is much of a barrier to entry. In my opinion, learning Clojure is a valuable investment in making oneself a more well-rounded developer. The technology ecosystem to which Storm belongs is becoming increasingly polyglot with respect to JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to that fact. As an example (perhaps not the best example), lets look at Kafka. If I didn’t like or want to learn Scala, would it make sense to write a clone in Java and attempt to keep feature parity and compatibility, or would it be better to bite the bullet and learn Scala? (Yes, I know it’s not the best example since Storm’s Clojure code is a subset of the codebase, and the way it’s designed makes it possible for alternative implementations). And in reality, someone from Alibaba had to at least learn enough Clojure to be able to replicate Storm’s Clojure implementation in java.
Looking at the history of JStorm [2][3], it appears a lot of work has gone into keeping feature parity with Apache Storm by copying in new code or reimplementing it. For example a kafka spout was added about two weeks ago [4], and Yahoo’s storm-on-yarn project was also recently copied in [5].
To me, this kind of work is rather counter-productive. If the Apache Storm community feels strongly that Storm should support multiple “core” implementations (i.e. one implemented in Clojure, and one implemented in Java), then it could be accommodated by making them pluggable and rearranging the codebase a bit. That being said, the Apache Storm community has invested a significant amount of time and effort maintaining and improving Storm’s Clojure “core.” One place this is most apparent, as Bobby mentioned, is the upcoming security release. To maintain feature parity with that release a “JStorm core” would have to be updated to implement that functionality, which is not a trivial undertaking. At some point as developers we need to ask ourselves “what's the point of that work?” There are a lot more people using Storm than developing internals. We need to make sure that both groups gain value from whatever we decide, and that we don’t sacrifice any value in the process.
So to answer the question in the subject line of the email thread: Could the JStorm project collaborate with Apache Storm? My answer would be a resounding “yes.”
Though the JStorm community is small in comparison to that of Apache Storm, it’s clear that the JStorm contributors have a solid grasp of Storm’s internals and would be welcomed by the Storm community. Reimplementing Storm’s core in Java no doubt provided insight into areas that might be improved.
That being said, replacing Storm’s Clojure-based core is a decision that should not be taken lightly, and would require the support of the Apache Storm PMC and entire community. A “dual-core” setup could be an option, but again could become a maintenance nightmare with little value added for end users. I’m open to discussing options further, and welcome input from other community members.
In short, I would rather see us work together than apart.
-Taylor
[1] https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a
[2] https://github.com/alibaba/jstorm/commits/master
[3] https://github.com/alibaba/jstorm/blob/master/history.md
[4] https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka
[5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn
On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
Sorry I am coming late to the conversation on this. Looking at the code for jstorm I am willing to entertain the idea of merging the two projects. I understand that by using java we can make the project more accessible to a larger number of developers, and that does seem to fit with the apache model of community over code. But I don't really want change just for change's sake. Look at kill_topology in both java:https://github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
and in clojure:https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
The java version is 61 lines of code and is not actually compatible with the clojure version because it just counts argument for the wait parameter instead of using a -w. The clojure version is 29, but the 15 of which are the apache header, so it is closer to 14 lines. Granted, to someone who does not know you have to work a bit to read it, but not that much.
My concerns would be two fold.
First is the effort involved in maintaining two separate code bases, sub project or otherwise, is a lot. I don't want to do it, and I assume you don't want to either, that is why you are talking about merging. We would need to decide which code to keep and which code to drop, and in all likelihood some of it will still be clojure. I also would not want to restrict people from using clojure for new work if they want to, just like we don't restrict java usage for new work.
Second is true feature parity. Storm is a moving target, I would not want to replace any code, even if it is faster, without verifying that it has feature parity, or it is a conscious decision that the missing features are not needed. Especially around the security branch, which is scheduled for the 0.10.0 release, but is not a part of jstorm at all.
- Bobby
From: 封仲淹(纪君祥) <zh...@alibaba-inc.com>
To: dev@storm.apache.org; tdunning@maprtech.com
Cc: jstorm_dev@googlegroups.com
Sent: Wednesday, October 29, 2014 4:27 AM
Subject: 答复: Could JStorm project collaborate with Storm?
Thanks for your suggestion.
@dev, Does anyone has proposal? please feel free to raise.
Thanks
Longda
-----邮件原件-----
发件人: Ted Dunning [mailto:tdunning@maprtech.com]
发送时间: 2014年10月25日 7:27
收件人: dev@storm.apache.org
主题: Re: Could JStorm project collaborate with Storm?
I just moved all of the non-Storm-dev addresses on this thread to bcc. If anybody is not on the Storm dev list and would like to continue to follow this conversation, you should subscribe to the storm dev list. We should not be spamming everybody else.
There is a standard Apache process to deal with this. JStorm can become an incubator project. Once JStorm demonstrates the ability to function as an Apache project, it can graduate either to a top-level project or to become part of a project like Storm.
[Longda]This will be a feasible solution. From the Apache side, it had better Jstorm can collaborate with Storm.
A second mechanism is that if the Storm group is interested, they can directly adopt the code as it is, possibly asking one or more of the contributors to become committers to help support this new code. That is a much harder thing to do than incubation since it is unusual for a project to be enthusiastic about supporting a large new codebase.
[Longda]I agree. It is a hard job.
On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
Hi Andy,
Thanks for your quick response. What you concern is reasonable and
understandable.
But maybe there is some misunderstanding on our proposal. According to
the current development situation in China, java core is indeed to
bring more contributors here.
That is reason why we begin to think if it is also a way for us to
provide our contribution to Storm project. It is true that it is a
risk to switch the core in a short period.
Actually, if possible, maybe we can consider to take JStorm as a
subproject of Storm(A branch or some other ways…). We will be
responsible for the maintain of it and try to propagate the features
from Storm to JStorm, and continue to develop our features and
improvements on this. You guys can try the new core and take a long
time to estimate if it is worth doing the switch in the future or
taking JStorm as official optional core of Storm for user.
Regards
Basti
*发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
*发送时间:* 2014年10月24日 0:24
*收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
ddas@hortonworks.com; m.ben.franklin@gmail.com;
benjamin.hindman@gmail.com
*抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
mlchen@iflytek.com; zeus@tencent.com
*主题:* Re: 答复: Could JStorm project collaborate with Storm?
Zhongyan:
Yahoo almost took your path 2 years ago. After some discussion with
Nathan, we decided to work with community on Storm, and some of us
learned Clojure quickly. We are very glad that we did that.
Under the assumption that Storm provides all features of JStorm, my
personal vote will be NO for your proposal.
- Storm has enjoyed its current success before Nathan built a very
solid core in Clojure. We should not replace that core until we are 100%
sure that alternative implementation is at least as good as our Clojure
implementation.
- Clojure has not prevent Storm attract contributors (currently 108).
We have contributors who write code in Clojure and Java. It doesn’t take
much time for one to be able to understand Clojure code.
- The convergence of streaming processing and batch processing will
occur at higher level of abstractions. Trident API, for example, is very
much similar to batch API such as Pig or Cascading.
- Including Jstorm under Storm will only create confusion to our user
community at this stage.
Why don’t you have discussion with 徐明明 to figure out an alternative
path for JStorm?
Thanks,
Andy
*From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
*Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
*Date: *Wednesday, October 22, 2014 at 11:37 PM
*To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
<fl...@infochimps.com>, "davidlao@microsoft.com"
<da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
m.ben.franklin@gmail.com" <m....@gmail.com>, "
benjamin.hindman@gmail.com" <be...@gmail.com>
*Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com"
<ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
"zeus@tencent.com" < zeus@tencent.com>
*Subject: *答复: Could JStorm project collaborate with Storm?
Andrew, Thanks for suggest.
The problem is that the core of Storm is implemented by Clojure, if it
is java, we are glad to merge all our commit into the Strom Trunk.
If the core of Storm is implemented by Java, I think the contributor
of Storm would be double.
May I put forward 2 proposal:
1. Could Jstorm be the son project of Storm?
2. Task a long time to change Storm-core as Jstorm-core.
a) If the core of Storm is implemented by Java, I think the
contributor of Storm would be double. Once one user found one bug,
maybe he is able to fix it by himself, in this method, Storm resolve
problem will be double
b) In the next one or two years, batch handling and stream handling
will merge into one solution, Spark/Flink are doing this job. If we
still use clojure, I am a little afraid that we can’t follow the steps
of other community and this direction.
Thanks
Longda
*发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <mailto:afeng@yahoo-inc.com%20%3cafeng@yahoo-inc.com%3e> <af...@yahoo-inc.com>]
*发送时间:* 2014年10月23日 13:31
*收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
ddas@hortonworks.com; m.ben.franklin@gmail.com;
benjamin.hindman@gmail.com
*抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
mlchen@iflytek.com; zeus@tencent.com
*主题:* Re: Could JStorm project collaborate with Storm?
Zhongyan:
I would suggest you identify a collection of key features in Jstorm,
and then propose pull requests to Storm.
Once these pull requests are accepted, you could then simply adopt Storm.
Andy
*From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
*Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
*Date: *Wednesday, October 22, 2014 at 8:23 PM
*To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>, "
davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
< ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
m.ben.franklin@gmail.com" <m....@gmail.com>, "
benjamin.hindman@gmail.com" <be...@gmail.com>
*Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com"
<ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
"zeus@tencent.com" < zeus@tencent.com>
*Subject: *Could JStorm project collaborate with Storm?
Hi all,
Firstly, congratulate Storm becoming the top project of Apache. Since
the first day of Storm opened source, it has become one of hottest
technologies in the BigData field and brought a new perspective to the
world, which has helped many companies resolve hundreds of requirements.
Since 2012/4, we have forked a project named "JStorm". At the
beginning time, we just wanted to implement Storm with Java. As time
goes on, more and more features have been added to JStorm. In 2013/9,
JStorm changed to be an open source project; from then on, more and
more people and companies join this project.
As JStorm is derived from Storm, and born in a great company named
Alibaba, which just successfully listed on the New York Stock Exchange
on Sept. 19, is a very open company. We also want to share our
technology with more people, and favor rapid evolution of streaming technology.
*So, could JStorm project collaborate with Storm? *
A little introduction for JStorm as follows:
JStorm is a distributed and fault-tolerant real-time computation system.
Inspired by Apache Storm, JStorm has been completely rewritten in Java
and provides more enhanced features. JStorm has been widely used in
many enterprise environments and proved robust and stable.
Begun from 2012/2, we have released 11 versions till 2014/09/28. More
than
10 companies are using it, including several leading companies such as
Alibaba, Tencent, Baidu, Huawei, ZTE in China.
Please refer to https://github.com/alibaba/jstorm for more details
Regards
Longda
Re: Could JStorm project collaborate with Storm?
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
After taking some time to review the JStorm code and history of both projects, here are my thoughts:
Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The disparity in LOC is primarily due to the conciseness of Clojure compared to Java. Apache Storm's Clojure code consists mainly of implementations of interfaces defined in Java. This fact makes it possible to provide an implementation in Java itself, which is exactly what the JStorm project seems to have done. JStorm is a copy of the Apache Storm codebase, with the Clojure code being replaced with a Java implementation, with some additional changes.
The justification of the project centers around a distaste for Clojure, and a seeming lack of developers with Clojure skills [1]:
"Storm is wonderful product, but it is implemented with Clojure. We don't like clojure. it isn't a populate language, so in Alibaba, few people can fix the bug of storm. We are the first users of Storm since it has been open source. During using, we found several problem such as zeromq, zookeeper, performance.”
While I agree that to someone experienced with Java development, the first exposure to Clojure can be a bit jarring, I don’t think it is much of a barrier to entry. In my opinion, learning Clojure is a valuable investment in making oneself a more well-rounded developer. The technology ecosystem to which Storm belongs is becoming increasingly polyglot with respect to JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to that fact. As an example (perhaps not the best example), lets look at Kafka. If I didn’t like or want to learn Scala, would it make sense to write a clone in Java and attempt to keep feature parity and compatibility, or would it be better to bite the bullet and learn Scala? (Yes, I know it’s not the best example since Storm’s Clojure code is a subset of the codebase, and the way it’s designed makes it possible for alternative implementations). And in reality, someone from Alibaba had to at least learn enough Clojure to be able to replicate Storm’s Clojure implementation in java.
Looking at the history of JStorm [2][3], it appears a lot of work has gone into keeping feature parity with Apache Storm by copying in new code or reimplementing it. For example a kafka spout was added about two weeks ago [4], and Yahoo’s storm-on-yarn project was also recently copied in [5].
To me, this kind of work is rather counter-productive. If the Apache Storm community feels strongly that Storm should support multiple “core” implementations (i.e. one implemented in Clojure, and one implemented in Java), then it could be accommodated by making them pluggable and rearranging the codebase a bit. That being said, the Apache Storm community has invested a significant amount of time and effort maintaining and improving Storm’s Clojure “core.” One place this is most apparent, as Bobby mentioned, is the upcoming security release. To maintain feature parity with that release a “JStorm core” would have to be updated to implement that functionality, which is not a trivial undertaking. At some point as developers we need to ask ourselves “what's the point of that work?” There are a lot more people using Storm than developing internals. We need to make sure that both groups gain value from whatever we decide, and that we don’t sacrifice any value in the process.
So to answer the question in the subject line of the email thread: Could the JStorm project collaborate with Apache Storm? My answer would be a resounding “yes.”
Though the JStorm community is small in comparison to that of Apache Storm, it’s clear that the JStorm contributors have a solid grasp of Storm’s internals and would be welcomed by the Storm community. Reimplementing Storm’s core in Java no doubt provided insight into areas that might be improved.
That being said, replacing Storm’s Clojure-based core is a decision that should not be taken lightly, and would require the support of the Apache Storm PMC and entire community. A “dual-core” setup could be an option, but again could become a maintenance nightmare with little value added for end users. I’m open to discussing options further, and welcome input from other community members.
In short, I would rather see us work together than apart.
-Taylor
[1] https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a
[2] https://github.com/alibaba/jstorm/commits/master
[3] https://github.com/alibaba/jstorm/blob/master/history.md
[4] https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka
[5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn
On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> Sorry I am coming late to the conversation on this. Looking at the code for jstorm I am willing to entertain the idea of merging the two projects. I understand that by using java we can make the project more accessible to a larger number of developers, and that does seem to fit with the apache model of community over code. But I don't really want change just for change's sake. Look at kill_topology in both java:https://github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
>
> and in clojure:https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
>
> The java version is 61 lines of code and is not actually compatible with the clojure version because it just counts argument for the wait parameter instead of using a -w. The clojure version is 29, but the 15 of which are the apache header, so it is closer to 14 lines. Granted, to someone who does not know you have to work a bit to read it, but not that much.
> My concerns would be two fold.
> First is the effort involved in maintaining two separate code bases, sub project or otherwise, is a lot. I don't want to do it, and I assume you don't want to either, that is why you are talking about merging. We would need to decide which code to keep and which code to drop, and in all likelihood some of it will still be clojure. I also would not want to restrict people from using clojure for new work if they want to, just like we don't restrict java usage for new work.
> Second is true feature parity. Storm is a moving target, I would not want to replace any code, even if it is faster, without verifying that it has feature parity, or it is a conscious decision that the missing features are not needed. Especially around the security branch, which is scheduled for the 0.10.0 release, but is not a part of jstorm at all.
> - Bobby
>
> From: 封仲淹(纪君祥) <zh...@alibaba-inc.com>
> To: dev@storm.apache.org; tdunning@maprtech.com
> Cc: jstorm_dev@googlegroups.com
> Sent: Wednesday, October 29, 2014 4:27 AM
> Subject: 答复: Could JStorm project collaborate with Storm?
>
> Thanks for your suggestion.
>
> @dev, Does anyone has proposal? please feel free to raise.
>
>
> Thanks
> Longda
>
> -----邮件原件-----
> 发件人: Ted Dunning [mailto:tdunning@maprtech.com]
> 发送时间: 2014年10月25日 7:27
> 收件人: dev@storm.apache.org
> 主题: Re: Could JStorm project collaborate with Storm?
>
> I just moved all of the non-Storm-dev addresses on this thread to bcc. If anybody is not on the Storm dev list and would like to continue to follow this conversation, you should subscribe to the storm dev list. We should not be spamming everybody else.
>
>
> There is a standard Apache process to deal with this. JStorm can become an incubator project. Once JStorm demonstrates the ability to function as an Apache project, it can graduate either to a top-level project or to become part of a project like Storm.
> [Longda]This will be a feasible solution. From the Apache side, it had better Jstorm can collaborate with Storm.
>
> A second mechanism is that if the Storm group is interested, they can directly adopt the code as it is, possibly asking one or more of the contributors to become committers to help support this new code. That is a much harder thing to do than incubation since it is unusual for a project to be enthusiastic about supporting a large new codebase.
> [Longda]I agree. It is a hard job.
>
>
>
>
>
>
> On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
>
>> Hi Andy,
>>
>>
>>
>> Thanks for your quick response. What you concern is reasonable and
>> understandable.
>>
>>
>>
>> But maybe there is some misunderstanding on our proposal. According to
>> the current development situation in China, java core is indeed to
>> bring more contributors here.
>>
>> That is reason why we begin to think if it is also a way for us to
>> provide our contribution to Storm project. It is true that it is a
>> risk to switch the core in a short period.
>>
>> Actually, if possible, maybe we can consider to take JStorm as a
>> subproject of Storm(A branch or some other ways…). We will be
>> responsible for the maintain of it and try to propagate the features
>> from Storm to JStorm, and continue to develop our features and
>> improvements on this. You guys can try the new core and take a long
>> time to estimate if it is worth doing the switch in the future or
>> taking JStorm as official optional core of Storm for user.
>>
>>
>>
>> Regards
>>
>> Basti
>>
>>
>>
>> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
>> *发送时间:* 2014年10月24日 0:24
>> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
>> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
>> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
>> ddas@hortonworks.com; m.ben.franklin@gmail.com;
>> benjamin.hindman@gmail.com
>> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
>> mlchen@iflytek.com; zeus@tencent.com
>> *主题:* Re: 答复: Could JStorm project collaborate with Storm?
>>
>>
>>
>> Zhongyan:
>>
>>
>>
>> Yahoo almost took your path 2 years ago. After some discussion with
>> Nathan, we decided to work with community on Storm, and some of us
>> learned Clojure quickly. We are very glad that we did that.
>>
>>
>>
>> Under the assumption that Storm provides all features of JStorm, my
>> personal vote will be NO for your proposal.
>>
>> - Storm has enjoyed its current success before Nathan built a very
>> solid core in Clojure. We should not replace that core until we are 100%
>> sure that alternative implementation is at least as good as our Clojure
>> implementation.
>> - Clojure has not prevent Storm attract contributors (currently 108).
>> We have contributors who write code in Clojure and Java. It doesn’t take
>> much time for one to be able to understand Clojure code.
>> - The convergence of streaming processing and batch processing will
>> occur at higher level of abstractions. Trident API, for example, is very
>> much similar to batch API such as Pig or Cascading.
>> - Including Jstorm under Storm will only create confusion to our user
>> community at this stage.
>>
>>
>>
>> Why don’t you have discussion with 徐明明 to figure out an alternative
>> path for JStorm?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Andy
>>
>>
>>
>> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
>> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
>> *Date: *Wednesday, October 22, 2014 at 11:37 PM
>> *To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
>> nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
>> jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
>> <fl...@infochimps.com>, "davidlao@microsoft.com"
>> <da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
>> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
>> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
>> m.ben.franklin@gmail.com" <m....@gmail.com>, "
>> benjamin.hindman@gmail.com" <be...@gmail.com>
>> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
>> <sc...@vipshop.com>, "yannianmu@tencent.com"
>> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
>> "zeus@tencent.com" < zeus@tencent.com>
>> *Subject: *答复: Could JStorm project collaborate with Storm?
>>
>>
>>
>> Andrew, Thanks for suggest.
>>
>>
>>
>> The problem is that the core of Storm is implemented by Clojure, if it
>> is java, we are glad to merge all our commit into the Strom Trunk.
>>
>> If the core of Storm is implemented by Java, I think the contributor
>> of Storm would be double.
>>
>>
>>
>>
>>
>> May I put forward 2 proposal:
>>
>> 1. Could Jstorm be the son project of Storm?
>>
>> 2. Task a long time to change Storm-core as Jstorm-core.
>>
>> a) If the core of Storm is implemented by Java, I think the
>> contributor of Storm would be double. Once one user found one bug,
>> maybe he is able to fix it by himself, in this method, Storm resolve
>> problem will be double
>>
>> b) In the next one or two years, batch handling and stream handling
>> will merge into one solution, Spark/Flink are doing this job. If we
>> still use clojure, I am a little afraid that we can’t follow the steps
>> of other community and this direction.
>>
>>
>>
>>
>>
>> Thanks
>>
>> Longda
>>
>>
>>
>> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
>> *发送时间:* 2014年10月23日 13:31
>> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
>> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
>> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
>> ddas@hortonworks.com; m.ben.franklin@gmail.com;
>> benjamin.hindman@gmail.com
>> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
>> mlchen@iflytek.com; zeus@tencent.com
>> *主题:* Re: Could JStorm project collaborate with Storm?
>>
>>
>>
>> Zhongyan:
>>
>>
>>
>> I would suggest you identify a collection of key features in Jstorm,
>> and then propose pull requests to Storm.
>>
>> Once these pull requests are accepted, you could then simply adopt Storm.
>>
>>
>>
>> Andy
>>
>>
>>
>> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
>> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
>> *Date: *Wednesday, October 22, 2014 at 8:23 PM
>> *To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
>> mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
>> Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>, "
>> davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
>> < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
>> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
>> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
>> m.ben.franklin@gmail.com" <m....@gmail.com>, "
>> benjamin.hindman@gmail.com" <be...@gmail.com>
>> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
>> <sc...@vipshop.com>, "yannianmu@tencent.com"
>> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
>> "zeus@tencent.com" < zeus@tencent.com>
>> *Subject: *Could JStorm project collaborate with Storm?
>>
>>
>>
>> Hi all,
>>
>>
>>
>> Firstly, congratulate Storm becoming the top project of Apache. Since
>> the first day of Storm opened source, it has become one of hottest
>> technologies in the BigData field and brought a new perspective to the
>> world, which has helped many companies resolve hundreds of requirements.
>>
>>
>>
>> Since 2012/4, we have forked a project named "JStorm". At the
>> beginning time, we just wanted to implement Storm with Java. As time
>> goes on, more and more features have been added to JStorm. In 2013/9,
>> JStorm changed to be an open source project; from then on, more and
>> more people and companies join this project.
>>
>>
>>
>> As JStorm is derived from Storm, and born in a great company named
>> Alibaba, which just successfully listed on the New York Stock Exchange
>> on Sept. 19, is a very open company. We also want to share our
>> technology with more people, and favor rapid evolution of streaming technology.
>>
>>
>>
>> *So, could JStorm project collaborate with Storm? *
>>
>>
>>
>> A little introduction for JStorm as follows:
>>
>> JStorm is a distributed and fault-tolerant real-time computation system.
>> Inspired by Apache Storm, JStorm has been completely rewritten in Java
>> and provides more enhanced features. JStorm has been widely used in
>> many enterprise environments and proved robust and stable.
>>
>>
>>
>> Begun from 2012/2, we have released 11 versions till 2014/09/28. More
>> than
>> 10 companies are using it, including several leading companies such as
>> Alibaba, Tencent, Baidu, Huawei, ZTE in China.
>>
>>
>>
>> Please refer to https://github.com/alibaba/jstorm for more details
>>
>>
>>
>>
>>
>> Regards
>>
>> Longda
>>
>
>
Re: 答复: Could JStorm project collaborate with Storm?
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Sorry I am coming late to the conversation on this. Looking at the code for jstorm I am willing to entertain the idea of merging the two projects. I understand that by using java we can make the project more accessible to a larger number of developers, and that does seem to fit with the apache model of community over code. But I don't really want change just for change's sake. Look at kill_topology in both java:https://github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java
and in clojure:https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj
The java version is 61 lines of code and is not actually compatible with the clojure version because it just counts argument for the wait parameter instead of using a -w. The clojure version is 29, but the 15 of which are the apache header, so it is closer to 14 lines. Granted, to someone who does not know you have to work a bit to read it, but not that much.
My concerns would be two fold.
First is the effort involved in maintaining two separate code bases, sub project or otherwise, is a lot. I don't want to do it, and I assume you don't want to either, that is why you are talking about merging. We would need to decide which code to keep and which code to drop, and in all likelihood some of it will still be clojure. I also would not want to restrict people from using clojure for new work if they want to, just like we don't restrict java usage for new work.
Second is true feature parity. Storm is a moving target, I would not want to replace any code, even if it is faster, without verifying that it has feature parity, or it is a conscious decision that the missing features are not needed. Especially around the security branch, which is scheduled for the 0.10.0 release, but is not a part of jstorm at all.
- Bobby
From: 封仲淹(纪君祥) <zh...@alibaba-inc.com>
To: dev@storm.apache.org; tdunning@maprtech.com
Cc: jstorm_dev@googlegroups.com
Sent: Wednesday, October 29, 2014 4:27 AM
Subject: 答复: Could JStorm project collaborate with Storm?
Thanks for your suggestion.
@dev, Does anyone has proposal? please feel free to raise.
Thanks
Longda
-----邮件原件-----
发件人: Ted Dunning [mailto:tdunning@maprtech.com]
发送时间: 2014年10月25日 7:27
收件人: dev@storm.apache.org
主题: Re: Could JStorm project collaborate with Storm?
I just moved all of the non-Storm-dev addresses on this thread to bcc. If anybody is not on the Storm dev list and would like to continue to follow this conversation, you should subscribe to the storm dev list. We should not be spamming everybody else.
There is a standard Apache process to deal with this. JStorm can become an incubator project. Once JStorm demonstrates the ability to function as an Apache project, it can graduate either to a top-level project or to become part of a project like Storm.
[Longda]This will be a feasible solution. From the Apache side, it had better Jstorm can collaborate with Storm.
A second mechanism is that if the Storm group is interested, they can directly adopt the code as it is, possibly asking one or more of the contributors to become committers to help support this new code. That is a much harder thing to do than incubation since it is unusual for a project to be enthusiastic about supporting a large new codebase.
[Longda]I agree. It is a hard job.
On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
> Hi Andy,
>
>
>
> Thanks for your quick response. What you concern is reasonable and
> understandable.
>
>
>
> But maybe there is some misunderstanding on our proposal. According to
> the current development situation in China, java core is indeed to
> bring more contributors here.
>
> That is reason why we begin to think if it is also a way for us to
> provide our contribution to Storm project. It is true that it is a
> risk to switch the core in a short period.
>
> Actually, if possible, maybe we can consider to take JStorm as a
> subproject of Storm(A branch or some other ways…). We will be
> responsible for the maintain of it and try to propagate the features
> from Storm to JStorm, and continue to develop our features and
> improvements on this. You guys can try the new core and take a long
> time to estimate if it is worth doing the switch in the future or
> taking JStorm as official optional core of Storm for user.
>
>
>
> Regards
>
> Basti
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月24日 0:24
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: 答复: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> Yahoo almost took your path 2 years ago. After some discussion with
> Nathan, we decided to work with community on Storm, and some of us
> learned Clojure quickly. We are very glad that we did that.
>
>
>
> Under the assumption that Storm provides all features of JStorm, my
> personal vote will be NO for your proposal.
>
> - Storm has enjoyed its current success before Nathan built a very
> solid core in Clojure. We should not replace that core until we are 100%
> sure that alternative implementation is at least as good as our Clojure
> implementation.
> - Clojure has not prevent Storm attract contributors (currently 108).
> We have contributors who write code in Clojure and Java. It doesn’t take
> much time for one to be able to understand Clojure code.
> - The convergence of streaming processing and batch processing will
> occur at higher level of abstractions. Trident API, for example, is very
> much similar to batch API such as Pig or Cascading.
> - Including Jstorm under Storm will only create confusion to our user
> community at this stage.
>
>
>
> Why don’t you have discussion with 徐明明 to figure out an alternative
> path for JStorm?
>
>
>
> Thanks,
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 11:37 PM
> *To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
> nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
> jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
> <fl...@infochimps.com>, "davidlao@microsoft.com"
> <da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *答复: Could JStorm project collaborate with Storm?
>
>
>
> Andrew, Thanks for suggest.
>
>
>
> The problem is that the core of Storm is implemented by Clojure, if it
> is java, we are glad to merge all our commit into the Strom Trunk.
>
> If the core of Storm is implemented by Java, I think the contributor
> of Storm would be double.
>
>
>
>
>
> May I put forward 2 proposal:
>
> 1. Could Jstorm be the son project of Storm?
>
> 2. Task a long time to change Storm-core as Jstorm-core.
>
> a) If the core of Storm is implemented by Java, I think the
> contributor of Storm would be double. Once one user found one bug,
> maybe he is able to fix it by himself, in this method, Storm resolve
> problem will be double
>
> b) In the next one or two years, batch handling and stream handling
> will merge into one solution, Spark/Flink are doing this job. If we
> still use clojure, I am a little afraid that we can’t follow the steps
> of other community and this direction.
>
>
>
>
>
> Thanks
>
> Longda
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月23日 13:31
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> I would suggest you identify a collection of key features in Jstorm,
> and then propose pull requests to Storm.
>
> Once these pull requests are accepted, you could then simply adopt Storm.
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 8:23 PM
> *To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
> mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
> Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>, "
> davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
> < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *Could JStorm project collaborate with Storm?
>
>
>
> Hi all,
>
>
>
> Firstly, congratulate Storm becoming the top project of Apache. Since
> the first day of Storm opened source, it has become one of hottest
> technologies in the BigData field and brought a new perspective to the
> world, which has helped many companies resolve hundreds of requirements.
>
>
>
> Since 2012/4, we have forked a project named "JStorm". At the
> beginning time, we just wanted to implement Storm with Java. As time
> goes on, more and more features have been added to JStorm. In 2013/9,
> JStorm changed to be an open source project; from then on, more and
> more people and companies join this project.
>
>
>
> As JStorm is derived from Storm, and born in a great company named
> Alibaba, which just successfully listed on the New York Stock Exchange
> on Sept. 19, is a very open company. We also want to share our
> technology with more people, and favor rapid evolution of streaming technology.
>
>
>
> *So, could JStorm project collaborate with Storm? *
>
>
>
> A little introduction for JStorm as follows:
>
> JStorm is a distributed and fault-tolerant real-time computation system.
> Inspired by Apache Storm, JStorm has been completely rewritten in Java
> and provides more enhanced features. JStorm has been widely used in
> many enterprise environments and proved robust and stable.
>
>
>
> Begun from 2012/2, we have released 11 versions till 2014/09/28. More
> than
> 10 companies are using it, including several leading companies such as
> Alibaba, Tencent, Baidu, Huawei, ZTE in China.
>
>
>
> Please refer to https://github.com/alibaba/jstorm for more details
>
>
>
>
>
> Regards
>
> Longda
>
答复: Could JStorm project collaborate with Storm?
Posted by "封仲淹(纪君祥)" <zh...@alibaba-inc.com>.
Thanks for your suggestion.
@dev, Does anyone has proposal? please feel free to raise.
Thanks
Longda
-----邮件原件-----
发件人: Ted Dunning [mailto:tdunning@maprtech.com]
发送时间: 2014年10月25日 7:27
收件人: dev@storm.apache.org
主题: Re: Could JStorm project collaborate with Storm?
I just moved all of the non-Storm-dev addresses on this thread to bcc. If anybody is not on the Storm dev list and would like to continue to follow this conversation, you should subscribe to the storm dev list. We should not be spamming everybody else.
There is a standard Apache process to deal with this. JStorm can become an incubator project. Once JStorm demonstrates the ability to function as an Apache project, it can graduate either to a top-level project or to become part of a project like Storm.
[Longda]This will be a feasible solution. From the Apache side, it had better Jstorm can collaborate with Storm.
A second mechanism is that if the Storm group is interested, they can directly adopt the code as it is, possibly asking one or more of the contributors to become committers to help support this new code. That is a much harder thing to do than incubation since it is unusual for a project to be enthusiastic about supporting a large new codebase.
[Longda]I agree. It is a hard job.
On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
> Hi Andy,
>
>
>
> Thanks for your quick response. What you concern is reasonable and
> understandable.
>
>
>
> But maybe there is some misunderstanding on our proposal. According to
> the current development situation in China, java core is indeed to
> bring more contributors here.
>
> That is reason why we begin to think if it is also a way for us to
> provide our contribution to Storm project. It is true that it is a
> risk to switch the core in a short period.
>
> Actually, if possible, maybe we can consider to take JStorm as a
> subproject of Storm(A branch or some other ways…). We will be
> responsible for the maintain of it and try to propagate the features
> from Storm to JStorm, and continue to develop our features and
> improvements on this. You guys can try the new core and take a long
> time to estimate if it is worth doing the switch in the future or
> taking JStorm as official optional core of Storm for user.
>
>
>
> Regards
>
> Basti
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月24日 0:24
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: 答复: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> Yahoo almost took your path 2 years ago. After some discussion with
> Nathan, we decided to work with community on Storm, and some of us
> learned Clojure quickly. We are very glad that we did that.
>
>
>
> Under the assumption that Storm provides all features of JStorm, my
> personal vote will be NO for your proposal.
>
> - Storm has enjoyed its current success before Nathan built a very
> solid core in Clojure. We should not replace that core until we are 100%
> sure that alternative implementation is at least as good as our Clojure
> implementation.
> - Clojure has not prevent Storm attract contributors (currently 108).
> We have contributors who write code in Clojure and Java. It doesn’t take
> much time for one to be able to understand Clojure code.
> - The convergence of streaming processing and batch processing will
> occur at higher level of abstractions. Trident API, for example, is very
> much similar to batch API such as Pig or Cascading.
> - Including Jstorm under Storm will only create confusion to our user
> community at this stage.
>
>
>
> Why don’t you have discussion with 徐明明 to figure out an alternative
> path for JStorm?
>
>
>
> Thanks,
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 11:37 PM
> *To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
> nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
> jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com"
> <fl...@infochimps.com>, "davidlao@microsoft.com"
> <da...@microsoft.com>, "ptgoetz@gmail.com" < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *答复: Could JStorm project collaborate with Storm?
>
>
>
> Andrew, Thanks for suggest.
>
>
>
> The problem is that the core of Storm is implemented by Clojure, if it
> is java, we are glad to merge all our commit into the Strom Trunk.
>
> If the core of Storm is implemented by Java, I think the contributor
> of Storm would be double.
>
>
>
>
>
> May I put forward 2 proposal:
>
> 1. Could Jstorm be the son project of Storm?
>
> 2. Task a long time to change Storm-core as Jstorm-core.
>
> a) If the core of Storm is implemented by Java, I think the
> contributor of Storm would be double. Once one user found one bug,
> maybe he is able to fix it by himself, in this method, Storm resolve
> problem will be double
>
> b) In the next one or two years, batch handling and stream handling
> will merge into one solution, Spark/Flink are doing this job. If we
> still use clojure, I am a little afraid that we can’t follow the steps
> of other community and this direction.
>
>
>
>
>
> Thanks
>
> Longda
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月23日 13:31
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com;
> benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> I would suggest you identify a collection of key features in Jstorm,
> and then propose pull requests to Storm.
>
> Once these pull requests are accepted, you could then simply adopt Storm.
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 8:23 PM
> *To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
> mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy
> Feng < afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>, "
> davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
> < ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com"
> <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>,
> "zeus@tencent.com" < zeus@tencent.com>
> *Subject: *Could JStorm project collaborate with Storm?
>
>
>
> Hi all,
>
>
>
> Firstly, congratulate Storm becoming the top project of Apache. Since
> the first day of Storm opened source, it has become one of hottest
> technologies in the BigData field and brought a new perspective to the
> world, which has helped many companies resolve hundreds of requirements.
>
>
>
> Since 2012/4, we have forked a project named "JStorm". At the
> beginning time, we just wanted to implement Storm with Java. As time
> goes on, more and more features have been added to JStorm. In 2013/9,
> JStorm changed to be an open source project; from then on, more and
> more people and companies join this project.
>
>
>
> As JStorm is derived from Storm, and born in a great company named
> Alibaba, which just successfully listed on the New York Stock Exchange
> on Sept. 19, is a very open company. We also want to share our
> technology with more people, and favor rapid evolution of streaming technology.
>
>
>
> *So, could JStorm project collaborate with Storm? *
>
>
>
> A little introduction for JStorm as follows:
>
> JStorm is a distributed and fault-tolerant real-time computation system.
> Inspired by Apache Storm, JStorm has been completely rewritten in Java
> and provides more enhanced features. JStorm has been widely used in
> many enterprise environments and proved robust and stable.
>
>
>
> Begun from 2012/2, we have released 11 versions till 2014/09/28. More
> than
> 10 companies are using it, including several leading companies such as
> Alibaba, Tencent, Baidu, Huawei, ZTE in China.
>
>
>
> Please refer to https://github.com/alibaba/jstorm for more details
>
>
>
>
>
> Regards
>
> Longda
>
Re: Could JStorm project collaborate with Storm?
Posted by Ted Dunning <td...@maprtech.com>.
I just moved all of the non-Storm-dev addresses on this thread to bcc. If
anybody is not on the Storm dev list and would like to continue to follow
this conversation, you should subscribe to the storm dev list. We should
not be spamming everybody else.
There is a standard Apache process to deal with this. JStorm can become an
incubator project. Once JStorm demonstrates the ability to function as an
Apache project, it can graduate either to a top-level project or to become
part of a project like Storm.
A second mechanism is that if the Storm group is interested, they can
directly adopt the code as it is, possibly asking one or more of the
contributors to become committers to help support this new code. That is a
much harder thing to do than incubation since it is unusual for a project
to be enthusiastic about supporting a large new codebase.
On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <ba...@alibaba-inc.com> wrote:
> Hi Andy,
>
>
>
> Thanks for your quick response. What you concern is reasonable and
> understandable.
>
>
>
> But maybe there is some misunderstanding on our proposal. According to the
> current development situation in China, java core is indeed to bring more
> contributors here.
>
> That is reason why we begin to think if it is also a way for us to provide
> our contribution to Storm project. It is true that it is a risk to switch
> the core in a short period.
>
> Actually, if possible, maybe we can consider to take JStorm as a
> subproject of Storm(A branch or some other ways…). We will be responsible
> for the maintain of it and try to propagate the features from Storm to
> JStorm, and continue to develop our features and improvements on this. You
> guys can try the new core and take a long time to estimate if it is worth
> doing the switch in the future or taking JStorm as official optional core
> of Storm for user.
>
>
>
> Regards
>
> Basti
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月24日 0:24
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com; benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: 答复: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> Yahoo almost took your path 2 years ago. After some discussion with
> Nathan, we decided to work with community on Storm, and some of us learned
> Clojure quickly. We are very glad that we did that.
>
>
>
> Under the assumption that Storm provides all features of JStorm, my
> personal vote will be NO for your proposal.
>
> - Storm has enjoyed its current success before Nathan built a very
> solid core in Clojure. We should not replace that core until we are 100%
> sure that alternative implementation is at least as good as our Clojure
> implementation.
> - Clojure has not prevent Storm attract contributors (currently 108).
> We have contributors who write code in Clojure and Java. It doesn’t take
> much time for one to be able to understand Clojure code.
> - The convergence of streaming processing and batch processing will
> occur at higher level of abstractions. Trident API, for example, is very
> much similar to batch API such as Pig or Cascading.
> - Including Jstorm under Storm will only create confusion to our user
> community at this stage.
>
>
>
> Why don’t you have discussion with 徐明明 to figure out an alternative path
> for JStorm?
>
>
>
> Thanks,
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 11:37 PM
> *To: *Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com" <
> nathan@nathanmarz.com>, "徐明明(护城)" <mi...@alibaba-inc.com>, "
> jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com" <fl...@infochimps.com>,
> "davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com" <
> ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com" <ya...@tencent.com>,
> "mlchen@iflytek.com" <ml...@iflytek.com>, "zeus@tencent.com" <
> zeus@tencent.com>
> *Subject: *答复: Could JStorm project collaborate with Storm?
>
>
>
> Andrew, Thanks for suggest.
>
>
>
> The problem is that the core of Storm is implemented by Clojure, if it is
> java, we are glad to merge all our commit into the Strom Trunk.
>
> If the core of Storm is implemented by Java, I think the contributor of
> Storm would be double.
>
>
>
>
>
> May I put forward 2 proposal:
>
> 1. Could Jstorm be the son project of Storm?
>
> 2. Task a long time to change Storm-core as Jstorm-core.
>
> a) If the core of Storm is implemented by Java, I think the
> contributor of Storm would be double. Once one user found one bug, maybe he
> is able to fix it by himself, in this method, Storm resolve problem will be
> double
>
> b) In the next one or two years, batch handling and stream handling
> will merge into one solution, Spark/Flink are doing this job. If we still
> use clojure, I am a little afraid that we can’t follow the steps of other
> community and this direction.
>
>
>
>
>
> Thanks
>
> Longda
>
>
>
> *发件人:* Andrew Feng [mailto:afeng@yahoo-inc.com <af...@yahoo-inc.com>]
> *发送时间:* 2014年10月23日 13:31
> *收件人:* 封仲淹(纪君祥); nathan@nathanmarz.com; 徐明明(护城); jason@cvk.ca;
> flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com;
> cutting@apache.org; tdunning@maprtech.com; arvind@apache.org;
> ddas@hortonworks.com; m.ben.franklin@gmail.com; benjamin.hindman@gmail.com
> *抄送:* aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com;
> mlchen@iflytek.com; zeus@tencent.com
> *主题:* Re: Could JStorm project collaborate with Storm?
>
>
>
> Zhongyan:
>
>
>
> I would suggest you identify a collection of key features in Jstorm, and
> then propose pull requests to Storm.
>
> Once these pull requests are accepted, you could then simply adopt Storm.
>
>
>
> Andy
>
>
>
> *From: *"封仲淹 (纪君祥)" <zh...@alibaba-inc.com>
> *Reply-To: *"封仲淹(纪君祥)" <zh...@alibaba-inc.com>
> *Date: *Wednesday, October 22, 2014 at 8:23 PM
> *To: *"nathan@nathanmarz.com" <na...@nathanmarz.com>, "徐明明(护城)" <
> mingming.xumm@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy Feng <
> afeng@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>, "
> davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com" <
> ptgoetz@gmail.com>, Doug Cutting <cu...@apache.org>, "
> tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org" <
> arvind@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "
> m.ben.franklin@gmail.com" <m....@gmail.com>, "
> benjamin.hindman@gmail.com" <be...@gmail.com>
> *Cc: *aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
> <sc...@vipshop.com>, "yannianmu@tencent.com" <ya...@tencent.com>,
> "mlchen@iflytek.com" <ml...@iflytek.com>, "zeus@tencent.com" <
> zeus@tencent.com>
> *Subject: *Could JStorm project collaborate with Storm?
>
>
>
> Hi all,
>
>
>
> Firstly, congratulate Storm becoming the top project of Apache. Since the
> first day of Storm opened source, it has become one of hottest technologies
> in the BigData field and brought a new perspective to the world, which has
> helped many companies resolve hundreds of requirements.
>
>
>
> Since 2012/4, we have forked a project named "JStorm". At the beginning
> time, we just wanted to implement Storm with Java. As time goes on, more
> and more features have been added to JStorm. In 2013/9, JStorm changed to
> be an open source project; from then on, more and more people and companies
> join this project.
>
>
>
> As JStorm is derived from Storm, and born in a great company named
> Alibaba, which just successfully listed on the New York Stock Exchange on
> Sept. 19, is a very open company. We also want to share our technology with
> more people, and favor rapid evolution of streaming technology.
>
>
>
> *So, could JStorm project collaborate with Storm? *
>
>
>
> A little introduction for JStorm as follows:
>
> JStorm is a distributed and fault-tolerant real-time computation system.
> Inspired by Apache Storm, JStorm has been completely rewritten in Java and
> provides more enhanced features. JStorm has been widely used in many
> enterprise environments and proved robust and stable.
>
>
>
> Begun from 2012/2, we have released 11 versions till 2014/09/28. More than
> 10 companies are using it, including several leading companies such as
> Alibaba, Tencent, Baidu, Huawei, ZTE in China.
>
>
>
> Please refer to https://github.com/alibaba/jstorm for more details
>
>
>
>
>
> Regards
>
> Longda
>
RE: Could JStorm project collaborate with Storm?
Posted by "Lian, Li" <ll...@ebay.com>.
Dear all,
This seems an interesting discussion and I want to just give my 2 cents here.
1. I disagree with this statement 'According to the current development situation in China, java core is indeed to bring more contributors here'. In recent QCon in Shanghai, I saw folks from China local internet companies did quite a lot contribution to Storm project without requiring it to have a Java core. Also, technically, with the minimum core code in Clojure and right interfaces in place, it's easy to extend Storm features and functions with Java.
2. If the proposal of using JStrom replacing Storm is accepted, then the community will spend a lot time migrating and the global users of Storm needs to do a lot regression testing to verify no large issues, no matter how good the unit testing will be done. The impact is too large for Storm user community to afford.
3. I firm believe the combination of some Java together with uprising JVM compatible language like Clojure and Scala, is the future for complex distributed application development. There are quite some successful example of new open source projects like Kafka and Spark, both developed using both Scala and expose APIs and extensions in Java. If we are going to replace all non-Java projects with a Java version, will it be a worth our precious time?
4. I understand Alibaba has certain special requirements and some implementation to satisfy the specific needs of shared cluster resource management, etc. As JStorm already has implementation of these kind of requirements, I suggest that folks from Alibaba to start some good communication with Storm committers and ask them to update just the extension points of Storm, to enable specific extension, but without replacing the core entirely.
Lex
eBay
-----Original Message-----
From: ����(���G) [mailto:basti.lj@alibaba-inc.com]
Sent: 2014��10��24�� 11:20
To: afeng@yahoo-inc.com; nathan@nathanmarz.com; ������(����); jason@cvk.ca; flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com; cutting@apache.org; tdunning@maprtech.com; arvind@apache.org; ddas@hortonworks.com; m.ben.franklin@gmail.com; benjamin.hindman@gmail.com; ������(�;���)
Cc: aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com; mlchen@iflytek.com; zeus@tencent.com; ���ţ�������; ����; ��С��(����); dev@storm.apache.org
Subject: Re: Could JStorm project collaborate with Storm?
Hi Andy,
Thanks for your quick response. What you concern is reasonable and understandable.
But maybe there is some misunderstanding on our proposal. According to the current development situation in China, java core is indeed to bring more contributors here.
That is reason why we begin to think if it is also a way for us to provide our contribution to Storm project. It is true that it is a risk to switch the core in a short period.
Actually, if possible, maybe we can consider to take JStorm as a subproject of Storm(A branch or some other ways��). We will be responsible for the maintain of it and try to propagate the features from Storm to JStorm, and continue to develop our features and improvements on this. You guys can try the new core and take a long time to estimate if it is worth doing the switch in the future or taking JStorm as official optional core of Storm for user.
Regards
Basti
������: Andrew Feng [mailto:afeng@yahoo-inc.com]
����ʱ��: 2014��10��24�� 0:24
�ռ���: ������(�;���); nathan@nathanmarz.com; ������(����); jason@cvk.ca; flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com; cutting@apache.org; tdunning@maprtech.com; arvind@apache.org; ddas@hortonworks.com; m.ben.franklin@gmail.com; benjamin.hindman@gmail.com
����: aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com; mlchen@iflytek.com; zeus@tencent.com
����: Re: ��: Could JStorm project collaborate with Storm?
Zhongyan:
Yahoo almost took your path 2 years ago. After some discussion with Nathan, we decided to work with community on Storm, and some of us learned Clojure quickly. We are very glad that we did that.
Under the assumption that Storm provides all features of JStorm, my personal vote will be NO for your proposal.
* Storm has enjoyed its current success before Nathan built a very
solid core in Clojure. We should not replace that core until we are 100% sure that alternative implementation is at least as good as our Clojure implementation.
* Clojure has not prevent Storm attract contributors (currently 108).
We have contributors who write code in Clojure and Java. It doesn��t take much time for one to be able to understand Clojure code.
* The convergence of streaming processing and batch processing will
occur at higher level of abstractions. Trident API, for example, is very much similar to batch API such as Pig or Cascading.
* Including Jstorm under Storm will only create confusion to our user
community at this stage.
Why don��t you have discussion with ������ to figure out an alternative path for JStorm?
Thanks,
Andy
From: "������ (�;���)" <zh...@alibaba-inc.com>
Reply-To: "������(�;���)" <zh...@alibaba-inc.com>
Date: Wednesday, October 22, 2014 at 11:37 PM
To: Andy Feng <af...@yahoo-inc.com>, "nathan@nathanmarz.com"
<na...@nathanmarz.com>, "������(����)" <mi...@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, "flip@infochimps.com" <fl...@infochimps.com>, "davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
<pt...@gmail.com>, Doug Cutting <cu...@apache.org>, "tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org"
<ar...@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "m.ben.franklin@gmail.com" <m....@gmail.com>, "benjamin.hindman@gmail.com" <be...@gmail.com>
Cc: aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com" <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>, "zeus@tencent.com" <zeus@tencent.
com>
Subject: ��: Could JStorm project collaborate with Storm?
Andrew, Thanks for suggest.
The problem is that the core of Storm is implemented by Clojure, if it is java, we are glad to merge all our commit into the Strom Trunk.
If the core of Storm is implemented by Java, I think the contributor of Storm would be double.
May I put forward 2 proposal:
1. Could Jstorm be the son project of Storm?
2. Task a long time to change Storm-core as Jstorm-core.
a) If the core of Storm is implemented by Java, I think the
contributor of Storm would be double. Once one user found one bug, maybe he is able to fix it by himself, in this method, Storm resolve problem will be double
b) In the next one or two years, batch handling and stream handling
will merge into one solution, Spark/Flink are doing this job. If we still use clojure, I am a little afraid that we can��t follow the steps of other community and this direction.
Thanks
Longda
������: Andrew Feng [mailto:afeng@yahoo-inc.com]
����ʱ��: 2014��10��23�� 13:31
�ռ���: ������(�;���); nathan@nathanmarz.com; ������(����); jason@cvk.ca; flip@infochimps.com; davidlao@microsoft.com; ptgoetz@gmail.com; cutting@apache.org; tdunning@maprtech.com; arvind@apache.org; ddas@hortonworks.com; m.ben.franklin@gmail.com; benjamin.hindman@gmail.com
����: aloha-dev; scott.zhai@vipshop.com; yannianmu@tencent.com; mlchen@iflytek.com; zeus@tencent.com
����: Re: Could JStorm project collaborate with Storm?
Zhongyan:
I would suggest you identify a collection of key features in Jstorm, and then propose pull requests to Storm.
Once these pull requests are accepted, you could then simply adopt Storm.
Andy
From: "������ (�;���)" <zh...@alibaba-inc.com>
Reply-To: "������(�;���)" <zh...@alibaba-inc.com>
Date: Wednesday, October 22, 2014 at 8:23 PM
To: "nathan@nathanmarz.com" <na...@nathanmarz.com>, "������(����)"
<mi...@alibaba-inc.com>, "jason@cvk.ca" <ja...@cvk.ca>, Andy Feng <af...@yahoo-inc.com>, "flip@infochimps.com" <fl...@infochimps.com>, "davidlao@microsoft.com" <da...@microsoft.com>, "ptgoetz@gmail.com"
<pt...@gmail.com>, Doug Cutting <cu...@apache.org>, "tdunning@maprtech.com" <td...@maprtech.com>, "arvind@apache.org"
<ar...@apache.org>, "ddas@hortonworks.com" <dd...@hortonworks.com>, "m.ben.franklin@gmail.com" <m....@gmail.com>, "benjamin.hindman@gmail.com" <be...@gmail.com>
Cc: aloha-dev <al...@list.alibaba-inc.com>, "scott.zhai@vipshop.com"
<sc...@vipshop.com>, "yannianmu@tencent.com" <ya...@tencent.com>, "mlchen@iflytek.com" <ml...@iflytek.com>, "zeus@tencent.com" <zeus@tencent.
com>
Subject: Could JStorm project collaborate with Storm?
Hi all,
Firstly, congratulate Storm becoming the top project of Apache. Since the first day of Storm opened source, it has become one of hottest technologies in the BigData field and brought a new perspective to the world, which has helped many companies resolve hundreds of requirements.
Since 2012/4, we have forked a project named "JStorm". At the beginning time, we just wanted to implement Storm with Java. As time goes on, more and more features have been added to JStorm. In 2013/9, JStorm changed to be an open source project; from then on, more and more people and companies join this project.
As JStorm is derived from Storm, and born in a great company named Alibaba, which just successfully listed on the New York Stock Exchange on Sept. 19, is a very open company. We also want to share our technology with more people, and favor rapid evolution of streaming technology.
So, could JStorm project collaborate with Storm?
A little introduction for JStorm as follows:
JStorm is a distributed and fault-tolerant real-time computation system.
Inspired by Apache Storm, JStorm has been completely rewritten in Java and provides more enhanced features. JStorm has been widely used in many enterprise environments and proved robust and stable.
Begun from 2012/2, we have released 11 versions till 2014/09/28. More than
10 companies are using it, including several leading companies such as Alibaba, Tencent, Baidu, Huawei, ZTE in China.
Please refer to https://github.com/alibaba/jstorm for more details
Regards
Longda