You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "封仲淹(纪君祥)" <zh...@alibaba-inc.com> on 2015/03/24 03:02:08 UTC
答复: begging for a bit of help reading storm-core code
You can read the source code of Jstorm, It is java Storm. It is easy to read
https://github.com/alibaba/jstorm
I can share one freemind file to you to demonstrate the architecture and work flow.
Best Regards
Longda
-----邮件原件-----
发件人: Erik Weathers [mailto:eweathers@groupon.com]
发送时间: 2015年3月24日 8:58
收件人: dev@storm.apache.org
主题: begging for a bit of help reading storm-core code
hi Storm Devs!
I've been trying to discern what the purpose is for the ubiquitous "callback" variables in cluster.clj and supervisor.clj code. It occurred to me that some of you fine folks on the dev list must have already braved this test and could help dispel my ignorance. Specifically, I'm trying to determine what the mk-storm-cluster-state function is actually doing, and understanding the callbacks seems critical to that endeavor.
I had hoped that commit log messages might help me infer the point of these callbacks, but this particular pattern hasn't changed since the initial commit by Nathan Marz.
Thanks for whatever help you can provide!
- Erik
P.S., I have many other similar questions about the storm-core Clojure code, so I wonder if someone might have a suggestion of a different route for to requesting such code-reading help?
Re: begging for a bit of help reading storm-core code
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Yes that message means that ZK does not always inform you of a change. This mostly happens when reconnecting after losing a connection.
- Bobby
On Wednesday, March 25, 2015 3:44 AM, Erik Weathers <ew...@groupon.com> wrote:
Thanks for the response Bobby, on both fronts.
Can you please confirm my new understanding of this potentially misleading
comment above *mk-storm-cluster-state?*
https://github.com/apache/storm/blob/v0.9.3/storm-core/src/clj/backtype/storm/cluster.clj#L233
;; Watches should be used for optimization. When ZK is reconnecting,
they're not guaranteed to be called.
This comment sounds like a TODO, but based on your response I assume the
code is already using ZooKeeper watches. So now I believe that this
comment is instead warning potential callers that these watch-based
callbacks aren't reliable. Is that correct?
Thanks again!
- Erik
On Tue, Mar 24, 2015 at 6:49 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Two things. First storm uses zookeeper to store state and offers a watch
> that can inform a client when a particular znode in zookeeper changes. The
> callbacks provide a way for a client to be informed that the particular
> part of the state they care about has changed, or nil if you don't want
> it. The ZK functionality is not always perfect in getting the callbacks,
> so it is always a good idea to back up the callback with polling
> periodically. Also at least for people I know working on DRUID the order
> of the callbacks on certain versions of ZK can be out of order, so if there
> are a lot of changes happening to a single ZNode you may need to be
> careful, luckily this is not an issue for how storm currently uses ZK.
>
> Now for the second thing. JStorm is a fork of Apache Storm where the
> clojure code was translated into java code, so if you are having trouble
> reading the clojure code you can look at what JStorm is doing for a hint.
> Be aware that the two projects have diverged somewhat so they are not going
> to be identical in their functionality.
> That being said both projects have been talking with one another about
> combining. https://issues.apache.org/jira/browse/STORM-717 is a JIRA to
> work through making that happen. If you or anyone else has an opinion on
> this please feel free to discuss it on that JIRA on here on the dev list,
> I'll send out another e-mail to provide a better place than piggybacking it
> here. The goal would be to maintain binary compatibility, except possibly
> in the case of dependencies. It would be great to have both groups working
> together instead of duplication of effort. We also felt that having more
> of the code base in java would possibly make it more accessible to a wider
> range of developers. That being said this project is a community effort so
> anyone who wants to help, or has opinions on this please help out/let us
> know what you think. - Bobby
>
>
>
> On Tuesday, March 24, 2015 12:51 AM, Erik Weathers <
> eweathers@groupon.com> wrote:
>
>
> hi Longda, thanks for the response. Interesting project there. I would
> appreciate seeing the architecture/work-flow file if you could please share
> it.
> However, we are using standard Storm, and I need to continue supporting our
> existing system -- which requires me to dig into the Clojure-based storm
> code.
>
> Any other kind passersby that can help me?
>
> - Erik
>
> On Monday, March 23, 2015, 封仲淹(纪君祥) <zh...@alibaba-inc.com> wrote:
>
> >
> > You can read the source code of Jstorm, It is java Storm. It is easy to
> > read
> > https://github.com/alibaba/jstorm
> > I can share one freemind file to you to demonstrate the architecture and
> > work flow.
> >
> >
> >
> > Best Regards
> > Longda
> >
> >
> > -----邮件原件-----
> > 发件人: Erik Weathers [mailto:eweathers@groupon.com]
> > 发送时间: 2015年3月24日 8:58
> > 收件人: dev@storm.apache.org
> > 主题: begging for a bit of help reading storm-core code
> >
> > hi Storm Devs!
> >
> > I've been trying to discern what the purpose is for the ubiquitous
> > "callback" variables in cluster.clj and supervisor.clj code. It occurred
> > to me that some of you fine folks on the dev list must have already
> braved
> > this test and could help dispel my ignorance. Specifically, I'm trying
> to
> > determine what the mk-storm-cluster-state function is actually doing, and
> > understanding the callbacks seems critical to that endeavor.
> >
> > I had hoped that commit log messages might help me infer the point of
> > these callbacks, but this particular pattern hasn't changed since the
> > initial commit by Nathan Marz.
> >
> > Thanks for whatever help you can provide!
> >
> > - Erik
> >
> > P.S., I have many other similar questions about the storm-core Clojure
> > code, so I wonder if someone might have a suggestion of a different route
> > for to requesting such code-reading help?
> >
> >
>
>
>
Re: begging for a bit of help reading storm-core code
Posted by Erik Weathers <ew...@groupon.com>.
Thanks for the response Bobby, on both fronts.
Can you please confirm my new understanding of this potentially misleading
comment above *mk-storm-cluster-state?*
https://github.com/apache/storm/blob/v0.9.3/storm-core/src/clj/backtype/storm/cluster.clj#L233
;; Watches should be used for optimization. When ZK is reconnecting,
they're not guaranteed to be called.
This comment sounds like a TODO, but based on your response I assume the
code is already using ZooKeeper watches. So now I believe that this
comment is instead warning potential callers that these watch-based
callbacks aren't reliable. Is that correct?
Thanks again!
- Erik
On Tue, Mar 24, 2015 at 6:49 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Two things. First storm uses zookeeper to store state and offers a watch
> that can inform a client when a particular znode in zookeeper changes. The
> callbacks provide a way for a client to be informed that the particular
> part of the state they care about has changed, or nil if you don't want
> it. The ZK functionality is not always perfect in getting the callbacks,
> so it is always a good idea to back up the callback with polling
> periodically. Also at least for people I know working on DRUID the order
> of the callbacks on certain versions of ZK can be out of order, so if there
> are a lot of changes happening to a single ZNode you may need to be
> careful, luckily this is not an issue for how storm currently uses ZK.
>
> Now for the second thing. JStorm is a fork of Apache Storm where the
> clojure code was translated into java code, so if you are having trouble
> reading the clojure code you can look at what JStorm is doing for a hint.
> Be aware that the two projects have diverged somewhat so they are not going
> to be identical in their functionality.
> That being said both projects have been talking with one another about
> combining. https://issues.apache.org/jira/browse/STORM-717 is a JIRA to
> work through making that happen. If you or anyone else has an opinion on
> this please feel free to discuss it on that JIRA on here on the dev list,
> I'll send out another e-mail to provide a better place than piggybacking it
> here. The goal would be to maintain binary compatibility, except possibly
> in the case of dependencies. It would be great to have both groups working
> together instead of duplication of effort. We also felt that having more
> of the code base in java would possibly make it more accessible to a wider
> range of developers. That being said this project is a community effort so
> anyone who wants to help, or has opinions on this please help out/let us
> know what you think. - Bobby
>
>
>
> On Tuesday, March 24, 2015 12:51 AM, Erik Weathers <
> eweathers@groupon.com> wrote:
>
>
> hi Longda, thanks for the response. Interesting project there. I would
> appreciate seeing the architecture/work-flow file if you could please share
> it.
> However, we are using standard Storm, and I need to continue supporting our
> existing system -- which requires me to dig into the Clojure-based storm
> code.
>
> Any other kind passersby that can help me?
>
> - Erik
>
> On Monday, March 23, 2015, 封仲淹(纪君祥) <zh...@alibaba-inc.com> wrote:
>
> >
> > You can read the source code of Jstorm, It is java Storm. It is easy to
> > read
> > https://github.com/alibaba/jstorm
> > I can share one freemind file to you to demonstrate the architecture and
> > work flow.
> >
> >
> >
> > Best Regards
> > Longda
> >
> >
> > -----邮件原件-----
> > 发件人: Erik Weathers [mailto:eweathers@groupon.com]
> > 发送时间: 2015年3月24日 8:58
> > 收件人: dev@storm.apache.org
> > 主题: begging for a bit of help reading storm-core code
> >
> > hi Storm Devs!
> >
> > I've been trying to discern what the purpose is for the ubiquitous
> > "callback" variables in cluster.clj and supervisor.clj code. It occurred
> > to me that some of you fine folks on the dev list must have already
> braved
> > this test and could help dispel my ignorance. Specifically, I'm trying
> to
> > determine what the mk-storm-cluster-state function is actually doing, and
> > understanding the callbacks seems critical to that endeavor.
> >
> > I had hoped that commit log messages might help me infer the point of
> > these callbacks, but this particular pattern hasn't changed since the
> > initial commit by Nathan Marz.
> >
> > Thanks for whatever help you can provide!
> >
> > - Erik
> >
> > P.S., I have many other similar questions about the storm-core Clojure
> > code, so I wonder if someone might have a suggestion of a different route
> > for to requesting such code-reading help?
> >
> >
>
>
>
Re: begging for a bit of help reading storm-core code
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Two things. First storm uses zookeeper to store state and offers a watch that can inform a client when a particular znode in zookeeper changes. The callbacks provide a way for a client to be informed that the particular part of the state they care about has changed, or nil if you don't want it. The ZK functionality is not always perfect in getting the callbacks, so it is always a good idea to back up the callback with polling periodically. Also at least for people I know working on DRUID the order of the callbacks on certain versions of ZK can be out of order, so if there are a lot of changes happening to a single ZNode you may need to be careful, luckily this is not an issue for how storm currently uses ZK.
Now for the second thing. JStorm is a fork of Apache Storm where the clojure code was translated into java code, so if you are having trouble reading the clojure code you can look at what JStorm is doing for a hint. Be aware that the two projects have diverged somewhat so they are not going to be identical in their functionality.
That being said both projects have been talking with one another about combining. https://issues.apache.org/jira/browse/STORM-717 is a JIRA to work through making that happen. If you or anyone else has an opinion on this please feel free to discuss it on that JIRA on here on the dev list, I'll send out another e-mail to provide a better place than piggybacking it here. The goal would be to maintain binary compatibility, except possibly in the case of dependencies. It would be great to have both groups working together instead of duplication of effort. We also felt that having more of the code base in java would possibly make it more accessible to a wider range of developers. That being said this project is a community effort so anyone who wants to help, or has opinions on this please help out/let us know what you think. - Bobby
On Tuesday, March 24, 2015 12:51 AM, Erik Weathers <ew...@groupon.com> wrote:
hi Longda, thanks for the response. Interesting project there. I would
appreciate seeing the architecture/work-flow file if you could please share
it.
However, we are using standard Storm, and I need to continue supporting our
existing system -- which requires me to dig into the Clojure-based storm
code.
Any other kind passersby that can help me?
- Erik
On Monday, March 23, 2015, 封仲淹(纪君祥) <zh...@alibaba-inc.com> wrote:
>
> You can read the source code of Jstorm, It is java Storm. It is easy to
> read
> https://github.com/alibaba/jstorm
> I can share one freemind file to you to demonstrate the architecture and
> work flow.
>
>
>
> Best Regards
> Longda
>
>
> -----邮件原件-----
> 发件人: Erik Weathers [mailto:eweathers@groupon.com]
> 发送时间: 2015年3月24日 8:58
> 收件人: dev@storm.apache.org
> 主题: begging for a bit of help reading storm-core code
>
> hi Storm Devs!
>
> I've been trying to discern what the purpose is for the ubiquitous
> "callback" variables in cluster.clj and supervisor.clj code. It occurred
> to me that some of you fine folks on the dev list must have already braved
> this test and could help dispel my ignorance. Specifically, I'm trying to
> determine what the mk-storm-cluster-state function is actually doing, and
> understanding the callbacks seems critical to that endeavor.
>
> I had hoped that commit log messages might help me infer the point of
> these callbacks, but this particular pattern hasn't changed since the
> initial commit by Nathan Marz.
>
> Thanks for whatever help you can provide!
>
> - Erik
>
> P.S., I have many other similar questions about the storm-core Clojure
> code, so I wonder if someone might have a suggestion of a different route
> for to requesting such code-reading help?
>
>
Re: begging for a bit of help reading storm-core code
Posted by Erik Weathers <ew...@groupon.com>.
hi Longda, thanks for the response. Interesting project there. I would
appreciate seeing the architecture/work-flow file if you could please share
it.
However, we are using standard Storm, and I need to continue supporting our
existing system -- which requires me to dig into the Clojure-based storm
code.
Any other kind passersby that can help me?
- Erik
On Monday, March 23, 2015, 封仲淹(纪君祥) <zh...@alibaba-inc.com> wrote:
>
> You can read the source code of Jstorm, It is java Storm. It is easy to
> read
> https://github.com/alibaba/jstorm
> I can share one freemind file to you to demonstrate the architecture and
> work flow.
>
>
>
> Best Regards
> Longda
>
>
> -----邮件原件-----
> 发件人: Erik Weathers [mailto:eweathers@groupon.com]
> 发送时间: 2015年3月24日 8:58
> 收件人: dev@storm.apache.org
> 主题: begging for a bit of help reading storm-core code
>
> hi Storm Devs!
>
> I've been trying to discern what the purpose is for the ubiquitous
> "callback" variables in cluster.clj and supervisor.clj code. It occurred
> to me that some of you fine folks on the dev list must have already braved
> this test and could help dispel my ignorance. Specifically, I'm trying to
> determine what the mk-storm-cluster-state function is actually doing, and
> understanding the callbacks seems critical to that endeavor.
>
> I had hoped that commit log messages might help me infer the point of
> these callbacks, but this particular pattern hasn't changed since the
> initial commit by Nathan Marz.
>
> Thanks for whatever help you can provide!
>
> - Erik
>
> P.S., I have many other similar questions about the storm-core Clojure
> code, so I wonder if someone might have a suggestion of a different route
> for to requesting such code-reading help?
>
>