You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Kang Xiao <kx...@gmail.com> on 2014/03/26 07:19:17 UTC
回复： [DISCUSS] Release Apache Storm 0.9.2-incubating
hi guys
How about adding a bin/build_release.sh (http://build_release.sh) script in this 0.9.2-incubating release?
Since some guys asked the question about building storm release package more than once in the mail list.
--
Best Regards!
肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
Distributed Software Engineer
在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
> Taylor, I am not very clear on "Lazy consensus +2". By this definition code
> can committed with no +1 from a committer, right?
>
> You may want to look at Hadoop bylaws -
> https://hadoop.apache.org/bylaws.html
>
> The code commits in Hadoop are consensus approval with minimum +1 from an
> active committer and no veto. This has worked well in my experience. It may
> be a good idea to also adapt minimum 3 +1s from active committers for
> merging feature branches.
>
>
>
>
> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)> wrote:
>
> > That brings up a good point that probably deserves a separate thread.
> >
> > We should establish by-laws soon. Specifically a commit/merge policy.
> >
> > For code changes, I've been operating under a "lazy consensus +2" model: 2
> > committer +1 votes and no vetoes (-1). If a committer submits the patch,
> > that's an implicit +1. Unless it's a somewhat urgent fix, I've been waiting
> > for 3 binding votes and no vetoes.
> >
> > That's kind of a middle ground between the traditional code modification
> > rule and lazy consensus [1].
> >
> > When wearing my "release manager" hat, I've also interpreted "code change"
> > to mean "anything that alters the behavior of the software we produce." In
> > terms of the build/packaging I've been a little looser. For large changes
> > (e.g. The switch to maven), I've waited for 3 binding votes. For some
> > changes I've committed directly -- I don't think we need to have a 3-day
> > vote on updating the CHANGELOG, for example.
> >
> > Anyway, it's something to think about. Sorry for hijacking the thread.
> >
> > +1 (again ;) )
> >
> > Taylor
> >
> > [1] https://www.apache.org/foundation/voting.html
> >
> > > On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com (mailto:nathan@nathanmarz.com)> wrote:
> > >
> > > Let's get https://issues.apache.org/jira/browse/STORM-262 in there. Just
> > > one more vote needed by a committer.
> > >
> > >
> > > > On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com (mailto:plucas@yelp.com)> wrote:
> > > >
> > > > A fix STORM-120 would be greatly appreciated. It's making it impossible
> > to
> > > > increase tasks/executors > 1 when there is a downstream shuffle
> > >
> >
> > grouping.
> > > >
> > > > I'm not sure why there haven't been more reports of problems with it.
> > Two
> > > > possibilities I can think of are that we are using exclusively shell
> > > > components--perhaps there's a root-cause bug in those component
> > > > classes--and
> > > > that we are dealing with a high volume stream of large tuples.
> > > >
> > >
> >
> > (thousands /
> > > > sec, KB in size)
> > > >
> > > >
> > > > On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
> > > > wrote:
> > > >
> > > > > Never mind... just found it.
> > > > >
> > > > > > On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
> > wrote:
> > > > > >
> > > > > > Derek do you have an idea for a fix?
> > > > > >
> > > > > > On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com (mailto:derekd@yahoo-inc.com)>
> > wrote:
> > > > > >
> > > > > > > > As I said above, this fix is the most important in my opinion.
> > > > > > > > STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
> > > > > > > > important as STORM-187 or not.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Yeah, we found it recently, and I created it this morning after
> > > > reading
> > > > > Taylor's mail.
> > > > > > >
> > > > > > > STORM-187 can be a problem with fewer than 30 retries (likelihood
> > > > > depends on configuration), but we will hit STORM-259 when retries
> > > > >
> > > >
> > > >
> > >
> >
> > exceeds
> > > > > 30.
> > > > > > >
> > > > > > > --
> > > > > > > Derek
> > > > > > >
> > > > > > > > On 3/20/14, 14:18, Michael G. Noll wrote:
> > > > > > > > On my side the most important change is, as you point out,
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> > STORM-187.
> > > > > > > > The primary reason is like Adam Lewis is pointing out because it's a
> > > > > > > > stability problem. The secondary aspect is that this issue taints
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > the
> > > > > > > > new Netty backend, and at least IMHO the faster Storm could
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > confidently
> > > > > > > > bury ZeroMQ the better. :-)
> > > > > > > >
> > > > > > > > As I said above, this fix is the most important in my opinion.
> > > > > > > > STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
> > > > > > > > important as STORM-187 or not.
> > > > > > > >
> > > > > > > > Switching to my non-essential wishlist I'd also +1 STORM-252
> > (Upgrade
> > > > > > > > Curator and thus ZooKeeper to 3.4.5). We have been running ZK 3.4.5
> > > > > > > > anyway for a couple of reasons, and it would be nice to have
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > official
> > > > > > > > Storm support for the latest ZK version (ok, the recently released
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ZK
> > > > > > > > 3.4.6 is actually the latest but hey). Although I don't know how
> > > > > > > > confident we are that the code in STORM-252 actually works, i.e.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > whether
> > > > > > > > integrating STORM-252 into 0.9.2 on such short notice would be
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > jumping
> > > > > > > > the gun or a safe move.
> > > > > > > >
> > > > > > > > Btw, in terms of Storm/Kafka integration Kafka is in the same boat:
> > > > > > > > it's built against ZK 3.3.x, and LinkedIn recommends the use of ZK
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > 3.3.4
> > > > > > > > in the docs. There's an open ticket KAFKA-854 [1] that's basically
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > the
> > > > > > > > equivalent of STORM-252, but I'm not sure how actively the Kafka
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> > team
> > > > > is
> > > > > > > > working on that.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Michael
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/KAFKA-854
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
> > > > > > > > > I'd like to get this discussion started, largely because the
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > "negative timeout" bug (STORM-187) really bothers me. I've not seen it
> > > > >
> > > >
> > >
> >
> > in
> > > > > the wild, but I've heard of a few cases where it was enough to hinder
> > > > > upgrading.
> > > > > > > > >
> > > > > > > > > HEAD looks good to me at the moment, with the major difference
> > being
> > > > > the zookeeper update and the patch mentioned above.
> > > > > > > > >
> > > > > > > > > Any thoughts on other PRs or patches to include?
> > > > > > > > >
> > > > > > > > > -Taylor
> > > >
> > > >
> > > > --
> > > > Patrick Lucas
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > Twitter: @nathanmarz
> > > http://nathanmarz.com
> > >
> >
> >
>
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Patrick Lucas <pl...@yelp.com>.
STORM-196 is an issue affecting anyone trying to use STORM_JAR_JVM_OPTS.
The PR on github is concise and ready to go:
https://github.com/apache/incubator-storm/pull/27
On Mon, Mar 31, 2014 at 1:46 PM, Milinda Pathirage <mp...@umail.iu.edu>wrote:
> Hi Taylor,
>
> Is there a fork or branch with Clojure 1.5 support? Or is this just a
> change of Clojure version in master branch?
>
> Thanks
> Milinda
>
> On Thu, Mar 27, 2014 at 1:05 AM, P. Taylor Goetz <pt...@gmail.com>
> wrote:
> > As promised, binaries can be found here:
> >
> >
> http://people.apache.org/~ptgoetz/storm/storm-0.9.2-incubating/dep-frankenbuild/
> >
> > This is not a release. It is merely a convenience build provided to
> enable further testing among the community.
> >
> > -Taylor
> >
> > On Mar 26, 2014, at 9:54 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> >
> >> I've submitted pull requests for most of these changes (I think I still
> need to submit one for http-client), where there wasn't already one.
> >>
> >> The only problem I saw was with upgrading to clojure 1.6.0. IIRC it had
> to do with ns conflicts with "some.*". Falling back to 1.5.1 solved it. I
> didn't investigate it. But it seems upgrading to 1.6.x will require some
> code change.
> >>
> >> Anyway I created a snapshot build with all these updates pulled in. All
> unit tests pass, and my basic fully distributed mode smoke tests pass, so
> I'm leaning toward a +1 for all.
> >>
> >> However, if anyone who is interested in seeing these upgrades merged
> has the time and resources to try it out, that would be a huge help to us.
> >>
> >> I'll post the binaries soon, when I have access to a real computer.
> >>
> >> -Taylor
> >>
> >>> On Mar 26, 2014, at 10:49 AM, Brian O'Neill <bo...@alumni.brown.edu>
> wrote:
> >>>
> >>>
> >>> I’m +1 on all of those (including the clojure version bump).
> >>>
> >>> We’ve been hit with the maven-clojure-plugin issue as well, with no
> real
> >>> way to diagnose.
> >>>
> >>> -brian
> >>>
> >>> ---
> >>> Brian O'Neill
> >>> Chief Technology Officer
> >>>
> >>> Health Market Science
> >>> The Science of Better Results
> >>> 2700 Horizon Drive · King of Prussia, PA · 19406
> >>> M: 215.588.6024 · @boneill42 <http://www.twitter.com/boneill42> ·
> >>> healthmarketscience.com
> >>>
> >>> This information transmitted in this email message is for the intended
> >>> recipient only and may contain confidential and/or privileged
> material. If
> >>> you received this email in error and are not the intended recipient, or
> >>> the person responsible to deliver it to the intended recipient, please
> >>> contact the sender at the email above and delete this email and any
> >>> attachments and destroy any copies thereof. Any review, retransmission,
> >>> dissemination, copying or other use of, or taking any action in
> reliance
> >>> upon, this information by persons or entities other than the intended
> >>> recipient is strictly prohibited.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> On 3/26/14, 10:09 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
> >>>>
> >>>> There are a number of email threads and JIRA issues regarding
> upgrading
> >>>> various storm dependencies, so I’d like to enumerate them and discuss
> >>>> them in one thread.
> >>>>
> >>>> Here’s the list so far:
> >>>>
> >>>> 1. Kryo/Carbonite (STORM-263)[1]
> >>>> 2. Clojure (STORM-265) [2]
> >>>> 3. commons-io (STORM-258) [3]
> >>>> 4. curator (STORM-252) [4]
> >>>> 5. http-client [5]
> >>>>
> >>>> I am +1 for all of the above, with the exception of #2, which I am +0
> >>>> only because I personally haven’t had a chance to do any testing with
> >>>> newer versions of clojure. I’d be interested to hear if anyone has
> done
> >>>> any testing with newer versions of clojure.
> >>>>
> >>>> I think we should at least consider a bump to clojure 1.5 since it
> >>>> includes Bobby Evan’s patch that fixes error output getting swallowed
> >>>> (this manifests itself as the maven-clojure-plugin failing without any
> >>>> useful information when there are certain AOT compilation issues —
> very
> >>>> annoying).
> >>>>
> >>>> - Taylor
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/STORM-263
> >>>> [2] https://issues.apache.org/jira/browse/STORM-265
> >>>> [3] https://issues.apache.org/jira/browse/STORM-258
> >>>> [4] https://issues.apache.org/jira/browse/STORM-252
> >>>> [5]
> >>>>
> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22
> >>>> E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
> >>>>
> >>>>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu>
> wrote:
> >>>>>
> >>>>> Agreed. One of our guys got hung up on that just yesterday.
> >>>>> It isn’t hard to track down the assembly, but it might be worth
> making
> >>>>> it a bit easier.
> >>>>>
> >>>>> Also, I’d love to see STORM-263 included in the next release. It
> >>>>> seems like a quick win.
> >>>>>
> >>>>> -brian
> >>>>>
> >>>>> ---
> >>>>> Brian O'Neill
> >>>>> Chief Technology Officer
> >>>>>
> >>>>> Health Market Science
> >>>>> The Science of Better Results
> >>>>> 2700 Horizon Drive · King of Prussia, PA · 19406
> >>>>> M: 215.588.6024 · @boneill42 · healthmarketscience.com
> >>>>>
> >>>>> This information transmitted in this email message is for the
> intended
> >>>>> recipient only and may contain confidential and/or privileged
> material.
> >>>>> If you received this email in error and are not the intended
> recipient,
> >>>>> or the person responsible to deliver it to the intended recipient,
> >>>>> please contact the sender at the email above and delete this email
> and
> >>>>> any attachments and destroy any copies thereof. Any review,
> >>>>> retransmission, dissemination, copying or other use of, or taking any
> >>>>> action in reliance upon, this information by persons or entities
> other
> >>>>> than the intended recipient is strictly prohibited.
> >>>>>
> >>>>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> hi guys
> >>>>>>
> >>>>>> How about adding a bin/build_release.sh (http://build_release.sh)
> >>>>>> script in this 0.9.2-incubating release?
> >>>>>>
> >>>>>> Since some guys asked the question about building storm release
> >>>>>> package more than once in the mail list.
> >>>>>>
> >>>>>> --
> >>>>>> Best Regards!
> >>>>>>
> >>>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com
> )>)
> >>>>>> Distributed Software Engineer
> >>>>>>
> >>>>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
> >>>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
> >>>>>>> definition code
> >>>>>>> can committed with no +1 from a committer, right?
> >>>>>>>
> >>>>>>> You may want to look at Hadoop bylaws -
> >>>>>>> https://hadoop.apache.org/bylaws.html
> >>>>>>>
> >>>>>>> The code commits in Hadoop are consensus approval with minimum +1
> >>>>>>> from an
> >>>>>>> active committer and no veto. This has worked well in my
> experience.
> >>>>>>> It may
> >>>>>>> be a good idea to also adapt minimum 3 +1s from active committers
> for
> >>>>>>> merging feature branches.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <
> ptgoetz@gmail.com
> >>>>>>> (mailto:ptgoetz@gmail.com)> wrote:
> >>>>>>>
> >>>>>>>> That brings up a good point that probably deserves a separate
> thread.
> >>>>>>>>
> >>>>>>>> We should establish by-laws soon. Specifically a commit/merge
> policy.
> >>>>>>>>
> >>>>>>>> For code changes, I've been operating under a "lazy consensus +2"
> >>>>>>>> model: 2
> >>>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
> >>>>>>>> patch,
> >>>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've
> been
> >>>>>>>> waiting
> >>>>>>>> for 3 binding votes and no vetoes.
> >>>>>>>>
> >>>>>>>> That's kind of a middle ground between the traditional code
> >>>>>>>> modification
> >>>>>>>> rule and lazy consensus [1].
> >>>>>>>>
> >>>>>>>> When wearing my "release manager" hat, I've also interpreted "code
> >>>>>>>> change"
> >>>>>>>> to mean "anything that alters the behavior of the software we
> >>>>>>>> produce." In
> >>>>>>>> terms of the build/packaging I've been a little looser. For large
> >>>>>>>> changes
> >>>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For
> some
> >>>>>>>> changes I've committed directly -- I don't think we need to have a
> >>>>>>>> 3-day
> >>>>>>>> vote on updating the CHANGELOG, for example.
> >>>>>>>>
> >>>>>>>> Anyway, it's something to think about. Sorry for hijacking the
> >>>>>>>> thread.
> >>>>>>>>
> >>>>>>>> +1 (again ;) )
> >>>>>>>>
> >>>>>>>> Taylor
> >>>>>>>>
> >>>>>>>> [1] https://www.apache.org/foundation/voting.html
> >>>>>>>>
> >>>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
> >>>>>>>>> (mailto:nathan@nathanmarz.com)> wrote:
> >>>>>>>>>
> >>>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in
> there.
> >>>>>>>>> Just
> >>>>>>>>> one more vote needed by a committer.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
> >>>>>>>>>> (mailto:plucas@yelp.com)> wrote:
> >>>>>>>>>>
> >>>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
> >>>>>>>>>> impossible
> >>>>>>>> to
> >>>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> grouping.
> >>>>>>>>>>
> >>>>>>>>>> I'm not sure why there haven't been more reports of problems
> with
> >>>>>>>>>> it.
> >>>>>>>> Two
> >>>>>>>>>> possibilities I can think of are that we are using exclusively
> >>>>>>>>>> shell
> >>>>>>>>>> components--perhaps there's a root-cause bug in those component
> >>>>>>>>>> classes--and
> >>>>>>>>>> that we are dealing with a high volume stream of large tuples.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> (thousands /
> >>>>>>>>>> sec, KB in size)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz
> >>>>>>>>>> <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Never mind... just found it.
> >>>>>>>>>>>
> >>>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <
> ptgoetz@gmail.com
> >>>>>>>>>>>> (mailto:ptgoetz@gmail.com)>
> >>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Derek do you have an idea for a fix?
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <
> derekd@yahoo-inc.com
> >>>>>>>>>>>> (mailto:derekd@yahoo-inc.com)>
> >>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> As I said above, this fix is the most important in my
> opinion.
> >>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
> >>>>>>>>>>>>>> it's as
> >>>>>>>>>>>>>> important as STORM-187 or not.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yeah, we found it recently, and I created it this morning
> after
> >>>>>>>>>> reading
> >>>>>>>>>>> Taylor's mail.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
> >>>>>>>>>>>>> (likelihood
> >>>>>>>>>>> depends on configuration), but we will hit STORM-259 when
> retries
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> exceeds
> >>>>>>>>>>> 30.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Derek
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
> >>>>>>>>>>>>>> On my side the most important change is, as you point out,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> STORM-187.
> >>>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out
> because
> >>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
> >>>>>>>>>>>>>> taints
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> the
> >>>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> confidently
> >>>>>>>>>>>>>> bury ZeroMQ the better. :-)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As I said above, this fix is the most important in my
> opinion.
> >>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
> >>>>>>>>>>>>>> it's as
> >>>>>>>>>>>>>> important as STORM-187 or not.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
> >>>>>>>> (Upgrade
> >>>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running
> ZK
> >>>>>>>>>>>>>> 3.4.5
> >>>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> official
> >>>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
> >>>>>>>>>>>>>> released
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> ZK
> >>>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
> >>>>>>>>>>>>>> how
> >>>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
> >>>>>>>>>>>>>> i.e.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> whether
> >>>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would
> be
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> jumping
> >>>>>>>>>>>>>> the gun or a safe move.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the
> same
> >>>>>>>>>>>>>> boat:
> >>>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
> >>>>>>>>>>>>>> of ZK
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 3.3.4
> >>>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
> >>>>>>>>>>>>>> basically
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> the
> >>>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
> >>>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> team
> >>>>>>>>>>> is
> >>>>>>>>>>>>>> working on that.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Michael
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
> >>>>>>>>>>>>>>> I'd like to get this discussion started, largely because
> the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
> >>>>>>>>>>> seen it
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> in
> >>>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
> >>>>>>>>>>> hinder
> >>>>>>>>>>> upgrading.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major
> difference
> >>>>>>>> being
> >>>>>>>>>>> the zookeeper update and the patch mentioned above.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Taylor
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Patrick Lucas
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Twitter: @nathanmarz
> >>>>>>>>> http://nathanmarz.com
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> http://hortonworks.com/download/
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>>> entity to
> >>>>>>> which it is addressed and may contain information that is
> >>>>>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> >>>>>>> notified that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
>
>
>
> --
> Milinda Pathirage
>
> PhD Student | Research Assistant
> School of Informatics and Computing | Data to Insight Center
> Indiana University
>
> twitter: milindalakmal
> skype: milinda.pathirage
> blog: http://milinda.pathirage.org
>
--
Patrick Lucas
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Milinda Pathirage <mp...@umail.iu.edu>.
Hi Taylor,
Is there a fork or branch with Clojure 1.5 support? Or is this just a
change of Clojure version in master branch?
Thanks
Milinda
On Thu, Mar 27, 2014 at 1:05 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> As promised, binaries can be found here:
>
> http://people.apache.org/~ptgoetz/storm/storm-0.9.2-incubating/dep-frankenbuild/
>
> This is not a release. It is merely a convenience build provided to enable further testing among the community.
>
> -Taylor
>
> On Mar 26, 2014, at 9:54 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> I've submitted pull requests for most of these changes (I think I still need to submit one for http-client), where there wasn't already one.
>>
>> The only problem I saw was with upgrading to clojure 1.6.0. IIRC it had to do with ns conflicts with "some.*". Falling back to 1.5.1 solved it. I didn't investigate it. But it seems upgrading to 1.6.x will require some code change.
>>
>> Anyway I created a snapshot build with all these updates pulled in. All unit tests pass, and my basic fully distributed mode smoke tests pass, so I'm leaning toward a +1 for all.
>>
>> However, if anyone who is interested in seeing these upgrades merged has the time and resources to try it out, that would be a huge help to us.
>>
>> I'll post the binaries soon, when I have access to a real computer.
>>
>> -Taylor
>>
>>> On Mar 26, 2014, at 10:49 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>>>
>>>
>>> I’m +1 on all of those (including the clojure version bump).
>>>
>>> We’ve been hit with the maven-clojure-plugin issue as well, with no real
>>> way to diagnose.
>>>
>>> -brian
>>>
>>> ---
>>> Brian O'Neill
>>> Chief Technology Officer
>>>
>>> Health Market Science
>>> The Science of Better Results
>>> 2700 Horizon Drive · King of Prussia, PA · 19406
>>> M: 215.588.6024 · @boneill42 <http://www.twitter.com/boneill42> ·
>>> healthmarketscience.com
>>>
>>> This information transmitted in this email message is for the intended
>>> recipient only and may contain confidential and/or privileged material. If
>>> you received this email in error and are not the intended recipient, or
>>> the person responsible to deliver it to the intended recipient, please
>>> contact the sender at the email above and delete this email and any
>>> attachments and destroy any copies thereof. Any review, retransmission,
>>> dissemination, copying or other use of, or taking any action in reliance
>>> upon, this information by persons or entities other than the intended
>>> recipient is strictly prohibited.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On 3/26/14, 10:09 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>>>>
>>>> There are a number of email threads and JIRA issues regarding upgrading
>>>> various storm dependencies, so I’d like to enumerate them and discuss
>>>> them in one thread.
>>>>
>>>> Here’s the list so far:
>>>>
>>>> 1. Kryo/Carbonite (STORM-263)[1]
>>>> 2. Clojure (STORM-265) [2]
>>>> 3. commons-io (STORM-258) [3]
>>>> 4. curator (STORM-252) [4]
>>>> 5. http-client [5]
>>>>
>>>> I am +1 for all of the above, with the exception of #2, which I am +0
>>>> only because I personally haven’t had a chance to do any testing with
>>>> newer versions of clojure. I’d be interested to hear if anyone has done
>>>> any testing with newer versions of clojure.
>>>>
>>>> I think we should at least consider a bump to clojure 1.5 since it
>>>> includes Bobby Evan’s patch that fixes error output getting swallowed
>>>> (this manifests itself as the maven-clojure-plugin failing without any
>>>> useful information when there are certain AOT compilation issues — very
>>>> annoying).
>>>>
>>>> - Taylor
>>>>
>>>> [1] https://issues.apache.org/jira/browse/STORM-263
>>>> [2] https://issues.apache.org/jira/browse/STORM-265
>>>> [3] https://issues.apache.org/jira/browse/STORM-258
>>>> [4] https://issues.apache.org/jira/browse/STORM-252
>>>> [5]
>>>> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22
>>>> E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>>>
>>>>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>>>>>
>>>>> Agreed. One of our guys got hung up on that just yesterday.
>>>>> It isn’t hard to track down the assembly, but it might be worth making
>>>>> it a bit easier.
>>>>>
>>>>> Also, I’d love to see STORM-263 included in the next release. It
>>>>> seems like a quick win.
>>>>>
>>>>> -brian
>>>>>
>>>>> ---
>>>>> Brian O'Neill
>>>>> Chief Technology Officer
>>>>>
>>>>> Health Market Science
>>>>> The Science of Better Results
>>>>> 2700 Horizon Drive · King of Prussia, PA · 19406
>>>>> M: 215.588.6024 · @boneill42 · healthmarketscience.com
>>>>>
>>>>> This information transmitted in this email message is for the intended
>>>>> recipient only and may contain confidential and/or privileged material.
>>>>> If you received this email in error and are not the intended recipient,
>>>>> or the person responsible to deliver it to the intended recipient,
>>>>> please contact the sender at the email above and delete this email and
>>>>> any attachments and destroy any copies thereof. Any review,
>>>>> retransmission, dissemination, copying or other use of, or taking any
>>>>> action in reliance upon, this information by persons or entities other
>>>>> than the intended recipient is strictly prohibited.
>>>>>
>>>>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>>>>
>>>>>> hi guys
>>>>>>
>>>>>> How about adding a bin/build_release.sh (http://build_release.sh)
>>>>>> script in this 0.9.2-incubating release?
>>>>>>
>>>>>> Since some guys asked the question about building storm release
>>>>>> package more than once in the mail list.
>>>>>>
>>>>>> --
>>>>>> Best Regards!
>>>>>>
>>>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>>>> Distributed Software Engineer
>>>>>>
>>>>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>>>>>>> definition code
>>>>>>> can committed with no +1 from a committer, right?
>>>>>>>
>>>>>>> You may want to look at Hadoop bylaws -
>>>>>>> https://hadoop.apache.org/bylaws.html
>>>>>>>
>>>>>>> The code commits in Hadoop are consensus approval with minimum +1
>>>>>>> from an
>>>>>>> active committer and no veto. This has worked well in my experience.
>>>>>>> It may
>>>>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>>>>> merging feature branches.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>>> (mailto:ptgoetz@gmail.com)> wrote:
>>>>>>>
>>>>>>>> That brings up a good point that probably deserves a separate thread.
>>>>>>>>
>>>>>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>>>>>
>>>>>>>> For code changes, I've been operating under a "lazy consensus +2"
>>>>>>>> model: 2
>>>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>>>>>>>> patch,
>>>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>>>>>>>> waiting
>>>>>>>> for 3 binding votes and no vetoes.
>>>>>>>>
>>>>>>>> That's kind of a middle ground between the traditional code
>>>>>>>> modification
>>>>>>>> rule and lazy consensus [1].
>>>>>>>>
>>>>>>>> When wearing my "release manager" hat, I've also interpreted "code
>>>>>>>> change"
>>>>>>>> to mean "anything that alters the behavior of the software we
>>>>>>>> produce." In
>>>>>>>> terms of the build/packaging I've been a little looser. For large
>>>>>>>> changes
>>>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>>>>>> changes I've committed directly -- I don't think we need to have a
>>>>>>>> 3-day
>>>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>>>
>>>>>>>> Anyway, it's something to think about. Sorry for hijacking the
>>>>>>>> thread.
>>>>>>>>
>>>>>>>> +1 (again ;) )
>>>>>>>>
>>>>>>>> Taylor
>>>>>>>>
>>>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>>>
>>>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
>>>>>>>>> (mailto:nathan@nathanmarz.com)> wrote:
>>>>>>>>>
>>>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there.
>>>>>>>>> Just
>>>>>>>>> one more vote needed by a committer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
>>>>>>>>>> (mailto:plucas@yelp.com)> wrote:
>>>>>>>>>>
>>>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>>>>>>>>>> impossible
>>>>>>>> to
>>>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>>>
>>>>>>>>
>>>>>>>> grouping.
>>>>>>>>>>
>>>>>>>>>> I'm not sure why there haven't been more reports of problems with
>>>>>>>>>> it.
>>>>>>>> Two
>>>>>>>>>> possibilities I can think of are that we are using exclusively
>>>>>>>>>> shell
>>>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>>>> classes--and
>>>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> (thousands /
>>>>>>>>>> sec, KB in size)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz
>>>>>>>>>> <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Never mind... just found it.
>>>>>>>>>>>
>>>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>>>>>>>> (mailto:ptgoetz@gmail.com)>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
>>>>>>>>>>>> (mailto:derekd@yahoo-inc.com)>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>>> it's as
>>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>>>>> reading
>>>>>>>>>>> Taylor's mail.
>>>>>>>>>>>>>
>>>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>>>>>>>>>>>>> (likelihood
>>>>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> exceeds
>>>>>>>>>>> 30.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Derek
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> STORM-187.
>>>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>>>>>>>>>>>>>> taints
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> confidently
>>>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>>> it's as
>>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>>>> (Upgrade
>>>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>>>>>>>>>>>>>> 3.4.5
>>>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> official
>>>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>>>>>>>>>>>>>> released
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ZK
>>>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>>>>>>>>>>>>>> i.e.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> whether
>>>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> jumping
>>>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>>>>>>>>>>>>>> boat:
>>>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>>>>>>>>>>>>>> of ZK
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 3.3.4
>>>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>>>>>>>>>>>>>> basically
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> team
>>>>>>>>>>> is
>>>>>>>>>>>>>> working on that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Michael
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>>>>>>>>>>> seen it
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> in
>>>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>>>>>>>>>>> hinder
>>>>>>>>>>> upgrading.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>>>>>> being
>>>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Taylor
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Patrick Lucas
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Twitter: @nathanmarz
>>>>>>>>> http://nathanmarz.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> http://hortonworks.com/download/
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity to
>>>>>>> which it is addressed and may contain information that is
>>>>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>>>>>> notified that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
As promised, binaries can be found here:
http://people.apache.org/~ptgoetz/storm/storm-0.9.2-incubating/dep-frankenbuild/
This is not a release. It is merely a convenience build provided to enable further testing among the community.
-Taylor
On Mar 26, 2014, at 9:54 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I've submitted pull requests for most of these changes (I think I still need to submit one for http-client), where there wasn't already one.
>
> The only problem I saw was with upgrading to clojure 1.6.0. IIRC it had to do with ns conflicts with "some.*". Falling back to 1.5.1 solved it. I didn't investigate it. But it seems upgrading to 1.6.x will require some code change.
>
> Anyway I created a snapshot build with all these updates pulled in. All unit tests pass, and my basic fully distributed mode smoke tests pass, so I'm leaning toward a +1 for all.
>
> However, if anyone who is interested in seeing these upgrades merged has the time and resources to try it out, that would be a huge help to us.
>
> I'll post the binaries soon, when I have access to a real computer.
>
> -Taylor
>
>> On Mar 26, 2014, at 10:49 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>>
>>
>> I’m +1 on all of those (including the clojure version bump).
>>
>> We’ve been hit with the maven-clojure-plugin issue as well, with no real
>> way to diagnose.
>>
>> -brian
>>
>> ---
>> Brian O'Neill
>> Chief Technology Officer
>>
>> Health Market Science
>> The Science of Better Results
>> 2700 Horizon Drive · King of Prussia, PA · 19406
>> M: 215.588.6024 · @boneill42 <http://www.twitter.com/boneill42> ·
>> healthmarketscience.com
>>
>> This information transmitted in this email message is for the intended
>> recipient only and may contain confidential and/or privileged material. If
>> you received this email in error and are not the intended recipient, or
>> the person responsible to deliver it to the intended recipient, please
>> contact the sender at the email above and delete this email and any
>> attachments and destroy any copies thereof. Any review, retransmission,
>> dissemination, copying or other use of, or taking any action in reliance
>> upon, this information by persons or entities other than the intended
>> recipient is strictly prohibited.
>>
>>
>>
>>
>>
>>
>>
>>> On 3/26/14, 10:09 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>>>
>>> There are a number of email threads and JIRA issues regarding upgrading
>>> various storm dependencies, so I’d like to enumerate them and discuss
>>> them in one thread.
>>>
>>> Here’s the list so far:
>>>
>>> 1. Kryo/Carbonite (STORM-263)[1]
>>> 2. Clojure (STORM-265) [2]
>>> 3. commons-io (STORM-258) [3]
>>> 4. curator (STORM-252) [4]
>>> 5. http-client [5]
>>>
>>> I am +1 for all of the above, with the exception of #2, which I am +0
>>> only because I personally haven’t had a chance to do any testing with
>>> newer versions of clojure. I’d be interested to hear if anyone has done
>>> any testing with newer versions of clojure.
>>>
>>> I think we should at least consider a bump to clojure 1.5 since it
>>> includes Bobby Evan’s patch that fixes error output getting swallowed
>>> (this manifests itself as the maven-clojure-plugin failing without any
>>> useful information when there are certain AOT compilation issues — very
>>> annoying).
>>>
>>> - Taylor
>>>
>>> [1] https://issues.apache.org/jira/browse/STORM-263
>>> [2] https://issues.apache.org/jira/browse/STORM-265
>>> [3] https://issues.apache.org/jira/browse/STORM-258
>>> [4] https://issues.apache.org/jira/browse/STORM-252
>>> [5]
>>> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22
>>> E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>>
>>>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>>>>
>>>> Agreed. One of our guys got hung up on that just yesterday.
>>>> It isn’t hard to track down the assembly, but it might be worth making
>>>> it a bit easier.
>>>>
>>>> Also, I’d love to see STORM-263 included in the next release. It
>>>> seems like a quick win.
>>>>
>>>> -brian
>>>>
>>>> ---
>>>> Brian O'Neill
>>>> Chief Technology Officer
>>>>
>>>> Health Market Science
>>>> The Science of Better Results
>>>> 2700 Horizon Drive · King of Prussia, PA · 19406
>>>> M: 215.588.6024 · @boneill42 · healthmarketscience.com
>>>>
>>>> This information transmitted in this email message is for the intended
>>>> recipient only and may contain confidential and/or privileged material.
>>>> If you received this email in error and are not the intended recipient,
>>>> or the person responsible to deliver it to the intended recipient,
>>>> please contact the sender at the email above and delete this email and
>>>> any attachments and destroy any copies thereof. Any review,
>>>> retransmission, dissemination, copying or other use of, or taking any
>>>> action in reliance upon, this information by persons or entities other
>>>> than the intended recipient is strictly prohibited.
>>>>
>>>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>>>
>>>>> hi guys
>>>>>
>>>>> How about adding a bin/build_release.sh (http://build_release.sh)
>>>>> script in this 0.9.2-incubating release?
>>>>>
>>>>> Since some guys asked the question about building storm release
>>>>> package more than once in the mail list.
>>>>>
>>>>> --
>>>>> Best Regards!
>>>>>
>>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>>> Distributed Software Engineer
>>>>>
>>>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>>>>>> definition code
>>>>>> can committed with no +1 from a committer, right?
>>>>>>
>>>>>> You may want to look at Hadoop bylaws -
>>>>>> https://hadoop.apache.org/bylaws.html
>>>>>>
>>>>>> The code commits in Hadoop are consensus approval with minimum +1
>>>>>> from an
>>>>>> active committer and no veto. This has worked well in my experience.
>>>>>> It may
>>>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>>>> merging feature branches.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>> (mailto:ptgoetz@gmail.com)> wrote:
>>>>>>
>>>>>>> That brings up a good point that probably deserves a separate thread.
>>>>>>>
>>>>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>>>>
>>>>>>> For code changes, I've been operating under a "lazy consensus +2"
>>>>>>> model: 2
>>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>>>>>>> patch,
>>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>>>>>>> waiting
>>>>>>> for 3 binding votes and no vetoes.
>>>>>>>
>>>>>>> That's kind of a middle ground between the traditional code
>>>>>>> modification
>>>>>>> rule and lazy consensus [1].
>>>>>>>
>>>>>>> When wearing my "release manager" hat, I've also interpreted "code
>>>>>>> change"
>>>>>>> to mean "anything that alters the behavior of the software we
>>>>>>> produce." In
>>>>>>> terms of the build/packaging I've been a little looser. For large
>>>>>>> changes
>>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>>>>> changes I've committed directly -- I don't think we need to have a
>>>>>>> 3-day
>>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>>
>>>>>>> Anyway, it's something to think about. Sorry for hijacking the
>>>>>>> thread.
>>>>>>>
>>>>>>> +1 (again ;) )
>>>>>>>
>>>>>>> Taylor
>>>>>>>
>>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>>
>>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
>>>>>>>> (mailto:nathan@nathanmarz.com)> wrote:
>>>>>>>>
>>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there.
>>>>>>>> Just
>>>>>>>> one more vote needed by a committer.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
>>>>>>>>> (mailto:plucas@yelp.com)> wrote:
>>>>>>>>>
>>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>>>>>>>>> impossible
>>>>>>> to
>>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>>
>>>>>>>
>>>>>>> grouping.
>>>>>>>>>
>>>>>>>>> I'm not sure why there haven't been more reports of problems with
>>>>>>>>> it.
>>>>>>> Two
>>>>>>>>> possibilities I can think of are that we are using exclusively
>>>>>>>>> shell
>>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>>> classes--and
>>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> (thousands /
>>>>>>>>> sec, KB in size)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz
>>>>>>>>> <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Never mind... just found it.
>>>>>>>>>>
>>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>>>>>>> (mailto:ptgoetz@gmail.com)>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>>
>>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
>>>>>>>>>>> (mailto:derekd@yahoo-inc.com)>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>> it's as
>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>>>> reading
>>>>>>>>>> Taylor's mail.
>>>>>>>>>>>>
>>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>>>>>>>>>>>> (likelihood
>>>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> exceeds
>>>>>>>>>> 30.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Derek
>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> STORM-187.
>>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>>>>>>>>>>>>> it's a
>>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>>>>>>>>>>>>> taints
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> confidently
>>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>> it's as
>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>>> (Upgrade
>>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>>>>>>>>>>>>> 3.4.5
>>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> official
>>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>>>>>>>>>>>>> released
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ZK
>>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>>>>>>>>>>>>> how
>>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>>>>>>>>>>>>> i.e.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> whether
>>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> jumping
>>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>>>>>>>>>>>>> boat:
>>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>>>>>>>>>>>>> of ZK
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 3.3.4
>>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>>>>>>>>>>>>> basically
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> team
>>>>>>>>>> is
>>>>>>>>>>>>> working on that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>>>>>>>>>> seen it
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> in
>>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>>>>>>>>>> hinder
>>>>>>>>>> upgrading.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>>>>> being
>>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Taylor
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Patrick Lucas
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Twitter: @nathanmarz
>>>>>>>> http://nathanmarz.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> http://hortonworks.com/download/
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to
>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby
>>>>>> notified that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
I've submitted pull requests for most of these changes (I think I still need to submit one for http-client), where there wasn't already one.
The only problem I saw was with upgrading to clojure 1.6.0. IIRC it had to do with ns conflicts with "some.*". Falling back to 1.5.1 solved it. I didn't investigate it. But it seems upgrading to 1.6.x will require some code change.
Anyway I created a snapshot build with all these updates pulled in. All unit tests pass, and my basic fully distributed mode smoke tests pass, so I'm leaning toward a +1 for all.
However, if anyone who is interested in seeing these upgrades merged has the time and resources to try it out, that would be a huge help to us.
I'll post the binaries soon, when I have access to a real computer.
-Taylor
> On Mar 26, 2014, at 10:49 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>
>
> I’m +1 on all of those (including the clojure version bump).
>
> We’ve been hit with the maven-clojure-plugin issue as well, with no real
> way to diagnose.
>
> -brian
>
> ---
> Brian O'Neill
> Chief Technology Officer
>
> Health Market Science
> The Science of Better Results
> 2700 Horizon Drive · King of Prussia, PA · 19406
> M: 215.588.6024 · @boneill42 <http://www.twitter.com/boneill42> ·
> healthmarketscience.com
>
> This information transmitted in this email message is for the intended
> recipient only and may contain confidential and/or privileged material. If
> you received this email in error and are not the intended recipient, or
> the person responsible to deliver it to the intended recipient, please
> contact the sender at the email above and delete this email and any
> attachments and destroy any copies thereof. Any review, retransmission,
> dissemination, copying or other use of, or taking any action in reliance
> upon, this information by persons or entities other than the intended
> recipient is strictly prohibited.
>
>
>
>
>
>
>
>> On 3/26/14, 10:09 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>>
>> There are a number of email threads and JIRA issues regarding upgrading
>> various storm dependencies, so I’d like to enumerate them and discuss
>> them in one thread.
>>
>> Here’s the list so far:
>>
>> 1. Kryo/Carbonite (STORM-263)[1]
>> 2. Clojure (STORM-265) [2]
>> 3. commons-io (STORM-258) [3]
>> 4. curator (STORM-252) [4]
>> 5. http-client [5]
>>
>> I am +1 for all of the above, with the exception of #2, which I am +0
>> only because I personally haven’t had a chance to do any testing with
>> newer versions of clojure. I’d be interested to hear if anyone has done
>> any testing with newer versions of clojure.
>>
>> I think we should at least consider a bump to clojure 1.5 since it
>> includes Bobby Evan’s patch that fixes error output getting swallowed
>> (this manifests itself as the maven-clojure-plugin failing without any
>> useful information when there are certain AOT compilation issues — very
>> annoying).
>>
>> - Taylor
>>
>> [1] https://issues.apache.org/jira/browse/STORM-263
>> [2] https://issues.apache.org/jira/browse/STORM-265
>> [3] https://issues.apache.org/jira/browse/STORM-258
>> [4] https://issues.apache.org/jira/browse/STORM-252
>> [5]
>> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22
>> E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>
>>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>>>
>>> Agreed. One of our guys got hung up on that just yesterday.
>>> It isn’t hard to track down the assembly, but it might be worth making
>>> it a bit easier.
>>>
>>> Also, I’d love to see STORM-263 included in the next release. It
>>> seems like a quick win.
>>>
>>> -brian
>>>
>>> ---
>>> Brian O'Neill
>>> Chief Technology Officer
>>>
>>> Health Market Science
>>> The Science of Better Results
>>> 2700 Horizon Drive · King of Prussia, PA · 19406
>>> M: 215.588.6024 · @boneill42 · healthmarketscience.com
>>>
>>> This information transmitted in this email message is for the intended
>>> recipient only and may contain confidential and/or privileged material.
>>> If you received this email in error and are not the intended recipient,
>>> or the person responsible to deliver it to the intended recipient,
>>> please contact the sender at the email above and delete this email and
>>> any attachments and destroy any copies thereof. Any review,
>>> retransmission, dissemination, copying or other use of, or taking any
>>> action in reliance upon, this information by persons or entities other
>>> than the intended recipient is strictly prohibited.
>>>
>>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>>
>>>> hi guys
>>>>
>>>> How about adding a bin/build_release.sh (http://build_release.sh)
>>>> script in this 0.9.2-incubating release?
>>>>
>>>> Since some guys asked the question about building storm release
>>>> package more than once in the mail list.
>>>>
>>>> --
>>>> Best Regards!
>>>>
>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>> Distributed Software Engineer
>>>>
>>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>>>>> definition code
>>>>> can committed with no +1 from a committer, right?
>>>>>
>>>>> You may want to look at Hadoop bylaws -
>>>>> https://hadoop.apache.org/bylaws.html
>>>>>
>>>>> The code commits in Hadoop are consensus approval with minimum +1
>>>>> from an
>>>>> active committer and no veto. This has worked well in my experience.
>>>>> It may
>>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>>> merging feature branches.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>> (mailto:ptgoetz@gmail.com)> wrote:
>>>>>
>>>>>> That brings up a good point that probably deserves a separate thread.
>>>>>>
>>>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>>>
>>>>>> For code changes, I've been operating under a "lazy consensus +2"
>>>>>> model: 2
>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>>>>>> patch,
>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>>>>>> waiting
>>>>>> for 3 binding votes and no vetoes.
>>>>>>
>>>>>> That's kind of a middle ground between the traditional code
>>>>>> modification
>>>>>> rule and lazy consensus [1].
>>>>>>
>>>>>> When wearing my "release manager" hat, I've also interpreted "code
>>>>>> change"
>>>>>> to mean "anything that alters the behavior of the software we
>>>>>> produce." In
>>>>>> terms of the build/packaging I've been a little looser. For large
>>>>>> changes
>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>>>> changes I've committed directly -- I don't think we need to have a
>>>>>> 3-day
>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>
>>>>>> Anyway, it's something to think about. Sorry for hijacking the
>>>>>> thread.
>>>>>>
>>>>>> +1 (again ;) )
>>>>>>
>>>>>> Taylor
>>>>>>
>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>
>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
>>>>>>> (mailto:nathan@nathanmarz.com)> wrote:
>>>>>>>
>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there.
>>>>>>> Just
>>>>>>> one more vote needed by a committer.
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
>>>>>>>> (mailto:plucas@yelp.com)> wrote:
>>>>>>>>
>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>>>>>>>> impossible
>>>>>> to
>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>
>>>>>>
>>>>>> grouping.
>>>>>>>>
>>>>>>>> I'm not sure why there haven't been more reports of problems with
>>>>>>>> it.
>>>>>> Two
>>>>>>>> possibilities I can think of are that we are using exclusively
>>>>>>>> shell
>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>> classes--and
>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> (thousands /
>>>>>>>> sec, KB in size)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz
>>>>>>>> <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Never mind... just found it.
>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>>>>>> (mailto:ptgoetz@gmail.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
>>>>>>>>>> (mailto:derekd@yahoo-inc.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>> it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>>> reading
>>>>>>>>> Taylor's mail.
>>>>>>>>>>>
>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>>>>>>>>>>> (likelihood
>>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> exceeds
>>>>>>>>> 30.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Derek
>>>>>>>>>>>
>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> STORM-187.
>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>>>>>>>>>>>> it's a
>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>>>>>>>>>>>> taints
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> confidently
>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>> it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>> (Upgrade
>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>>>>>>>>>>>> 3.4.5
>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> official
>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>>>>>>>>>>>> released
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ZK
>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>>>>>>>>>>>> how
>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>>>>>>>>>>>> i.e.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> whether
>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> jumping
>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>
>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>>>>>>>>>>>> boat:
>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>>>>>>>>>>>> of ZK
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 3.3.4
>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>>>>>>>>>>>> basically
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>>>>>>>>>>>> Kafka
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> team
>>>>>>>>> is
>>>>>>>>>>>> working on that.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Michael
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>>>>>>>>> seen it
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> in
>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>>>>>>>>> hinder
>>>>>>>>> upgrading.
>>>>>>>>>>>>>
>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>>>> being
>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Taylor
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Patrick Lucas
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: @nathanmarz
>>>>>>> http://nathanmarz.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to
>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Brian O'Neill <bo...@alumni.brown.edu>.
I’m +1 on all of those (including the clojure version bump).
We’ve been hit with the maven-clojure-plugin issue as well, with no real
way to diagnose.
-brian
---
Brian O'Neill
Chief Technology Officer
Health Market Science
The Science of Better Results
2700 Horizon Drive ・ King of Prussia, PA ・ 19406
M: 215.588.6024 ・ @boneill42 <http://www.twitter.com/boneill42> ・
healthmarketscience.com
This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or
the person responsible to deliver it to the intended recipient, please
contact the sender at the email above and delete this email and any
attachments and destroy any copies thereof. Any review, retransmission,
dissemination, copying or other use of, or taking any action in reliance
upon, this information by persons or entities other than the intended
recipient is strictly prohibited.
On 3/26/14, 10:09 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>There are a number of email threads and JIRA issues regarding upgrading
>various storm dependencies, so I’d like to enumerate them and discuss
>them in one thread.
>
>Here’s the list so far:
>
>1. Kryo/Carbonite (STORM-263)[1]
>2. Clojure (STORM-265) [2]
>3. commons-io (STORM-258) [3]
>4. curator (STORM-252) [4]
>5. http-client [5]
>
>I am +1 for all of the above, with the exception of #2, which I am +0
>only because I personally haven’t had a chance to do any testing with
>newer versions of clojure. I’d be interested to hear if anyone has done
>any testing with newer versions of clojure.
>
>I think we should at least consider a bump to clojure 1.5 since it
>includes Bobby Evan’s patch that fixes error output getting swallowed
>(this manifests itself as the maven-clojure-plugin failing without any
>useful information when there are certain AOT compilation issues ― very
>annoying).
>
>- Taylor
>
>[1] https://issues.apache.org/jira/browse/STORM-263
>[2] https://issues.apache.org/jira/browse/STORM-265
>[3] https://issues.apache.org/jira/browse/STORM-258
>[4] https://issues.apache.org/jira/browse/STORM-252
>[5]
>http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22
>E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>
>On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>
>> Agreed. One of our guys got hung up on that just yesterday.
>> It isn’t hard to track down the assembly, but it might be worth making
>>it a bit easier.
>>
>> Also, I’d love to see STORM-263 included in the next release. It
>>seems like a quick win.
>>
>> -brian
>>
>> ---
>> Brian O'Neill
>> Chief Technology Officer
>>
>> Health Market Science
>> The Science of Better Results
>> 2700 Horizon Drive ・ King of Prussia, PA ・ 19406
>> M: 215.588.6024 ・ @boneill42 ・ healthmarketscience.com
>>
>> This information transmitted in this email message is for the intended
>>recipient only and may contain confidential and/or privileged material.
>>If you received this email in error and are not the intended recipient,
>>or the person responsible to deliver it to the intended recipient,
>>please contact the sender at the email above and delete this email and
>>any attachments and destroy any copies thereof. Any review,
>>retransmission, dissemination, copying or other use of, or taking any
>>action in reliance upon, this information by persons or entities other
>>than the intended recipient is strictly prohibited.
>>
>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>
>>> hi guys
>>>
>>> How about adding a bin/build_release.sh (http://build_release.sh)
>>>script in this 0.9.2-incubating release?
>>>
>>> Since some guys asked the question about building storm release
>>>package more than once in the mail list.
>>>
>>> --
>>> Best Regards!
>>>
>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>> Distributed Software Engineer
>>>
>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>>>>definition code
>>>> can committed with no +1 from a committer, right?
>>>>
>>>> You may want to look at Hadoop bylaws -
>>>> https://hadoop.apache.org/bylaws.html
>>>>
>>>> The code commits in Hadoop are consensus approval with minimum +1
>>>>from an
>>>> active committer and no veto. This has worked well in my experience.
>>>>It may
>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>> merging feature branches.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>(mailto:ptgoetz@gmail.com)> wrote:
>>>>
>>>>> That brings up a good point that probably deserves a separate thread.
>>>>>
>>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>>
>>>>> For code changes, I've been operating under a "lazy consensus +2"
>>>>>model: 2
>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>>>>>patch,
>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>>>>>waiting
>>>>> for 3 binding votes and no vetoes.
>>>>>
>>>>> That's kind of a middle ground between the traditional code
>>>>>modification
>>>>> rule and lazy consensus [1].
>>>>>
>>>>> When wearing my "release manager" hat, I've also interpreted "code
>>>>>change"
>>>>> to mean "anything that alters the behavior of the software we
>>>>>produce." In
>>>>> terms of the build/packaging I've been a little looser. For large
>>>>>changes
>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>>> changes I've committed directly -- I don't think we need to have a
>>>>>3-day
>>>>> vote on updating the CHANGELOG, for example.
>>>>>
>>>>> Anyway, it's something to think about. Sorry for hijacking the
>>>>>thread.
>>>>>
>>>>> +1 (again ;) )
>>>>>
>>>>> Taylor
>>>>>
>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>
>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
>>>>>>(mailto:nathan@nathanmarz.com)> wrote:
>>>>>>
>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there.
>>>>>>Just
>>>>>> one more vote needed by a committer.
>>>>>>
>>>>>>
>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
>>>>>>>(mailto:plucas@yelp.com)> wrote:
>>>>>>>
>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>>>>>>>impossible
>>>>> to
>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>
>>>>>
>>>>> grouping.
>>>>>>>
>>>>>>> I'm not sure why there haven't been more reports of problems with
>>>>>>>it.
>>>>> Two
>>>>>>> possibilities I can think of are that we are using exclusively
>>>>>>>shell
>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>> classes--and
>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>
>>>>>>
>>>>>
>>>>> (thousands /
>>>>>>> sec, KB in size)
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz
>>>>>>><ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Never mind... just found it.
>>>>>>>>
>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>>>>>(mailto:ptgoetz@gmail.com)>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>
>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
>>>>>>>>>(mailto:derekd@yahoo-inc.com)>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>it's as
>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>> reading
>>>>>>>> Taylor's mail.
>>>>>>>>>>
>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>>>>>>>>>>(likelihood
>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> exceeds
>>>>>>>> 30.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Derek
>>>>>>>>>>
>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> STORM-187.
>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>>>>>>>>>>>it's a
>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>>>>>>>>>>>taints
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> the
>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> confidently
>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>
>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>it's as
>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>
>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>> (Upgrade
>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>>>>>>>>>>>3.4.5
>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> official
>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>>>>>>>>>>>released
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ZK
>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>>>>>>>>>>>how
>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>>>>>>>>>>>i.e.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> whether
>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> jumping
>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>
>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>>>>>>>>>>>boat:
>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>>>>>>>>>>>of ZK
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> 3.3.4
>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>>>>>>>>>>>basically
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> the
>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>>>>>>>>>>>Kafka
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> team
>>>>>>>> is
>>>>>>>>>>> working on that.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Michael
>>>>>>>>>>>
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>>>>>>>>seen it
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> in
>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>>>>>>>>hinder
>>>>>>>> upgrading.
>>>>>>>>>>>>
>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>>> being
>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>
>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>
>>>>>>>>>>>> -Taylor
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Patrick Lucas
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Twitter: @nathanmarz
>>>>>> http://nathanmarz.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> http://hortonworks.com/download/
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>>entity to
>>>> which it is addressed and may contain information that is
>>>>confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>>>>reader
>>>> of this message is not the intended recipient, you are hereby
>>>>notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>>>immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Bobby Evans <ev...@yahoo-inc.com>.
I would also like to see STORM-294 go it makes using command line configs
a lot better.
- Bobby
On 4/25/14, 9:57 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>This release has been baking for a while and a number of important
>improvements and bug fixes have been merged into the master branch.
>
>There are a few dependency updates that are still pending:
>
>- STORM-265 (clojure)
>- STORM-252 (curator)
>- STORM-291 (http-client)
>
>Would committers be able to review the patches above and +1/-1 as
>appropriate?
>
>Any other patches that we should include in this release?
>
>- Taylor
>
>On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> There are a number of email threads and JIRA issues regarding upgrading
>>various storm dependencies, so I’d like to enumerate them and discuss
>>them in one thread.
>>
>> Here’s the list so far:
>>
>> 1. Kryo/Carbonite (STORM-263)[1]
>> 2. Clojure (STORM-265) [2]
>> 3. commons-io (STORM-258) [3]
>> 4. curator (STORM-252) [4]
>> 5. http-client [5]
>>
>> I am +1 for all of the above, with the exception of #2, which I am +0
>>only because I personally haven’t had a chance to do any testing with
>>newer versions of clojure. I’d be interested to hear if anyone has done
>>any testing with newer versions of clojure.
>>
>> I think we should at least consider a bump to clojure 1.5 since it
>>includes Bobby Evan’s patch that fixes error output getting swallowed
>>(this manifests itself as the maven-clojure-plugin failing without any
>>useful information when there are certain AOT compilation issues ― very
>>annoying).
>>
>> - Taylor
>>
>> [1] https://issues.apache.org/jira/browse/STORM-263
>> [2] https://issues.apache.org/jira/browse/STORM-265
>> [3] https://issues.apache.org/jira/browse/STORM-258
>> [4] https://issues.apache.org/jira/browse/STORM-252
>> [5]
>>http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B2
>>2E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>
>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu>
>>wrote:
>>
>>> Agreed. One of our guys got hung up on that just yesterday.
>>> It isn’t hard to track down the assembly, but it might be worth making
>>>it a bit easier.
>>>
>>> Also, I’d love to see STORM-263 included in the next release. It
>>>seems like a quick win.
>>>
>>> -brian
>>>
>>> ---
>>> Brian O'Neill
>>> Chief Technology Officer
>>>
>>> Health Market Science
>>> The Science of Better Results
>>> 2700 Horizon Drive ・ King of Prussia, PA ・ 19406
>>> M: 215.588.6024 ・ @boneill42 ・ healthmarketscience.com
>>>
>>> This information transmitted in this email message is for the intended
>>>recipient only and may contain confidential and/or privileged material.
>>>If you received this email in error and are not the intended recipient,
>>>or the person responsible to deliver it to the intended recipient,
>>>please contact the sender at the email above and delete this email and
>>>any attachments and destroy any copies thereof. Any review,
>>>retransmission, dissemination, copying or other use of, or taking any
>>>action in reliance upon, this information by persons or entities other
>>>than the intended recipient is strictly prohibited.
>>>
>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>
>>>> hi guys
>>>>
>>>> How about adding a bin/build_release.sh (http://build_release.sh)
>>>>script in this 0.9.2-incubating release?
>>>>
>>>> Since some guys asked the question about building storm release
>>>>package more than once in the mail list.
>>>>
>>>> --
>>>> Best Regards!
>>>>
>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>> Distributed Software Engineer
>>>>
>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>>>>>definition code
>>>>> can committed with no +1 from a committer, right?
>>>>>
>>>>> You may want to look at Hadoop bylaws -
>>>>> https://hadoop.apache.org/bylaws.html
>>>>>
>>>>> The code commits in Hadoop are consensus approval with minimum +1
>>>>>from an
>>>>> active committer and no veto. This has worked well in my experience.
>>>>>It may
>>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>>> merging feature branches.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>(mailto:ptgoetz@gmail.com)> wrote:
>>>>>
>>>>>> That brings up a good point that probably deserves a separate
>>>>>>thread.
>>>>>>
>>>>>> We should establish by-laws soon. Specifically a commit/merge
>>>>>>policy.
>>>>>>
>>>>>> For code changes, I've been operating under a "lazy consensus +2"
>>>>>>model: 2
>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>>>>>>patch,
>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>>>>>>waiting
>>>>>> for 3 binding votes and no vetoes.
>>>>>>
>>>>>> That's kind of a middle ground between the traditional code
>>>>>>modification
>>>>>> rule and lazy consensus [1].
>>>>>>
>>>>>> When wearing my "release manager" hat, I've also interpreted "code
>>>>>>change"
>>>>>> to mean "anything that alters the behavior of the software we
>>>>>>produce." In
>>>>>> terms of the build/packaging I've been a little looser. For large
>>>>>>changes
>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For
>>>>>>some
>>>>>> changes I've committed directly -- I don't think we need to have a
>>>>>>3-day
>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>
>>>>>> Anyway, it's something to think about. Sorry for hijacking the
>>>>>>thread.
>>>>>>
>>>>>> +1 (again ;) )
>>>>>>
>>>>>> Taylor
>>>>>>
>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>
>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
>>>>>>>(mailto:nathan@nathanmarz.com)> wrote:
>>>>>>>
>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in
>>>>>>>there. Just
>>>>>>> one more vote needed by a committer.
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
>>>>>>>>(mailto:plucas@yelp.com)> wrote:
>>>>>>>>
>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>>>>>>>>impossible
>>>>>> to
>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>
>>>>>>
>>>>>> grouping.
>>>>>>>>
>>>>>>>> I'm not sure why there haven't been more reports of problems with
>>>>>>>>it.
>>>>>> Two
>>>>>>>> possibilities I can think of are that we are using exclusively
>>>>>>>>shell
>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>> classes--and
>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> (thousands /
>>>>>>>> sec, KB in size)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz
>>>>>>>><ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Never mind... just found it.
>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>>>>>>(mailto:ptgoetz@gmail.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
>>>>>>>>>>(mailto:derekd@yahoo-inc.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>>> reading
>>>>>>>>> Taylor's mail.
>>>>>>>>>>>
>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>>>>>>>>>>>(likelihood
>>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> exceeds
>>>>>>>>> 30.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Derek
>>>>>>>>>>>
>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> STORM-187.
>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>>>>>>>>>>>>it's a
>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>>>>>>>>>>>>taints
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> confidently
>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>> (Upgrade
>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>>>>>>>>>>>>3.4.5
>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> official
>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>>>>>>>>>>>>released
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ZK
>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>>>>>>>>>>>>how
>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>>>>>>>>>>>>i.e.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> whether
>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> jumping
>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>
>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>>>>>>>>>>>>boat:
>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>>>>>>>>>>>>of ZK
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 3.3.4
>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>>>>>>>>>>>>basically
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>>>>>>>>>>>>Kafka
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> team
>>>>>>>>> is
>>>>>>>>>>>> working on that.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Michael
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>>>>>>>>>seen it
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> in
>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>>>>>>>>>hinder
>>>>>>>>> upgrading.
>>>>>>>>>>>>>
>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major
>>>>>>>>>>>>>difference
>>>>>> being
>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Taylor
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Patrick Lucas
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: @nathanmarz
>>>>>>> http://nathanmarz.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>entity to
>>>>> which it is addressed and may contain information that is
>>>>>confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>reader
>>>>> of this message is not the intended recipient, you are hereby
>>>>>notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>
>>>>> received this communication in error, please contact the sender
>>>>>immediately
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Bobby Evans <ev...@yahoo-inc.com>.
STORM-291 does not seem to have a pull request linked to it. But
https://github.com/apache/incubator-storm/pull/68 looks like it is the
right one.
―Bobby
On 4/25/14, 9:57 AM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>This release has been baking for a while and a number of important
>improvements and bug fixes have been merged into the master branch.
>
>There are a few dependency updates that are still pending:
>
>- STORM-265 (clojure)
>- STORM-252 (curator)
>- STORM-291 (http-client)
>
>Would committers be able to review the patches above and +1/-1 as
>appropriate?
>
>Any other patches that we should include in this release?
>
>- Taylor
>
>On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> There are a number of email threads and JIRA issues regarding upgrading
>>various storm dependencies, so I’d like to enumerate them and discuss
>>them in one thread.
>>
>> Here’s the list so far:
>>
>> 1. Kryo/Carbonite (STORM-263)[1]
>> 2. Clojure (STORM-265) [2]
>> 3. commons-io (STORM-258) [3]
>> 4. curator (STORM-252) [4]
>> 5. http-client [5]
>>
>> I am +1 for all of the above, with the exception of #2, which I am +0
>>only because I personally haven’t had a chance to do any testing with
>>newer versions of clojure. I’d be interested to hear if anyone has done
>>any testing with newer versions of clojure.
>>
>> I think we should at least consider a bump to clojure 1.5 since it
>>includes Bobby Evan’s patch that fixes error output getting swallowed
>>(this manifests itself as the maven-clojure-plugin failing without any
>>useful information when there are certain AOT compilation issues ― very
>>annoying).
>>
>> - Taylor
>>
>> [1] https://issues.apache.org/jira/browse/STORM-263
>> [2] https://issues.apache.org/jira/browse/STORM-265
>> [3] https://issues.apache.org/jira/browse/STORM-258
>> [4] https://issues.apache.org/jira/browse/STORM-252
>> [5]
>>http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B2
>>2E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>
>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu>
>>wrote:
>>
>>> Agreed. One of our guys got hung up on that just yesterday.
>>> It isn’t hard to track down the assembly, but it might be worth making
>>>it a bit easier.
>>>
>>> Also, I’d love to see STORM-263 included in the next release. It
>>>seems like a quick win.
>>>
>>> -brian
>>>
>>> ---
>>> Brian O'Neill
>>> Chief Technology Officer
>>>
>>> Health Market Science
>>> The Science of Better Results
>>> 2700 Horizon Drive ・ King of Prussia, PA ・ 19406
>>> M: 215.588.6024 ・ @boneill42 ・ healthmarketscience.com
>>>
>>> This information transmitted in this email message is for the intended
>>>recipient only and may contain confidential and/or privileged material.
>>>If you received this email in error and are not the intended recipient,
>>>or the person responsible to deliver it to the intended recipient,
>>>please contact the sender at the email above and delete this email and
>>>any attachments and destroy any copies thereof. Any review,
>>>retransmission, dissemination, copying or other use of, or taking any
>>>action in reliance upon, this information by persons or entities other
>>>than the intended recipient is strictly prohibited.
>>>
>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>
>>>> hi guys
>>>>
>>>> How about adding a bin/build_release.sh (http://build_release.sh)
>>>>script in this 0.9.2-incubating release?
>>>>
>>>> Since some guys asked the question about building storm release
>>>>package more than once in the mail list.
>>>>
>>>> --
>>>> Best Regards!
>>>>
>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>> Distributed Software Engineer
>>>>
>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>>>>>definition code
>>>>> can committed with no +1 from a committer, right?
>>>>>
>>>>> You may want to look at Hadoop bylaws -
>>>>> https://hadoop.apache.org/bylaws.html
>>>>>
>>>>> The code commits in Hadoop are consensus approval with minimum +1
>>>>>from an
>>>>> active committer and no veto. This has worked well in my experience.
>>>>>It may
>>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>>> merging feature branches.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>(mailto:ptgoetz@gmail.com)> wrote:
>>>>>
>>>>>> That brings up a good point that probably deserves a separate
>>>>>>thread.
>>>>>>
>>>>>> We should establish by-laws soon. Specifically a commit/merge
>>>>>>policy.
>>>>>>
>>>>>> For code changes, I've been operating under a "lazy consensus +2"
>>>>>>model: 2
>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>>>>>>patch,
>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>>>>>>waiting
>>>>>> for 3 binding votes and no vetoes.
>>>>>>
>>>>>> That's kind of a middle ground between the traditional code
>>>>>>modification
>>>>>> rule and lazy consensus [1].
>>>>>>
>>>>>> When wearing my "release manager" hat, I've also interpreted "code
>>>>>>change"
>>>>>> to mean "anything that alters the behavior of the software we
>>>>>>produce." In
>>>>>> terms of the build/packaging I've been a little looser. For large
>>>>>>changes
>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For
>>>>>>some
>>>>>> changes I've committed directly -- I don't think we need to have a
>>>>>>3-day
>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>
>>>>>> Anyway, it's something to think about. Sorry for hijacking the
>>>>>>thread.
>>>>>>
>>>>>> +1 (again ;) )
>>>>>>
>>>>>> Taylor
>>>>>>
>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>
>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
>>>>>>>(mailto:nathan@nathanmarz.com)> wrote:
>>>>>>>
>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in
>>>>>>>there. Just
>>>>>>> one more vote needed by a committer.
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
>>>>>>>>(mailto:plucas@yelp.com)> wrote:
>>>>>>>>
>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>>>>>>>>impossible
>>>>>> to
>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>
>>>>>>
>>>>>> grouping.
>>>>>>>>
>>>>>>>> I'm not sure why there haven't been more reports of problems with
>>>>>>>>it.
>>>>>> Two
>>>>>>>> possibilities I can think of are that we are using exclusively
>>>>>>>>shell
>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>> classes--and
>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> (thousands /
>>>>>>>> sec, KB in size)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz
>>>>>>>><ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Never mind... just found it.
>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com
>>>>>>>>>>(mailto:ptgoetz@gmail.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
>>>>>>>>>>(mailto:derekd@yahoo-inc.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>>> reading
>>>>>>>>> Taylor's mail.
>>>>>>>>>>>
>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>>>>>>>>>>>(likelihood
>>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> exceeds
>>>>>>>>> 30.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Derek
>>>>>>>>>>>
>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> STORM-187.
>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>>>>>>>>>>>>it's a
>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>>>>>>>>>>>>taints
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> confidently
>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>>>>>>>>>>it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>> (Upgrade
>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>>>>>>>>>>>>3.4.5
>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> official
>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>>>>>>>>>>>>released
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ZK
>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>>>>>>>>>>>>how
>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>>>>>>>>>>>>i.e.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> whether
>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> jumping
>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>
>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>>>>>>>>>>>>boat:
>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>>>>>>>>>>>>of ZK
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 3.3.4
>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>>>>>>>>>>>>basically
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>>>>>>>>>>>>Kafka
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> team
>>>>>>>>> is
>>>>>>>>>>>> working on that.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Michael
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>>>>>>>>>seen it
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> in
>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>>>>>>>>>hinder
>>>>>>>>> upgrading.
>>>>>>>>>>>>>
>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major
>>>>>>>>>>>>>difference
>>>>>> being
>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Taylor
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Patrick Lucas
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: @nathanmarz
>>>>>>> http://nathanmarz.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>entity to
>>>>> which it is addressed and may contain information that is
>>>>>confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>reader
>>>>> of this message is not the intended recipient, you are hereby
>>>>>notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>
>>>>> received this communication in error, please contact the sender
>>>>>immediately
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
I can’t give a specific date, but I would say “soon.”
Once a candidate is cut, it has to be voted on approved by the Storm PPMC, then the Incubator IPMC. Each of those votes takes at least 72 hours.
Right now we’re in a preparatory phase — determining what patches to include, and any TODO items left over from the last release. Once that’s complete we’ll cut a build and initiate a Storm PPMC vote.
- Taylor
On Apr 25, 2014, at 12:54 PM, Adrian Petrescu <ap...@gmail.com> wrote:
> Is 0.9.2 scheduled for release in the very near future?
>
>
> On Fri, Apr 25, 2014 at 12:29 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> We have a CHANGELOG.md file:
>> https://github.com/apache/incubator-storm/blob/master/CHANGELOG.md
>>
>>
>> On Apr 25, 2014, at 12:26 PM, Suresh Srinivas <su...@hortonworks.com>
>> wrote:
>>
>>> I would like STORM-295 to be picked up for the release.
>>>
>>> BTW how does Storm project maintain information on what changes are in a
>>> release. Is there anything similar to CHANGES.txt as maintained in some
>> of
>>> the projects?
>>>
>>>
>>> On Fri, Apr 25, 2014 at 7:57 AM, P. Taylor Goetz <pt...@gmail.com>
>> wrote:
>>>
>>>> This release has been baking for a while and a number of important
>>>> improvements and bug fixes have been merged into the master branch.
>>>>
>>>> There are a few dependency updates that are still pending:
>>>>
>>>> - STORM-265 (clojure)
>>>> - STORM-252 (curator)
>>>> - STORM-291 (http-client)
>>>>
>>>> Would committers be able to review the patches above and +1/-1 as
>>>> appropriate?
>>>>
>>>> Any other patches that we should include in this release?
>>>>
>>>> - Taylor
>>>>
>>>> On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com>
>> wrote:
>>>>
>>>>> There are a number of email threads and JIRA issues regarding upgrading
>>>> various storm dependencies, so I’d like to enumerate them and discuss
>> them
>>>> in one thread.
>>>>>
>>>>> Here’s the list so far:
>>>>>
>>>>> 1. Kryo/Carbonite (STORM-263)[1]
>>>>> 2. Clojure (STORM-265) [2]
>>>>> 3. commons-io (STORM-258) [3]
>>>>> 4. curator (STORM-252) [4]
>>>>> 5. http-client [5]
>>>>>
>>>>> I am +1 for all of the above, with the exception of #2, which I am +0
>>>> only because I personally haven’t had a chance to do any testing with
>> newer
>>>> versions of clojure. I’d be interested to hear if anyone has done any
>>>> testing with newer versions of clojure.
>>>>>
>>>>> I think we should at least consider a bump to clojure 1.5 since it
>>>> includes Bobby Evan’s patch that fixes error output getting swallowed
>> (this
>>>> manifests itself as the maven-clojure-plugin failing without any useful
>>>> information when there are certain AOT compilation issues — very
>> annoying).
>>>>>
>>>>> - Taylor
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/STORM-263
>>>>> [2] https://issues.apache.org/jira/browse/STORM-265
>>>>> [3] https://issues.apache.org/jira/browse/STORM-258
>>>>> [4] https://issues.apache.org/jira/browse/STORM-252
>>>>> [5]
>>>>
>> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>>>>
>>>>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu>
>>>> wrote:
>>>>>
>>>>>> Agreed. One of our guys got hung up on that just yesterday.
>>>>>> It isn’t hard to track down the assembly, but it might be worth making
>>>> it a bit easier.
>>>>>>
>>>>>> Also, I’d love to see STORM-263 included in the next release. It
>>>> seems like a quick win.
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>> ---
>>>>>> Brian O'Neill
>>>>>> Chief Technology Officer
>>>>>>
>>>>>> Health Market Science
>>>>>> The Science of Better Results
>>>>>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>>>>> M: 215.588.6024 • @boneill42 • healthmarketscience.com
>>>>>>
>>>>>> This information transmitted in this email message is for the intended
>>>> recipient only and may contain confidential and/or privileged material.
>> If
>>>> you received this email in error and are not the intended recipient, or
>> the
>>>> person responsible to deliver it to the intended recipient, please
>> contact
>>>> the sender at the email above and delete this email and any attachments
>> and
>>>> destroy any copies thereof. Any review, retransmission, dissemination,
>>>> copying or other use of, or taking any action in reliance upon, this
>>>> information by persons or entities other than the intended recipient is
>>>> strictly prohibited.
>>>>>>
>>>>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>>>>
>>>>>>> hi guys
>>>>>>>
>>>>>>> How about adding a bin/build_release.sh (http://build_release.sh)
>>>> script in this 0.9.2-incubating release?
>>>>>>>
>>>>>>> Since some guys asked the question about building storm release
>>>> package more than once in the mail list.
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards!
>>>>>>>
>>>>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>>>>> Distributed Software Engineer
>>>>>>>
>>>>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>>>> definition code
>>>>>>>> can committed with no +1 from a committer, right?
>>>>>>>>
>>>>>>>> You may want to look at Hadoop bylaws -
>>>>>>>> https://hadoop.apache.org/bylaws.html
>>>>>>>>
>>>>>>>> The code commits in Hadoop are consensus approval with minimum +1
>>>> from an
>>>>>>>> active committer and no veto. This has worked well in my experience.
>>>> It may
>>>>>>>> be a good idea to also adapt minimum 3 +1s from active committers
>> for
>>>>>>>> merging feature branches.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
>> (mailto:
>>>> ptgoetz@gmail.com)> wrote:
>>>>>>>>
>>>>>>>>> That brings up a good point that probably deserves a separate
>> thread.
>>>>>>>>>
>>>>>>>>> We should establish by-laws soon. Specifically a commit/merge
>> policy.
>>>>>>>>>
>>>>>>>>> For code changes, I've been operating under a "lazy consensus +2"
>>>> model: 2
>>>>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>>>> patch,
>>>>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>>>> waiting
>>>>>>>>> for 3 binding votes and no vetoes.
>>>>>>>>>
>>>>>>>>> That's kind of a middle ground between the traditional code
>>>> modification
>>>>>>>>> rule and lazy consensus [1].
>>>>>>>>>
>>>>>>>>> When wearing my "release manager" hat, I've also interpreted "code
>>>> change"
>>>>>>>>> to mean "anything that alters the behavior of the software we
>>>> produce." In
>>>>>>>>> terms of the build/packaging I've been a little looser. For large
>>>> changes
>>>>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For
>> some
>>>>>>>>> changes I've committed directly -- I don't think we need to have a
>>>> 3-day
>>>>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>>>>
>>>>>>>>> Anyway, it's something to think about. Sorry for hijacking the
>>>> thread.
>>>>>>>>>
>>>>>>>>> +1 (again ;) )
>>>>>>>>>
>>>>>>>>> Taylor
>>>>>>>>>
>>>>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>>>>
>>>>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
>> (mailto:
>>>> nathan@nathanmarz.com)> wrote:
>>>>>>>>>>
>>>>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in
>>>> there. Just
>>>>>>>>>> one more vote needed by a committer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
>> (mailto:
>>>> plucas@yelp.com)> wrote:
>>>>>>>>>>>
>>>>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>>>> impossible
>>>>>>>>> to
>>>>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> grouping.
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure why there haven't been more reports of problems with
>>>> it.
>>>>>>>>> Two
>>>>>>>>>>> possibilities I can think of are that we are using exclusively
>>>> shell
>>>>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>>>>> classes--and
>>>>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> (thousands /
>>>>>>>>>>> sec, KB in size)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <
>>>> ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Never mind... just found it.
>>>>>>>>>>>>
>>>>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <
>> ptgoetz@gmail.com(mailto:
>>>> ptgoetz@gmail.com)>
>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
>> (mailto:
>>>> derekd@yahoo-inc.com)>
>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As I said above, this fix is the most important in my
>> opinion.
>>>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>> it's as
>>>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yeah, we found it recently, and I created it this morning
>> after
>>>>>>>>>>> reading
>>>>>>>>>>>> Taylor's mail.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>>>> (likelihood
>>>>>>>>>>>> depends on configuration), but we will hit STORM-259 when
>> retries
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> exceeds
>>>>>>>>>>>> 30.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Derek
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> STORM-187.
>>>>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>>>> it's a
>>>>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>>>> taints
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> the
>>>>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> confidently
>>>>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As I said above, this fix is the most important in my
>> opinion.
>>>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>>>> it's as
>>>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>>>>> (Upgrade
>>>>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>>>> 3.4.5
>>>>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> official
>>>>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>>>> released
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ZK
>>>>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>>>> how
>>>>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>>>> i.e.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> whether
>>>>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would
>> be
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> jumping
>>>>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>>>> boat:
>>>>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>>>> of ZK
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 3.3.4
>>>>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>>>> basically
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> the
>>>>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>>>> Kafka
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> team
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> working on that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Michael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>>>> seen it
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> in
>>>>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>>>> hinder
>>>>>>>>>>>> upgrading.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major
>> difference
>>>>>>>>> being
>>>>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Taylor
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Patrick Lucas
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Twitter: @nathanmarz
>>>>>>>>>> http://nathanmarz.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> http://hortonworks.com/download/
>>>>>>>>
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to
>>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby
>>>> notified that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>>> received this communication in error, please contact the sender
>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> http://hortonworks.com/download/
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>>
>>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Adrian Petrescu <ap...@gmail.com>.
Is 0.9.2 scheduled for release in the very near future?
On Fri, Apr 25, 2014 at 12:29 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> We have a CHANGELOG.md file:
> https://github.com/apache/incubator-storm/blob/master/CHANGELOG.md
>
>
> On Apr 25, 2014, at 12:26 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > I would like STORM-295 to be picked up for the release.
> >
> > BTW how does Storm project maintain information on what changes are in a
> > release. Is there anything similar to CHANGES.txt as maintained in some
> of
> > the projects?
> >
> >
> > On Fri, Apr 25, 2014 at 7:57 AM, P. Taylor Goetz <pt...@gmail.com>
> wrote:
> >
> >> This release has been baking for a while and a number of important
> >> improvements and bug fixes have been merged into the master branch.
> >>
> >> There are a few dependency updates that are still pending:
> >>
> >> - STORM-265 (clojure)
> >> - STORM-252 (curator)
> >> - STORM-291 (http-client)
> >>
> >> Would committers be able to review the patches above and +1/-1 as
> >> appropriate?
> >>
> >> Any other patches that we should include in this release?
> >>
> >> - Taylor
> >>
> >> On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com>
> wrote:
> >>
> >>> There are a number of email threads and JIRA issues regarding upgrading
> >> various storm dependencies, so I’d like to enumerate them and discuss
> them
> >> in one thread.
> >>>
> >>> Here’s the list so far:
> >>>
> >>> 1. Kryo/Carbonite (STORM-263)[1]
> >>> 2. Clojure (STORM-265) [2]
> >>> 3. commons-io (STORM-258) [3]
> >>> 4. curator (STORM-252) [4]
> >>> 5. http-client [5]
> >>>
> >>> I am +1 for all of the above, with the exception of #2, which I am +0
> >> only because I personally haven’t had a chance to do any testing with
> newer
> >> versions of clojure. I’d be interested to hear if anyone has done any
> >> testing with newer versions of clojure.
> >>>
> >>> I think we should at least consider a bump to clojure 1.5 since it
> >> includes Bobby Evan’s patch that fixes error output getting swallowed
> (this
> >> manifests itself as the maven-clojure-plugin failing without any useful
> >> information when there are certain AOT compilation issues — very
> annoying).
> >>>
> >>> - Taylor
> >>>
> >>> [1] https://issues.apache.org/jira/browse/STORM-263
> >>> [2] https://issues.apache.org/jira/browse/STORM-265
> >>> [3] https://issues.apache.org/jira/browse/STORM-258
> >>> [4] https://issues.apache.org/jira/browse/STORM-252
> >>> [5]
> >>
> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
> >>>
> >>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu>
> >> wrote:
> >>>
> >>>> Agreed. One of our guys got hung up on that just yesterday.
> >>>> It isn’t hard to track down the assembly, but it might be worth making
> >> it a bit easier.
> >>>>
> >>>> Also, I’d love to see STORM-263 included in the next release. It
> >> seems like a quick win.
> >>>>
> >>>> -brian
> >>>>
> >>>> ---
> >>>> Brian O'Neill
> >>>> Chief Technology Officer
> >>>>
> >>>> Health Market Science
> >>>> The Science of Better Results
> >>>> 2700 Horizon Drive • King of Prussia, PA • 19406
> >>>> M: 215.588.6024 • @boneill42 • healthmarketscience.com
> >>>>
> >>>> This information transmitted in this email message is for the intended
> >> recipient only and may contain confidential and/or privileged material.
> If
> >> you received this email in error and are not the intended recipient, or
> the
> >> person responsible to deliver it to the intended recipient, please
> contact
> >> the sender at the email above and delete this email and any attachments
> and
> >> destroy any copies thereof. Any review, retransmission, dissemination,
> >> copying or other use of, or taking any action in reliance upon, this
> >> information by persons or entities other than the intended recipient is
> >> strictly prohibited.
> >>>>
> >>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
> >>>>
> >>>>> hi guys
> >>>>>
> >>>>> How about adding a bin/build_release.sh (http://build_release.sh)
> >> script in this 0.9.2-incubating release?
> >>>>>
> >>>>> Since some guys asked the question about building storm release
> >> package more than once in the mail list.
> >>>>>
> >>>>> --
> >>>>> Best Regards!
> >>>>>
> >>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
> >>>>> Distributed Software Engineer
> >>>>>
> >>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
> >>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
> >> definition code
> >>>>>> can committed with no +1 from a committer, right?
> >>>>>>
> >>>>>> You may want to look at Hadoop bylaws -
> >>>>>> https://hadoop.apache.org/bylaws.html
> >>>>>>
> >>>>>> The code commits in Hadoop are consensus approval with minimum +1
> >> from an
> >>>>>> active committer and no veto. This has worked well in my experience.
> >> It may
> >>>>>> be a good idea to also adapt minimum 3 +1s from active committers
> for
> >>>>>> merging feature branches.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com
> (mailto:
> >> ptgoetz@gmail.com)> wrote:
> >>>>>>
> >>>>>>> That brings up a good point that probably deserves a separate
> thread.
> >>>>>>>
> >>>>>>> We should establish by-laws soon. Specifically a commit/merge
> policy.
> >>>>>>>
> >>>>>>> For code changes, I've been operating under a "lazy consensus +2"
> >> model: 2
> >>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
> >> patch,
> >>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
> >> waiting
> >>>>>>> for 3 binding votes and no vetoes.
> >>>>>>>
> >>>>>>> That's kind of a middle ground between the traditional code
> >> modification
> >>>>>>> rule and lazy consensus [1].
> >>>>>>>
> >>>>>>> When wearing my "release manager" hat, I've also interpreted "code
> >> change"
> >>>>>>> to mean "anything that alters the behavior of the software we
> >> produce." In
> >>>>>>> terms of the build/packaging I've been a little looser. For large
> >> changes
> >>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For
> some
> >>>>>>> changes I've committed directly -- I don't think we need to have a
> >> 3-day
> >>>>>>> vote on updating the CHANGELOG, for example.
> >>>>>>>
> >>>>>>> Anyway, it's something to think about. Sorry for hijacking the
> >> thread.
> >>>>>>>
> >>>>>>> +1 (again ;) )
> >>>>>>>
> >>>>>>> Taylor
> >>>>>>>
> >>>>>>> [1] https://www.apache.org/foundation/voting.html
> >>>>>>>
> >>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com
> (mailto:
> >> nathan@nathanmarz.com)> wrote:
> >>>>>>>>
> >>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in
> >> there. Just
> >>>>>>>> one more vote needed by a committer.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com
> (mailto:
> >> plucas@yelp.com)> wrote:
> >>>>>>>>>
> >>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
> >> impossible
> >>>>>>> to
> >>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
> >>>>>>>>
> >>>>>>>
> >>>>>>> grouping.
> >>>>>>>>>
> >>>>>>>>> I'm not sure why there haven't been more reports of problems with
> >> it.
> >>>>>>> Two
> >>>>>>>>> possibilities I can think of are that we are using exclusively
> >> shell
> >>>>>>>>> components--perhaps there's a root-cause bug in those component
> >>>>>>>>> classes--and
> >>>>>>>>> that we are dealing with a high volume stream of large tuples.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> (thousands /
> >>>>>>>>> sec, KB in size)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <
> >> ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Never mind... just found it.
> >>>>>>>>>>
> >>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <
> ptgoetz@gmail.com(mailto:
> >> ptgoetz@gmail.com)>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Derek do you have an idea for a fix?
> >>>>>>>>>>>
> >>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com
> (mailto:
> >> derekd@yahoo-inc.com)>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>> As I said above, this fix is the most important in my
> opinion.
> >>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
> >> it's as
> >>>>>>>>>>>>> important as STORM-187 or not.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yeah, we found it recently, and I created it this morning
> after
> >>>>>>>>> reading
> >>>>>>>>>> Taylor's mail.
> >>>>>>>>>>>>
> >>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
> >> (likelihood
> >>>>>>>>>> depends on configuration), but we will hit STORM-259 when
> retries
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> exceeds
> >>>>>>>>>> 30.
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Derek
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
> >>>>>>>>>>>>> On my side the most important change is, as you point out,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> STORM-187.
> >>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
> >> it's a
> >>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
> >> taints
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> confidently
> >>>>>>>>>>>>> bury ZeroMQ the better. :-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> As I said above, this fix is the most important in my
> opinion.
> >>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
> >> it's as
> >>>>>>>>>>>>> important as STORM-187 or not.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
> >>>>>>> (Upgrade
> >>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
> >> 3.4.5
> >>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> official
> >>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
> >> released
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> ZK
> >>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
> >> how
> >>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
> >> i.e.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> whether
> >>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would
> be
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> jumping
> >>>>>>>>>>>>> the gun or a safe move.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
> >> boat:
> >>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
> >> of ZK
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 3.3.4
> >>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
> >> basically
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
> >> Kafka
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> team
> >>>>>>>>>> is
> >>>>>>>>>>>>> working on that.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Michael
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
> >>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
> >> seen it
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> in
> >>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
> >> hinder
> >>>>>>>>>> upgrading.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major
> difference
> >>>>>>> being
> >>>>>>>>>> the zookeeper update and the patch mentioned above.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Taylor
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Patrick Lucas
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Twitter: @nathanmarz
> >>>>>>>> http://nathanmarz.com
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> http://hortonworks.com/download/
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >> entity to
> >>>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> >> notified that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>>> received this communication in error, please contact the sender
> >> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
We have a CHANGELOG.md file: https://github.com/apache/incubator-storm/blob/master/CHANGELOG.md
On Apr 25, 2014, at 12:26 PM, Suresh Srinivas <su...@hortonworks.com> wrote:
> I would like STORM-295 to be picked up for the release.
>
> BTW how does Storm project maintain information on what changes are in a
> release. Is there anything similar to CHANGES.txt as maintained in some of
> the projects?
>
>
> On Fri, Apr 25, 2014 at 7:57 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> This release has been baking for a while and a number of important
>> improvements and bug fixes have been merged into the master branch.
>>
>> There are a few dependency updates that are still pending:
>>
>> - STORM-265 (clojure)
>> - STORM-252 (curator)
>> - STORM-291 (http-client)
>>
>> Would committers be able to review the patches above and +1/-1 as
>> appropriate?
>>
>> Any other patches that we should include in this release?
>>
>> - Taylor
>>
>> On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>>
>>> There are a number of email threads and JIRA issues regarding upgrading
>> various storm dependencies, so I’d like to enumerate them and discuss them
>> in one thread.
>>>
>>> Here’s the list so far:
>>>
>>> 1. Kryo/Carbonite (STORM-263)[1]
>>> 2. Clojure (STORM-265) [2]
>>> 3. commons-io (STORM-258) [3]
>>> 4. curator (STORM-252) [4]
>>> 5. http-client [5]
>>>
>>> I am +1 for all of the above, with the exception of #2, which I am +0
>> only because I personally haven’t had a chance to do any testing with newer
>> versions of clojure. I’d be interested to hear if anyone has done any
>> testing with newer versions of clojure.
>>>
>>> I think we should at least consider a bump to clojure 1.5 since it
>> includes Bobby Evan’s patch that fixes error output getting swallowed (this
>> manifests itself as the maven-clojure-plugin failing without any useful
>> information when there are certain AOT compilation issues — very annoying).
>>>
>>> - Taylor
>>>
>>> [1] https://issues.apache.org/jira/browse/STORM-263
>>> [2] https://issues.apache.org/jira/browse/STORM-265
>>> [3] https://issues.apache.org/jira/browse/STORM-258
>>> [4] https://issues.apache.org/jira/browse/STORM-252
>>> [5]
>> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>>
>>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu>
>> wrote:
>>>
>>>> Agreed. One of our guys got hung up on that just yesterday.
>>>> It isn’t hard to track down the assembly, but it might be worth making
>> it a bit easier.
>>>>
>>>> Also, I’d love to see STORM-263 included in the next release. It
>> seems like a quick win.
>>>>
>>>> -brian
>>>>
>>>> ---
>>>> Brian O'Neill
>>>> Chief Technology Officer
>>>>
>>>> Health Market Science
>>>> The Science of Better Results
>>>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>>> M: 215.588.6024 • @boneill42 • healthmarketscience.com
>>>>
>>>> This information transmitted in this email message is for the intended
>> recipient only and may contain confidential and/or privileged material. If
>> you received this email in error and are not the intended recipient, or the
>> person responsible to deliver it to the intended recipient, please contact
>> the sender at the email above and delete this email and any attachments and
>> destroy any copies thereof. Any review, retransmission, dissemination,
>> copying or other use of, or taking any action in reliance upon, this
>> information by persons or entities other than the intended recipient is
>> strictly prohibited.
>>>>
>>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>>
>>>>> hi guys
>>>>>
>>>>> How about adding a bin/build_release.sh (http://build_release.sh)
>> script in this 0.9.2-incubating release?
>>>>>
>>>>> Since some guys asked the question about building storm release
>> package more than once in the mail list.
>>>>>
>>>>> --
>>>>> Best Regards!
>>>>>
>>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>>> Distributed Software Engineer
>>>>>
>>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this
>> definition code
>>>>>> can committed with no +1 from a committer, right?
>>>>>>
>>>>>> You may want to look at Hadoop bylaws -
>>>>>> https://hadoop.apache.org/bylaws.html
>>>>>>
>>>>>> The code commits in Hadoop are consensus approval with minimum +1
>> from an
>>>>>> active committer and no veto. This has worked well in my experience.
>> It may
>>>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>>>> merging feature branches.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com(mailto:
>> ptgoetz@gmail.com)> wrote:
>>>>>>
>>>>>>> That brings up a good point that probably deserves a separate thread.
>>>>>>>
>>>>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>>>>
>>>>>>> For code changes, I've been operating under a "lazy consensus +2"
>> model: 2
>>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the
>> patch,
>>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
>> waiting
>>>>>>> for 3 binding votes and no vetoes.
>>>>>>>
>>>>>>> That's kind of a middle ground between the traditional code
>> modification
>>>>>>> rule and lazy consensus [1].
>>>>>>>
>>>>>>> When wearing my "release manager" hat, I've also interpreted "code
>> change"
>>>>>>> to mean "anything that alters the behavior of the software we
>> produce." In
>>>>>>> terms of the build/packaging I've been a little looser. For large
>> changes
>>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>>>>> changes I've committed directly -- I don't think we need to have a
>> 3-day
>>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>>
>>>>>>> Anyway, it's something to think about. Sorry for hijacking the
>> thread.
>>>>>>>
>>>>>>> +1 (again ;) )
>>>>>>>
>>>>>>> Taylor
>>>>>>>
>>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>>
>>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com(mailto:
>> nathan@nathanmarz.com)> wrote:
>>>>>>>>
>>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in
>> there. Just
>>>>>>>> one more vote needed by a committer.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com(mailto:
>> plucas@yelp.com)> wrote:
>>>>>>>>>
>>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
>> impossible
>>>>>>> to
>>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>>
>>>>>>>
>>>>>>> grouping.
>>>>>>>>>
>>>>>>>>> I'm not sure why there haven't been more reports of problems with
>> it.
>>>>>>> Two
>>>>>>>>> possibilities I can think of are that we are using exclusively
>> shell
>>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>>> classes--and
>>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> (thousands /
>>>>>>>>> sec, KB in size)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <
>> ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Never mind... just found it.
>>>>>>>>>>
>>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com(mailto:
>> ptgoetz@gmail.com)>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>>
>>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com(mailto:
>> derekd@yahoo-inc.com)>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>> it's as
>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>>>> reading
>>>>>>>>>> Taylor's mail.
>>>>>>>>>>>>
>>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
>> (likelihood
>>>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> exceeds
>>>>>>>>>> 30.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Derek
>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> STORM-187.
>>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
>> it's a
>>>>>>>>>>>>> stability problem. The secondary aspect is that this issue
>> taints
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> confidently
>>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
>> it's as
>>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>>> (Upgrade
>>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
>> 3.4.5
>>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> official
>>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
>> released
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ZK
>>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
>> how
>>>>>>>>>>>>> confident we are that the code in STORM-252 actually works,
>> i.e.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> whether
>>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> jumping
>>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
>> boat:
>>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
>> of ZK
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 3.3.4
>>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
>> basically
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
>> Kafka
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> team
>>>>>>>>>> is
>>>>>>>>>>>>> working on that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
>> seen it
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> in
>>>>>>>>>> the wild, but I've heard of a few cases where it was enough to
>> hinder
>>>>>>>>>> upgrading.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>>>>> being
>>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Taylor
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Patrick Lucas
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Twitter: @nathanmarz
>>>>>>>> http://nathanmarz.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> http://hortonworks.com/download/
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>> entity to
>>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>> reader
>>>>>> of this message is not the intended recipient, you are hereby
>> notified that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>> received this communication in error, please contact the sender
>> immediately
>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Suresh Srinivas <su...@hortonworks.com>.
I would like STORM-295 to be picked up for the release.
BTW how does Storm project maintain information on what changes are in a
release. Is there anything similar to CHANGES.txt as maintained in some of
the projects?
On Fri, Apr 25, 2014 at 7:57 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> This release has been baking for a while and a number of important
> improvements and bug fixes have been merged into the master branch.
>
> There are a few dependency updates that are still pending:
>
> - STORM-265 (clojure)
> - STORM-252 (curator)
> - STORM-291 (http-client)
>
> Would committers be able to review the patches above and +1/-1 as
> appropriate?
>
> Any other patches that we should include in this release?
>
> - Taylor
>
> On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
> > There are a number of email threads and JIRA issues regarding upgrading
> various storm dependencies, so I’d like to enumerate them and discuss them
> in one thread.
> >
> > Here’s the list so far:
> >
> > 1. Kryo/Carbonite (STORM-263)[1]
> > 2. Clojure (STORM-265) [2]
> > 3. commons-io (STORM-258) [3]
> > 4. curator (STORM-252) [4]
> > 5. http-client [5]
> >
> > I am +1 for all of the above, with the exception of #2, which I am +0
> only because I personally haven’t had a chance to do any testing with newer
> versions of clojure. I’d be interested to hear if anyone has done any
> testing with newer versions of clojure.
> >
> > I think we should at least consider a bump to clojure 1.5 since it
> includes Bobby Evan’s patch that fixes error output getting swallowed (this
> manifests itself as the maven-clojure-plugin failing without any useful
> information when there are certain AOT compilation issues — very annoying).
> >
> > - Taylor
> >
> > [1] https://issues.apache.org/jira/browse/STORM-263
> > [2] https://issues.apache.org/jira/browse/STORM-265
> > [3] https://issues.apache.org/jira/browse/STORM-258
> > [4] https://issues.apache.org/jira/browse/STORM-252
> > [5]
> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
> >
> > On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu>
> wrote:
> >
> >> Agreed. One of our guys got hung up on that just yesterday.
> >> It isn’t hard to track down the assembly, but it might be worth making
> it a bit easier.
> >>
> >> Also, I’d love to see STORM-263 included in the next release. It
> seems like a quick win.
> >>
> >> -brian
> >>
> >> ---
> >> Brian O'Neill
> >> Chief Technology Officer
> >>
> >> Health Market Science
> >> The Science of Better Results
> >> 2700 Horizon Drive • King of Prussia, PA • 19406
> >> M: 215.588.6024 • @boneill42 • healthmarketscience.com
> >>
> >> This information transmitted in this email message is for the intended
> recipient only and may contain confidential and/or privileged material. If
> you received this email in error and are not the intended recipient, or the
> person responsible to deliver it to the intended recipient, please contact
> the sender at the email above and delete this email and any attachments and
> destroy any copies thereof. Any review, retransmission, dissemination,
> copying or other use of, or taking any action in reliance upon, this
> information by persons or entities other than the intended recipient is
> strictly prohibited.
> >>
> >> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
> >>
> >>> hi guys
> >>>
> >>> How about adding a bin/build_release.sh (http://build_release.sh)
> script in this 0.9.2-incubating release?
> >>>
> >>> Since some guys asked the question about building storm release
> package more than once in the mail list.
> >>>
> >>> --
> >>> Best Regards!
> >>>
> >>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
> >>> Distributed Software Engineer
> >>>
> >>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
> >>>> Taylor, I am not very clear on "Lazy consensus +2". By this
> definition code
> >>>> can committed with no +1 from a committer, right?
> >>>>
> >>>> You may want to look at Hadoop bylaws -
> >>>> https://hadoop.apache.org/bylaws.html
> >>>>
> >>>> The code commits in Hadoop are consensus approval with minimum +1
> from an
> >>>> active committer and no veto. This has worked well in my experience.
> It may
> >>>> be a good idea to also adapt minimum 3 +1s from active committers for
> >>>> merging feature branches.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com(mailto:
> ptgoetz@gmail.com)> wrote:
> >>>>
> >>>>> That brings up a good point that probably deserves a separate thread.
> >>>>>
> >>>>> We should establish by-laws soon. Specifically a commit/merge policy.
> >>>>>
> >>>>> For code changes, I've been operating under a "lazy consensus +2"
> model: 2
> >>>>> committer +1 votes and no vetoes (-1). If a committer submits the
> patch,
> >>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been
> waiting
> >>>>> for 3 binding votes and no vetoes.
> >>>>>
> >>>>> That's kind of a middle ground between the traditional code
> modification
> >>>>> rule and lazy consensus [1].
> >>>>>
> >>>>> When wearing my "release manager" hat, I've also interpreted "code
> change"
> >>>>> to mean "anything that alters the behavior of the software we
> produce." In
> >>>>> terms of the build/packaging I've been a little looser. For large
> changes
> >>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
> >>>>> changes I've committed directly -- I don't think we need to have a
> 3-day
> >>>>> vote on updating the CHANGELOG, for example.
> >>>>>
> >>>>> Anyway, it's something to think about. Sorry for hijacking the
> thread.
> >>>>>
> >>>>> +1 (again ;) )
> >>>>>
> >>>>> Taylor
> >>>>>
> >>>>> [1] https://www.apache.org/foundation/voting.html
> >>>>>
> >>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com(mailto:
> nathan@nathanmarz.com)> wrote:
> >>>>>>
> >>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in
> there. Just
> >>>>>> one more vote needed by a committer.
> >>>>>>
> >>>>>>
> >>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com(mailto:
> plucas@yelp.com)> wrote:
> >>>>>>>
> >>>>>>> A fix STORM-120 would be greatly appreciated. It's making it
> impossible
> >>>>> to
> >>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
> >>>>>>
> >>>>>
> >>>>> grouping.
> >>>>>>>
> >>>>>>> I'm not sure why there haven't been more reports of problems with
> it.
> >>>>> Two
> >>>>>>> possibilities I can think of are that we are using exclusively
> shell
> >>>>>>> components--perhaps there's a root-cause bug in those component
> >>>>>>> classes--and
> >>>>>>> that we are dealing with a high volume stream of large tuples.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> (thousands /
> >>>>>>> sec, KB in size)
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <
> ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Never mind... just found it.
> >>>>>>>>
> >>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com(mailto:
> ptgoetz@gmail.com)>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Derek do you have an idea for a fix?
> >>>>>>>>>
> >>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com(mailto:
> derekd@yahoo-inc.com)>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>>> As I said above, this fix is the most important in my opinion.
> >>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
> it's as
> >>>>>>>>>>> important as STORM-187 or not.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Yeah, we found it recently, and I created it this morning after
> >>>>>>> reading
> >>>>>>>> Taylor's mail.
> >>>>>>>>>>
> >>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries
> (likelihood
> >>>>>>>> depends on configuration), but we will hit STORM-259 when retries
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> exceeds
> >>>>>>>> 30.
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Derek
> >>>>>>>>>>
> >>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
> >>>>>>>>>>> On my side the most important change is, as you point out,
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> STORM-187.
> >>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because
> it's a
> >>>>>>>>>>> stability problem. The secondary aspect is that this issue
> taints
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> the
> >>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> confidently
> >>>>>>>>>>> bury ZeroMQ the better. :-)
> >>>>>>>>>>>
> >>>>>>>>>>> As I said above, this fix is the most important in my opinion.
> >>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether
> it's as
> >>>>>>>>>>> important as STORM-187 or not.
> >>>>>>>>>>>
> >>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
> >>>>> (Upgrade
> >>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK
> 3.4.5
> >>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> official
> >>>>>>>>>>> Storm support for the latest ZK version (ok, the recently
> released
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> ZK
> >>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know
> how
> >>>>>>>>>>> confident we are that the code in STORM-252 actually works,
> i.e.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> whether
> >>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> jumping
> >>>>>>>>>>> the gun or a safe move.
> >>>>>>>>>>>
> >>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same
> boat:
> >>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use
> of ZK
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> 3.3.4
> >>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's
> basically
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> the
> >>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the
> Kafka
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> team
> >>>>>>>> is
> >>>>>>>>>>> working on that.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Michael
> >>>>>>>>>>>
> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
> >>>>>>>>>>>> I'd like to get this discussion started, largely because the
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not
> seen it
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> in
> >>>>>>>> the wild, but I've heard of a few cases where it was enough to
> hinder
> >>>>>>>> upgrading.
> >>>>>>>>>>>>
> >>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
> >>>>> being
> >>>>>>>> the zookeeper update and the patch mentioned above.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Any thoughts on other PRs or patches to include?
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Taylor
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Patrick Lucas
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Twitter: @nathanmarz
> >>>>>> http://nathanmarz.com
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby
> notified that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> immediately
> >>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
>
>
--
http://hortonworks.com/download/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
One more to consider is STORM-269, which I just submitted a pr for.
The fact that the log viewer will happily serve any readable file on the filesystem is a security hole we should plug.
- Taylor
On Apr 25, 2014, at 10:57 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> This release has been baking for a while and a number of important improvements and bug fixes have been merged into the master branch.
>
> There are a few dependency updates that are still pending:
>
> - STORM-265 (clojure)
> - STORM-252 (curator)
> - STORM-291 (http-client)
>
> Would committers be able to review the patches above and +1/-1 as appropriate?
>
> Any other patches that we should include in this release?
>
> - Taylor
>
> On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> There are a number of email threads and JIRA issues regarding upgrading various storm dependencies, so I’d like to enumerate them and discuss them in one thread.
>>
>> Here’s the list so far:
>>
>> 1. Kryo/Carbonite (STORM-263)[1]
>> 2. Clojure (STORM-265) [2]
>> 3. commons-io (STORM-258) [3]
>> 4. curator (STORM-252) [4]
>> 5. http-client [5]
>>
>> I am +1 for all of the above, with the exception of #2, which I am +0 only because I personally haven’t had a chance to do any testing with newer versions of clojure. I’d be interested to hear if anyone has done any testing with newer versions of clojure.
>>
>> I think we should at least consider a bump to clojure 1.5 since it includes Bobby Evan’s patch that fixes error output getting swallowed (this manifests itself as the maven-clojure-plugin failing without any useful information when there are certain AOT compilation issues — very annoying).
>>
>> - Taylor
>>
>> [1] https://issues.apache.org/jira/browse/STORM-263
>> [2] https://issues.apache.org/jira/browse/STORM-265
>> [3] https://issues.apache.org/jira/browse/STORM-258
>> [4] https://issues.apache.org/jira/browse/STORM-252
>> [5] http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>>
>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>>
>>> Agreed. One of our guys got hung up on that just yesterday.
>>> It isn’t hard to track down the assembly, but it might be worth making it a bit easier.
>>>
>>> Also, I’d love to see STORM-263 included in the next release. It seems like a quick win.
>>>
>>> -brian
>>>
>>> ---
>>> Brian O'Neill
>>> Chief Technology Officer
>>>
>>> Health Market Science
>>> The Science of Better Results
>>> 2700 Horizon Drive • King of Prussia, PA • 19406
>>> M: 215.588.6024 • @boneill42 • healthmarketscience.com
>>>
>>> This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited.
>>>
>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>>
>>>> hi guys
>>>>
>>>> How about adding a bin/build_release.sh (http://build_release.sh) script in this 0.9.2-incubating release?
>>>>
>>>> Since some guys asked the question about building storm release package more than once in the mail list.
>>>>
>>>> --
>>>> Best Regards!
>>>>
>>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>> Distributed Software Engineer
>>>>
>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this definition code
>>>>> can committed with no +1 from a committer, right?
>>>>>
>>>>> You may want to look at Hadoop bylaws -
>>>>> https://hadoop.apache.org/bylaws.html
>>>>>
>>>>> The code commits in Hadoop are consensus approval with minimum +1 from an
>>>>> active committer and no veto. This has worked well in my experience. It may
>>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>>> merging feature branches.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)> wrote:
>>>>>
>>>>>> That brings up a good point that probably deserves a separate thread.
>>>>>>
>>>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>>>
>>>>>> For code changes, I've been operating under a "lazy consensus +2" model: 2
>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the patch,
>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been waiting
>>>>>> for 3 binding votes and no vetoes.
>>>>>>
>>>>>> That's kind of a middle ground between the traditional code modification
>>>>>> rule and lazy consensus [1].
>>>>>>
>>>>>> When wearing my "release manager" hat, I've also interpreted "code change"
>>>>>> to mean "anything that alters the behavior of the software we produce." In
>>>>>> terms of the build/packaging I've been a little looser. For large changes
>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>>>> changes I've committed directly -- I don't think we need to have a 3-day
>>>>>> vote on updating the CHANGELOG, for example.
>>>>>>
>>>>>> Anyway, it's something to think about. Sorry for hijacking the thread.
>>>>>>
>>>>>> +1 (again ;) )
>>>>>>
>>>>>> Taylor
>>>>>>
>>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>>
>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com (mailto:nathan@nathanmarz.com)> wrote:
>>>>>>>
>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there. Just
>>>>>>> one more vote needed by a committer.
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com (mailto:plucas@yelp.com)> wrote:
>>>>>>>>
>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it impossible
>>>>>> to
>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>>
>>>>>>
>>>>>> grouping.
>>>>>>>>
>>>>>>>> I'm not sure why there haven't been more reports of problems with it.
>>>>>> Two
>>>>>>>> possibilities I can think of are that we are using exclusively shell
>>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>>> classes--and
>>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> (thousands /
>>>>>>>> sec, KB in size)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Never mind... just found it.
>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>>
>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com (mailto:derekd@yahoo-inc.com)>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>>> reading
>>>>>>>>> Taylor's mail.
>>>>>>>>>>>
>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries (likelihood
>>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> exceeds
>>>>>>>>> 30.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Derek
>>>>>>>>>>>
>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> STORM-187.
>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because it's a
>>>>>>>>>>>> stability problem. The secondary aspect is that this issue taints
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> confidently
>>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>>
>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>>> (Upgrade
>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK 3.4.5
>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> official
>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently released
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ZK
>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know how
>>>>>>>>>>>> confident we are that the code in STORM-252 actually works, i.e.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> whether
>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> jumping
>>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>>
>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same boat:
>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use of ZK
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 3.3.4
>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's basically
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> the
>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the Kafka
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> team
>>>>>>>>> is
>>>>>>>>>>>> working on that.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Michael
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not seen it
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> in
>>>>>>>>> the wild, but I've heard of a few cases where it was enough to hinder
>>>>>>>>> upgrading.
>>>>>>>>>>>>>
>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>>>> being
>>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Taylor
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Patrick Lucas
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: @nathanmarz
>>>>>>> http://nathanmarz.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://hortonworks.com/download/
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>>> of this message is not the intended recipient, you are hereby notified that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender immediately
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
This release has been baking for a while and a number of important improvements and bug fixes have been merged into the master branch.
There are a few dependency updates that are still pending:
- STORM-265 (clojure)
- STORM-252 (curator)
- STORM-291 (http-client)
Would committers be able to review the patches above and +1/-1 as appropriate?
Any other patches that we should include in this release?
- Taylor
On Mar 26, 2014, at 10:09 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> There are a number of email threads and JIRA issues regarding upgrading various storm dependencies, so I’d like to enumerate them and discuss them in one thread.
>
> Here’s the list so far:
>
> 1. Kryo/Carbonite (STORM-263)[1]
> 2. Clojure (STORM-265) [2]
> 3. commons-io (STORM-258) [3]
> 4. curator (STORM-252) [4]
> 5. http-client [5]
>
> I am +1 for all of the above, with the exception of #2, which I am +0 only because I personally haven’t had a chance to do any testing with newer versions of clojure. I’d be interested to hear if anyone has done any testing with newer versions of clojure.
>
> I think we should at least consider a bump to clojure 1.5 since it includes Bobby Evan’s patch that fixes error output getting swallowed (this manifests itself as the maven-clojure-plugin failing without any useful information when there are certain AOT compilation issues — very annoying).
>
> - Taylor
>
> [1] https://issues.apache.org/jira/browse/STORM-263
> [2] https://issues.apache.org/jira/browse/STORM-265
> [3] https://issues.apache.org/jira/browse/STORM-258
> [4] https://issues.apache.org/jira/browse/STORM-252
> [5] http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
>
> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
>
>> Agreed. One of our guys got hung up on that just yesterday.
>> It isn’t hard to track down the assembly, but it might be worth making it a bit easier.
>>
>> Also, I’d love to see STORM-263 included in the next release. It seems like a quick win.
>>
>> -brian
>>
>> ---
>> Brian O'Neill
>> Chief Technology Officer
>>
>> Health Market Science
>> The Science of Better Results
>> 2700 Horizon Drive • King of Prussia, PA • 19406
>> M: 215.588.6024 • @boneill42 • healthmarketscience.com
>>
>> This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited.
>>
>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>>
>>> hi guys
>>>
>>> How about adding a bin/build_release.sh (http://build_release.sh) script in this 0.9.2-incubating release?
>>>
>>> Since some guys asked the question about building storm release package more than once in the mail list.
>>>
>>> --
>>> Best Regards!
>>>
>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>> Distributed Software Engineer
>>>
>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>>> Taylor, I am not very clear on "Lazy consensus +2". By this definition code
>>>> can committed with no +1 from a committer, right?
>>>>
>>>> You may want to look at Hadoop bylaws -
>>>> https://hadoop.apache.org/bylaws.html
>>>>
>>>> The code commits in Hadoop are consensus approval with minimum +1 from an
>>>> active committer and no veto. This has worked well in my experience. It may
>>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>>> merging feature branches.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)> wrote:
>>>>
>>>>> That brings up a good point that probably deserves a separate thread.
>>>>>
>>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>>
>>>>> For code changes, I've been operating under a "lazy consensus +2" model: 2
>>>>> committer +1 votes and no vetoes (-1). If a committer submits the patch,
>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been waiting
>>>>> for 3 binding votes and no vetoes.
>>>>>
>>>>> That's kind of a middle ground between the traditional code modification
>>>>> rule and lazy consensus [1].
>>>>>
>>>>> When wearing my "release manager" hat, I've also interpreted "code change"
>>>>> to mean "anything that alters the behavior of the software we produce." In
>>>>> terms of the build/packaging I've been a little looser. For large changes
>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>>> changes I've committed directly -- I don't think we need to have a 3-day
>>>>> vote on updating the CHANGELOG, for example.
>>>>>
>>>>> Anyway, it's something to think about. Sorry for hijacking the thread.
>>>>>
>>>>> +1 (again ;) )
>>>>>
>>>>> Taylor
>>>>>
>>>>> [1] https://www.apache.org/foundation/voting.html
>>>>>
>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com (mailto:nathan@nathanmarz.com)> wrote:
>>>>>>
>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there. Just
>>>>>> one more vote needed by a committer.
>>>>>>
>>>>>>
>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com (mailto:plucas@yelp.com)> wrote:
>>>>>>>
>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it impossible
>>>>> to
>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>>
>>>>>
>>>>> grouping.
>>>>>>>
>>>>>>> I'm not sure why there haven't been more reports of problems with it.
>>>>> Two
>>>>>>> possibilities I can think of are that we are using exclusively shell
>>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>>> classes--and
>>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>>
>>>>>>
>>>>>
>>>>> (thousands /
>>>>>>> sec, KB in size)
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Never mind... just found it.
>>>>>>>>
>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>>
>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com (mailto:derekd@yahoo-inc.com)>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>>> reading
>>>>>>>> Taylor's mail.
>>>>>>>>>>
>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries (likelihood
>>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> exceeds
>>>>>>>> 30.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Derek
>>>>>>>>>>
>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> STORM-187.
>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because it's a
>>>>>>>>>>> stability problem. The secondary aspect is that this issue taints
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> the
>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> confidently
>>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>>
>>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>>
>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>>> (Upgrade
>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK 3.4.5
>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> official
>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently released
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ZK
>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know how
>>>>>>>>>>> confident we are that the code in STORM-252 actually works, i.e.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> whether
>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> jumping
>>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>>
>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same boat:
>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use of ZK
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> 3.3.4
>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's basically
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> the
>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the Kafka
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> team
>>>>>>>> is
>>>>>>>>>>> working on that.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Michael
>>>>>>>>>>>
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not seen it
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> in
>>>>>>>> the wild, but I've heard of a few cases where it was enough to hinder
>>>>>>>> upgrading.
>>>>>>>>>>>>
>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>>> being
>>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>>
>>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>>
>>>>>>>>>>>> -Taylor
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Patrick Lucas
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Twitter: @nathanmarz
>>>>>> http://nathanmarz.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> http://hortonworks.com/download/
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
There are a number of email threads and JIRA issues regarding upgrading various storm dependencies, so I’d like to enumerate them and discuss them in one thread.
Here’s the list so far:
1. Kryo/Carbonite (STORM-263)[1]
2. Clojure (STORM-265) [2]
3. commons-io (STORM-258) [3]
4. curator (STORM-252) [4]
5. http-client [5]
I am +1 for all of the above, with the exception of #2, which I am +0 only because I personally haven’t had a chance to do any testing with newer versions of clojure. I’d be interested to hear if anyone has done any testing with newer versions of clojure.
I think we should at least consider a bump to clojure 1.5 since it includes Bobby Evan’s patch that fixes error output getting swallowed (this manifests itself as the maven-clojure-plugin failing without any useful information when there are certain AOT compilation issues — very annoying).
- Taylor
[1] https://issues.apache.org/jira/browse/STORM-263
[2] https://issues.apache.org/jira/browse/STORM-265
[3] https://issues.apache.org/jira/browse/STORM-258
[4] https://issues.apache.org/jira/browse/STORM-252
[5] http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22E-1D47-4512-B876-5FA7A28D1E8F@gmail.com%3E
On Mar 26, 2014, at 7:13 AM, Brian O'Neill <bo...@alumni.brown.edu> wrote:
> Agreed. One of our guys got hung up on that just yesterday.
> It isn’t hard to track down the assembly, but it might be worth making it a bit easier.
>
> Also, I’d love to see STORM-263 included in the next release. It seems like a quick win.
>
> -brian
>
> ---
> Brian O'Neill
> Chief Technology Officer
>
> Health Market Science
> The Science of Better Results
> 2700 Horizon Drive • King of Prussia, PA • 19406
> M: 215.588.6024 • @boneill42 • healthmarketscience.com
>
> This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited.
>
> On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
>
>> hi guys
>>
>> How about adding a bin/build_release.sh (http://build_release.sh) script in this 0.9.2-incubating release?
>>
>> Since some guys asked the question about building storm release package more than once in the mail list.
>>
>> --
>> Best Regards!
>>
>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>> Distributed Software Engineer
>>
>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>>> Taylor, I am not very clear on "Lazy consensus +2". By this definition code
>>> can committed with no +1 from a committer, right?
>>>
>>> You may want to look at Hadoop bylaws -
>>> https://hadoop.apache.org/bylaws.html
>>>
>>> The code commits in Hadoop are consensus approval with minimum +1 from an
>>> active committer and no veto. This has worked well in my experience. It may
>>> be a good idea to also adapt minimum 3 +1s from active committers for
>>> merging feature branches.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)> wrote:
>>>
>>>> That brings up a good point that probably deserves a separate thread.
>>>>
>>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>>
>>>> For code changes, I've been operating under a "lazy consensus +2" model: 2
>>>> committer +1 votes and no vetoes (-1). If a committer submits the patch,
>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been waiting
>>>> for 3 binding votes and no vetoes.
>>>>
>>>> That's kind of a middle ground between the traditional code modification
>>>> rule and lazy consensus [1].
>>>>
>>>> When wearing my "release manager" hat, I've also interpreted "code change"
>>>> to mean "anything that alters the behavior of the software we produce." In
>>>> terms of the build/packaging I've been a little looser. For large changes
>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>>> changes I've committed directly -- I don't think we need to have a 3-day
>>>> vote on updating the CHANGELOG, for example.
>>>>
>>>> Anyway, it's something to think about. Sorry for hijacking the thread.
>>>>
>>>> +1 (again ;) )
>>>>
>>>> Taylor
>>>>
>>>> [1] https://www.apache.org/foundation/voting.html
>>>>
>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com (mailto:nathan@nathanmarz.com)> wrote:
>>>>>
>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there. Just
>>>>> one more vote needed by a committer.
>>>>>
>>>>>
>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com (mailto:plucas@yelp.com)> wrote:
>>>>>>
>>>>>> A fix STORM-120 would be greatly appreciated. It's making it impossible
>>>> to
>>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>>
>>>>
>>>> grouping.
>>>>>>
>>>>>> I'm not sure why there haven't been more reports of problems with it.
>>>> Two
>>>>>> possibilities I can think of are that we are using exclusively shell
>>>>>> components--perhaps there's a root-cause bug in those component
>>>>>> classes--and
>>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>>
>>>>>
>>>>
>>>> (thousands /
>>>>>> sec, KB in size)
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>>> wrote:
>>>>>>
>>>>>>> Never mind... just found it.
>>>>>>>
>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>> wrote:
>>>>>>>>
>>>>>>>> Derek do you have an idea for a fix?
>>>>>>>>
>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com (mailto:derekd@yahoo-inc.com)>
>>>> wrote:
>>>>>>>>
>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>>> reading
>>>>>>> Taylor's mail.
>>>>>>>>>
>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries (likelihood
>>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> exceeds
>>>>>>> 30.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Derek
>>>>>>>>>
>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> STORM-187.
>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because it's a
>>>>>>>>>> stability problem. The secondary aspect is that this issue taints
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> the
>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> confidently
>>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>>
>>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>>
>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>>> (Upgrade
>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK 3.4.5
>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> official
>>>>>>>>>> Storm support for the latest ZK version (ok, the recently released
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ZK
>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know how
>>>>>>>>>> confident we are that the code in STORM-252 actually works, i.e.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> whether
>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> jumping
>>>>>>>>>> the gun or a safe move.
>>>>>>>>>>
>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same boat:
>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use of ZK
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> 3.3.4
>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's basically
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> the
>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the Kafka
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> team
>>>>>>> is
>>>>>>>>>> working on that.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Michael
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not seen it
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> in
>>>>>>> the wild, but I've heard of a few cases where it was enough to hinder
>>>>>>> upgrading.
>>>>>>>>>>>
>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>>> being
>>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>>
>>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>>
>>>>>>>>>>> -Taylor
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Patrick Lucas
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Twitter: @nathanmarz
>>>>> http://nathanmarz.com
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> http://hortonworks.com/download/
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>>
>
Re: 回复： [DISCUSS] Release Apache Storm 0.9.2-incubating
Posted by Brian O'Neill <bo...@alumni.brown.edu>.
Agreed. One of our guys got hung up on that just yesterday.
It isn’t hard to track down the assembly, but it might be worth making it a bit easier.
Also, I’d love to see STORM-263 included in the next release. It seems like a quick win.
-brian
---
Brian O'Neill
Chief Technology Officer
Health Market Science
The Science of Better Results
2700 Horizon Drive • King of Prussia, PA • 19406
M: 215.588.6024 • @boneill42 • healthmarketscience.com
This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited.
On Mar 26, 2014, at 2:19 AM, Kang Xiao <kx...@gmail.com> wrote:
> hi guys
>
> How about adding a bin/build_release.sh (http://build_release.sh) script in this 0.9.2-incubating release?
>
> Since some guys asked the question about building storm release package more than once in the mail list.
>
> --
> Best Regards!
>
> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
> Distributed Software Engineer
>
> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道：
>> Taylor, I am not very clear on "Lazy consensus +2". By this definition code
>> can committed with no +1 from a committer, right?
>>
>> You may want to look at Hadoop bylaws -
>> https://hadoop.apache.org/bylaws.html
>>
>> The code commits in Hadoop are consensus approval with minimum +1 from an
>> active committer and no veto. This has worked well in my experience. It may
>> be a good idea to also adapt minimum 3 +1s from active committers for
>> merging feature branches.
>>
>>
>>
>>
>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)> wrote:
>>
>>> That brings up a good point that probably deserves a separate thread.
>>>
>>> We should establish by-laws soon. Specifically a commit/merge policy.
>>>
>>> For code changes, I've been operating under a "lazy consensus +2" model: 2
>>> committer +1 votes and no vetoes (-1). If a committer submits the patch,
>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been waiting
>>> for 3 binding votes and no vetoes.
>>>
>>> That's kind of a middle ground between the traditional code modification
>>> rule and lazy consensus [1].
>>>
>>> When wearing my "release manager" hat, I've also interpreted "code change"
>>> to mean "anything that alters the behavior of the software we produce." In
>>> terms of the build/packaging I've been a little looser. For large changes
>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some
>>> changes I've committed directly -- I don't think we need to have a 3-day
>>> vote on updating the CHANGELOG, for example.
>>>
>>> Anyway, it's something to think about. Sorry for hijacking the thread.
>>>
>>> +1 (again ;) )
>>>
>>> Taylor
>>>
>>> [1] https://www.apache.org/foundation/voting.html
>>>
>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <nathan@nathanmarz.com (mailto:nathan@nathanmarz.com)> wrote:
>>>>
>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there. Just
>>>> one more vote needed by a committer.
>>>>
>>>>
>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <plucas@yelp.com (mailto:plucas@yelp.com)> wrote:
>>>>>
>>>>> A fix STORM-120 would be greatly appreciated. It's making it impossible
>>> to
>>>>> increase tasks/executors > 1 when there is a downstream shuffle
>>>>
>>>
>>> grouping.
>>>>>
>>>>> I'm not sure why there haven't been more reports of problems with it.
>>> Two
>>>>> possibilities I can think of are that we are using exclusively shell
>>>>> components--perhaps there's a root-cause bug in those component
>>>>> classes--and
>>>>> that we are dealing with a high volume stream of large tuples.
>>>>>
>>>>
>>>
>>> (thousands /
>>>>> sec, KB in size)
>>>>>
>>>>>
>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>>>> wrote:
>>>>>
>>>>>> Never mind... just found it.
>>>>>>
>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <ptgoetz@gmail.com (mailto:ptgoetz@gmail.com)>
>>> wrote:
>>>>>>>
>>>>>>> Derek do you have an idea for a fix?
>>>>>>>
>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <derekd@yahoo-inc.com (mailto:derekd@yahoo-inc.com)>
>>> wrote:
>>>>>>>
>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Yeah, we found it recently, and I created it this morning after
>>>>> reading
>>>>>> Taylor's mail.
>>>>>>>>
>>>>>>>> STORM-187 can be a problem with fewer than 30 retries (likelihood
>>>>>> depends on configuration), but we will hit STORM-259 when retries
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> exceeds
>>>>>> 30.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Derek
>>>>>>>>
>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote:
>>>>>>>>> On my side the most important change is, as you point out,
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> STORM-187.
>>>>>>>>> The primary reason is like Adam Lewis is pointing out because it's a
>>>>>>>>> stability problem. The secondary aspect is that this issue taints
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> the
>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> confidently
>>>>>>>>> bury ZeroMQ the better. :-)
>>>>>>>>>
>>>>>>>>> As I said above, this fix is the most important in my opinion.
>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether it's as
>>>>>>>>> important as STORM-187 or not.
>>>>>>>>>
>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252
>>> (Upgrade
>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK 3.4.5
>>>>>>>>> anyway for a couple of reasons, and it would be nice to have
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> official
>>>>>>>>> Storm support for the latest ZK version (ok, the recently released
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> ZK
>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know how
>>>>>>>>> confident we are that the code in STORM-252 actually works, i.e.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> whether
>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> jumping
>>>>>>>>> the gun or a safe move.
>>>>>>>>>
>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same boat:
>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use of ZK
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> 3.3.4
>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's basically
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> the
>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the Kafka
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> team
>>>>>> is
>>>>>>>>> working on that.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Michael
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote:
>>>>>>>>>> I'd like to get this discussion started, largely because the
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not seen it
>>>>>>
>>>>>
>>>>
>>>
>>> in
>>>>>> the wild, but I've heard of a few cases where it was enough to hinder
>>>>>> upgrading.
>>>>>>>>>>
>>>>>>>>>> HEAD looks good to me at the moment, with the major difference
>>> being
>>>>>> the zookeeper update and the patch mentioned above.
>>>>>>>>>>
>>>>>>>>>> Any thoughts on other PRs or patches to include?
>>>>>>>>>>
>>>>>>>>>> -Taylor
>>>>>
>>>>>
>>>>> --
>>>>> Patrick Lucas
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Twitter: @nathanmarz
>>>> http://nathanmarz.com
>>>>
>>>
>>>
>>
>>
>>
>>
>> --
>> http://hortonworks.com/download/
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>>
>>
>
>