You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Alexandre Vermeerbergen <av...@gmail.com> on 2017/11/22 16:43:29 UTC
[Discuss] Release Storm 1.2.0 (cont.)
Hello All,
I seems that the "[Discuss] Release Storm 1.2.0" thread reached some limits
as my latest posts on it aren't being received.
Whatever, here's the follow-up of my tests with storm-1.2.0 preview with
latest version of Stig's storm-kafka-client:
It works !
But... to make my topologies compatible with both 1.1.0 and 1.2.0, I had to
write an ugly method based on reflection so as to activate "acks" on the
official Kafka spout when used in autocommit mode:
/**
* Toggle the mode that makes Kafka spout able to send acks, if it's
supported.
* The support for this toggle has been introduced in Storm 1.2.0, so we
* internal use reflection in order to keep our topologies compatible
with pre-1.2.0
* @param builder A kafka spout config builder.
* @return The spout config builder that was passed in argument, with
modified toggle if it's supported.
*/
public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?, ?>
builder) {
try {
Method m =
builder.getClass().getMethod("setTupleTrackingEnforced", boolean.class);
m.invoke(builder, true);
} catch (NoSuchMethodException | SecurityException |
IllegalAccessException | IllegalArgumentException |
InvocationTargetException e) {
// Assume we're running with storm-kafka-client 1.1.0
}
return builder;
Next step for me: now that basic functionnal tests are OK, I want to run
the tests on my PPD environments having real big data rate and compare it
with my PRD one with same volume... stay tuned!
Thanks a lot to Stig for his quite reactive fixes on storm-kafka-client !
Best regards,
Alexandre
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Arun Mahadevan <ar...@apache.org>.
+1 to start the release process once STORM-2153 is merged.
If STORM-2860 can be merged soon we can include that as well.
Thanks,
Arun
On 1/7/18, 4:14 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>Now we merged STORM-2867 and STORM-2869.
>
>Remaining issues are STORM-2153 and STORM-2860, and STORM-2860 doesn't seem
>to bring benefit on 1.x version line hence unless I'm missing here, we just
>need to make sure STORM-2153 is resolved so that we could start release
>phase for Storm 1.2.0.
>
>- Jungtaek Lim (HeartSaVioR)
>
>2017년 12월 29일 (금) 오전 4:51, Arun Mahadevan <ar...@apache.org>님이 작성:
>
>> >STORM-2869: KafkaSpout discards all pending record when adjusting the
>> >consumer position after a commit [1]
>>
>> Hope we could get it merged this week or early next week.
>>
>>
>> >>New Feature
>> >STORM-2153: New Metrics Reporting API [2]
>>
>> I think this is waiting for a final +1 from revans2.
>>
>>
>> >STORM-2867: Add consumer lag metrics to KafkaSpout [3]
>>
>>
>> If required we can call out the kafka dependency since its a minor version
>> change. It may not be an issue if we use the reflection workaround proposed
>> in the PR ?
>>
>> IMO, it will be ideal to start the release process for 1.2.0 in the first
>> week of Jan after the above three are addressed.
>>
>> Thanks,
>> Arun
>>
>>
>>
>> On 12/27/17, 11:46 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>
>> >Looks like we got lost the chance to make release phase be started in
>> 2017,
>> >but I think we are really close to be sure we could start the process in
>> >early Jan. 2018.
>> >
>> >We haven't had "feature freeze" before releasing, so typically we still
>> >have a chance to get more features in Storm 1.2.0.
>> >
>> >So far, what we have remaining issues for Storm 1.2.0:
>> >
>> >> Bug
>> >STORM-2869: KafkaSpout discards all pending record when adjusting the
>> >consumer position after a commit [1]
>> >
>> >The PR for master got +1, so once we have PR for 1.x-branch, we could go
>> on
>> >merging. I expect this will be done in several days, unless Stig is going
>> >for long vacation.
>> >
>> >> New Feature
>> >STORM-2153: New Metrics Reporting API [2]
>> >
>> >This is likely waiting for final review, so I expect this patch to be
>> >finished within couple of weeks (early Jan. 2018), and if not I'd like to
>> >propose moving out of 1.2.0.
>> >
>> >STORM-2867: Add consumer lag metrics to KafkaSpout [3]
>> >
>> >The patch looks good, but it requires Kafka dependency to be updated from
>> >0.10.0.0 to 0.10.1.0 which might make Kafka 0.10.0.x user unable to use
>> >storm-kafka-client in Storm 1.2.0. Do we want to have a poll, or would it
>> >be not a big deal?
>> >
>> >STORM-2860: Add Kerberos support to Solr bolt [4]
>> >
>> >This patch breaks backward compatibility and we are discussing about
>> >alternative way to not break backward compatibility for 1.2.0. If we can
>> >get alternative in time, we could bring it to Storm 1.2.0, but if not, it
>> >should not block the release.
>> >
>> >Please participate reviewing, or provide any missing issues for Storm
>> >1.2.0, or give opinions on Storm 1.2.0.
>> >
>> >Thanks,
>> >Jungtaek Lim (HeartSaVioR)
>> >
>> >1. https://issues.apache.org/jira/browse/STORM-2869
>> >2. https://issues.apache.org/jira/browse/STORM-2153
>> >3. https://issues.apache.org/jira/browse/STORM-2867
>> >4. https://issues.apache.org/jira/browse/STORM-2860
>> >
>> >2017년 12월 18일 (월) 오전 3:50, Stig Rohde Døssing <st...@gmail.com>님이
>> 작성:
>> >
>> >> Alexandre,
>> >>
>> >> There are a couple more issues pending, see
>> >> https://issues.apache.org/jira/browse/STORM-2710. It might be easier if
>> >> you
>> >> build the code yourself. There's a guide at
>> >>
>> >>
>> https://github.com/apache/storm/blob/master/DEVELOPER.md#create-a-storm-distribution-packaging
>> >> .
>> >> The only change you should need to make is to run "mvn package
>> -Dgpg.skip"
>> >> instead of "mvn package" in the storm-dist directory to skip the GPG
>> >> signature part.
>> >>
>> >> 2017-12-17 15:09 GMT+01:00 Alexandre Vermeerbergen <
>> >> avermeerbergen@gmail.com
>> >> >:
>> >>
>> >> > Hello Storm developers,
>> >> >
>> >> > Now that I see that everything planned for Storm 1.2.0 release is done
>> >> (as
>> >> > I see at
>> https://issues.apache.org/jira/projects/STORM/versions/12341047
>> >> ),
>> >> > would it be possible to have new binaries for us to assess this new
>> >> release
>> >> > non-regression?
>> >> >
>> >> > In particular, I would like to check whether or not the bizarre
>> >> capacities
>> >> > metrics I get with the 1+ month-old Storm 1.2.0 preview are still
>> there.
>> >> >
>> >> > Best regards,
>> >> > Alexandre
>> >> >
>> >> >
>> >> > 2017-12-09 10:38 GMT+01:00 Alexandre Vermeerbergen <
>> >> > avermeerbergen@gmail.com
>> >> > >:
>> >> >
>> >> > > Hello Storm developers,
>> >> > >
>> >> > > It's been about 2 weeks that I running Storm 1.2.0 preview with 15
>> >> > > topologies, 6 Supervisors, 1 Nimbus, 3 Zookeepers, and Kafka 0.10
>> libs
>> >> > all
>> >> > > with Storm Kafka client spout instead of our own Kafka 0.10 spout.
>> >> > >
>> >> > > I noticed that statistics are going a bit nuts on bolts, with
>> >> capacities
>> >> > > reaching hundreds or more while everything seem to be running fine.
>> >> > > This look like this is tied to topologies relying on tick tuple -
>> not
>> >> > sure
>> >> > > but just a guess.
>> >> > >
>> >> > > See what kind of ridiculous capacities I can reach:
>> >> > >
>> >> > > Id Executors Tasks Emitted Transferred Capacity (last
>> >> > > 10m) Execute latency (ms) Executed Process latency (ms)
>> >> > > Acked Failed Error Host Error Port Last error Error
>> Time
>> >> > > aggregate 2 2 130660 130660 0.000 0.027 130620
>> >> > > 0.031 130580 0
>> >> > > alertsToKafka 3 3 0 0 0.010 0.054 2238580
>> >> > > 337.080 2238420 0
>> >> > > checkUnknown 20 20 145840 145840 120.391 52060.445
>> >> > > 19060 15062.337 18780 0
>> >> > > evaluateTriggers 17 17 2815520 2815520 130.115
>> >> > > 112.998 4755580 28.311 4756320 0
>> >> > > eventTimeFilter 1 1 13983880 13983880 51.324
>> 21.711
>> >> > > 13989060 37.980 13989260 0
>> >> > > filter 6 6 9103880 9091640 249.974 67.828
>> 13990120
>> >> > > 0.063 13989800 0
>> >> > > filterEventsWithDimensions 3 3 4754520 4754520
>> 233.222
>> >> > > 143.596 8958840 234.242 8960180 0
>> >> > > flappingDetection 8 8 2229860 2229860 0.003 0.020
>> >> > > 2229820 0.017 2229880 0
>> >> > >
>> >> > > (sorry for the lost formating)
>> >> > >
>> >> > > we have capacity at 249.974 for filter?
>> >> > >
>> >> > >
>> >> > > Is it a known issue?
>> >> > >
>> >> > > Best regards,
>> >> > > Alexandre
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > 2017-12-01 22:38 GMT+01:00 Alexandre Vermeerbergen <
>> >> > > avermeerbergen@gmail.com>:
>> >> > >
>> >> > >> Hello Junktaek,
>> >> > >>
>> >> > >> Thanks for your answer.
>> >> > >>
>> >> > >> Actually the sooner Storm 1.2.0 could be released (now that we
>> have a
>> >> > >> working storm-kafka-client with an impressible list of fixed
>> issue),
>> >> the
>> >> > >> better.
>> >> > >>
>> >> > >> Is it realistic to have some maximum date for starting the RC on
>> 15th
>> >> of
>> >> > >> December?
>> >> > >>
>> >> > >> Best regards,
>> >> > >> Alexandre Vermeerbergen
>> >> > >>
>> >> > >>
>> >> > >> 2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
>> >> > >>
>> >> > >>> We're getting closer to merge Metrics V2 in, so unless it will
>> take
>> >> > more
>> >> > >>> than couple of weeks, we will include to 1.2.0 as well.
>> >> > >>> I think STORM-2835 could come in couple of days, so that's not the
>> >> > issue
>> >> > >>> for releasing 1.2.0.
>> >> > >>>
>> >> > >>> -Jungtaek Lim (HeartSaVioR)
>> >> > >>>
>> >> > >>> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
>> >> > >>> avermeerbergen@gmail.com>님이
>> >> > >>> 작성:
>> >> > >>>
>> >> > >>> > Hello Storm Dev team,
>> >> > >>> >
>> >> > >>> > Our tests with Storm 1.2.0 preview on our (large) preproduction
>> has
>> >> > >>> been
>> >> > >>> > running fine for 1 week.
>> >> > >>> >
>> >> > >>> > This sound like a good opportunity to ask you if a Storm 1.2.0
>> >> > release
>> >> > >>> > could come soon so that we could target it for our next
>> production
>> >> > >>> upgrade.
>> >> > >>> >
>> >> > >>> > Yet I noticed this new JIRA:
>> >> > >>> > https://issues.apache.org/jira/browse/STORM-2835
>> >> > >>> >
>> >> > >>> > Could it be important enough that we need it to be included in
>> >> Storm
>> >> > >>> 1.2.0
>> >> > >>> > ?
>> >> > >>> >
>> >> > >>> > best regards,
>> >> > >>> > Alexandre Vermeerbergen
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
>> >> > >>> > avermeerbergen@gmail.com
>> >> > >>> > >:
>> >> > >>> >
>> >> > >>> > > Hello All,
>> >> > >>> > >
>> >> > >>> > > I seems that the "[Discuss] Release Storm 1.2.0" thread
>> reached
>> >> > some
>> >> > >>> > > limits as my latest posts on it aren't being received.
>> >> > >>> > >
>> >> > >>> > > Whatever, here's the follow-up of my tests with storm-1.2.0
>> >> preview
>> >> > >>> with
>> >> > >>> > > latest version of Stig's storm-kafka-client:
>> >> > >>> > >
>> >> > >>> > > It works !
>> >> > >>> > >
>> >> > >>> > > But... to make my topologies compatible with both 1.1.0 and
>> >> 1.2.0,
>> >> > I
>> >> > >>> had
>> >> > >>> > > to write an ugly method based on reflection so as to activate
>> >> > "acks"
>> >> > >>> on
>> >> > >>> > the
>> >> > >>> > > official Kafka spout when used in autocommit mode:
>> >> > >>> > >
>> >> > >>> > > /**
>> >> > >>> > > * Toggle the mode that makes Kafka spout able to send
>> acks,
>> >> if
>> >> > >>> it's
>> >> > >>> > > supported.
>> >> > >>> > > * The support for this toggle has been introduced in
>> Storm
>> >> > >>> 1.2.0, so
>> >> > >>> > > we
>> >> > >>> > > * internal use reflection in order to keep our topologies
>> >> > >>> compatible
>> >> > >>> > > with pre-1.2.0
>> >> > >>> > > * @param builder A kafka spout config builder.
>> >> > >>> > > * @return The spout config builder that was passed in
>> >> > argument,
>> >> > >>> with
>> >> > >>> > > modified toggle if it's supported.
>> >> > >>> > > */
>> >> > >>> > > public static Builder<?, ?>
>> fixKafkaSpoutAcking(Builder<?,
>> >> > ?>
>> >> > >>> > > builder) {
>> >> > >>> > > try {
>> >> > >>> > > Method m =
>> >> > >>> > builder.getClass().getMethod("setTupleTrackingEnforced",
>> >> > >>> > > boolean.class);
>> >> > >>> > > m.invoke(builder, true);
>> >> > >>> > > } catch (NoSuchMethodException | SecurityException |
>> >> > >>> > > IllegalAccessException | IllegalArgumentException |
>> >> > >>> > > InvocationTargetException e) {
>> >> > >>> > > // Assume we're running with storm-kafka-client
>> 1.1.0
>> >> > >>> > > }
>> >> > >>> > > return builder;
>> >> > >>> > >
>> >> > >>> > > Next step for me: now that basic functionnal tests are OK, I
>> want
>> >> > to
>> >> > >>> run
>> >> > >>> > > the tests on my PPD environments having real big data rate and
>> >> > >>> compare it
>> >> > >>> > > with my PRD one with same volume... stay tuned!
>> >> > >>> > >
>> >> > >>> > > Thanks a lot to Stig for his quite reactive fixes on
>> >> > >>> storm-kafka-client !
>> >> > >>> > >
>> >> > >>> > > Best regards,
>> >> > >>> > > Alexandre
>> >> > >>> > >
>> >> > >>> >
>> >> > >>>
>> >> > >>
>> >> > >>
>> >> > >
>> >> >
>> >>
>>
>>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Jungtaek Lim <ka...@gmail.com>.
Now we merged STORM-2867 and STORM-2869.
Remaining issues are STORM-2153 and STORM-2860, and STORM-2860 doesn't seem
to bring benefit on 1.x version line hence unless I'm missing here, we just
need to make sure STORM-2153 is resolved so that we could start release
phase for Storm 1.2.0.
- Jungtaek Lim (HeartSaVioR)
2017년 12월 29일 (금) 오전 4:51, Arun Mahadevan <ar...@apache.org>님이 작성:
> >STORM-2869: KafkaSpout discards all pending record when adjusting the
> >consumer position after a commit [1]
>
> Hope we could get it merged this week or early next week.
>
>
> >>New Feature
> >STORM-2153: New Metrics Reporting API [2]
>
> I think this is waiting for a final +1 from revans2.
>
>
> >STORM-2867: Add consumer lag metrics to KafkaSpout [3]
>
>
> If required we can call out the kafka dependency since its a minor version
> change. It may not be an issue if we use the reflection workaround proposed
> in the PR ?
>
> IMO, it will be ideal to start the release process for 1.2.0 in the first
> week of Jan after the above three are addressed.
>
> Thanks,
> Arun
>
>
>
> On 12/27/17, 11:46 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>
> >Looks like we got lost the chance to make release phase be started in
> 2017,
> >but I think we are really close to be sure we could start the process in
> >early Jan. 2018.
> >
> >We haven't had "feature freeze" before releasing, so typically we still
> >have a chance to get more features in Storm 1.2.0.
> >
> >So far, what we have remaining issues for Storm 1.2.0:
> >
> >> Bug
> >STORM-2869: KafkaSpout discards all pending record when adjusting the
> >consumer position after a commit [1]
> >
> >The PR for master got +1, so once we have PR for 1.x-branch, we could go
> on
> >merging. I expect this will be done in several days, unless Stig is going
> >for long vacation.
> >
> >> New Feature
> >STORM-2153: New Metrics Reporting API [2]
> >
> >This is likely waiting for final review, so I expect this patch to be
> >finished within couple of weeks (early Jan. 2018), and if not I'd like to
> >propose moving out of 1.2.0.
> >
> >STORM-2867: Add consumer lag metrics to KafkaSpout [3]
> >
> >The patch looks good, but it requires Kafka dependency to be updated from
> >0.10.0.0 to 0.10.1.0 which might make Kafka 0.10.0.x user unable to use
> >storm-kafka-client in Storm 1.2.0. Do we want to have a poll, or would it
> >be not a big deal?
> >
> >STORM-2860: Add Kerberos support to Solr bolt [4]
> >
> >This patch breaks backward compatibility and we are discussing about
> >alternative way to not break backward compatibility for 1.2.0. If we can
> >get alternative in time, we could bring it to Storm 1.2.0, but if not, it
> >should not block the release.
> >
> >Please participate reviewing, or provide any missing issues for Storm
> >1.2.0, or give opinions on Storm 1.2.0.
> >
> >Thanks,
> >Jungtaek Lim (HeartSaVioR)
> >
> >1. https://issues.apache.org/jira/browse/STORM-2869
> >2. https://issues.apache.org/jira/browse/STORM-2153
> >3. https://issues.apache.org/jira/browse/STORM-2867
> >4. https://issues.apache.org/jira/browse/STORM-2860
> >
> >2017년 12월 18일 (월) 오전 3:50, Stig Rohde Døssing <st...@gmail.com>님이
> 작성:
> >
> >> Alexandre,
> >>
> >> There are a couple more issues pending, see
> >> https://issues.apache.org/jira/browse/STORM-2710. It might be easier if
> >> you
> >> build the code yourself. There's a guide at
> >>
> >>
> https://github.com/apache/storm/blob/master/DEVELOPER.md#create-a-storm-distribution-packaging
> >> .
> >> The only change you should need to make is to run "mvn package
> -Dgpg.skip"
> >> instead of "mvn package" in the storm-dist directory to skip the GPG
> >> signature part.
> >>
> >> 2017-12-17 15:09 GMT+01:00 Alexandre Vermeerbergen <
> >> avermeerbergen@gmail.com
> >> >:
> >>
> >> > Hello Storm developers,
> >> >
> >> > Now that I see that everything planned for Storm 1.2.0 release is done
> >> (as
> >> > I see at
> https://issues.apache.org/jira/projects/STORM/versions/12341047
> >> ),
> >> > would it be possible to have new binaries for us to assess this new
> >> release
> >> > non-regression?
> >> >
> >> > In particular, I would like to check whether or not the bizarre
> >> capacities
> >> > metrics I get with the 1+ month-old Storm 1.2.0 preview are still
> there.
> >> >
> >> > Best regards,
> >> > Alexandre
> >> >
> >> >
> >> > 2017-12-09 10:38 GMT+01:00 Alexandre Vermeerbergen <
> >> > avermeerbergen@gmail.com
> >> > >:
> >> >
> >> > > Hello Storm developers,
> >> > >
> >> > > It's been about 2 weeks that I running Storm 1.2.0 preview with 15
> >> > > topologies, 6 Supervisors, 1 Nimbus, 3 Zookeepers, and Kafka 0.10
> libs
> >> > all
> >> > > with Storm Kafka client spout instead of our own Kafka 0.10 spout.
> >> > >
> >> > > I noticed that statistics are going a bit nuts on bolts, with
> >> capacities
> >> > > reaching hundreds or more while everything seem to be running fine.
> >> > > This look like this is tied to topologies relying on tick tuple -
> not
> >> > sure
> >> > > but just a guess.
> >> > >
> >> > > See what kind of ridiculous capacities I can reach:
> >> > >
> >> > > Id Executors Tasks Emitted Transferred Capacity (last
> >> > > 10m) Execute latency (ms) Executed Process latency (ms)
> >> > > Acked Failed Error Host Error Port Last error Error
> Time
> >> > > aggregate 2 2 130660 130660 0.000 0.027 130620
> >> > > 0.031 130580 0
> >> > > alertsToKafka 3 3 0 0 0.010 0.054 2238580
> >> > > 337.080 2238420 0
> >> > > checkUnknown 20 20 145840 145840 120.391 52060.445
> >> > > 19060 15062.337 18780 0
> >> > > evaluateTriggers 17 17 2815520 2815520 130.115
> >> > > 112.998 4755580 28.311 4756320 0
> >> > > eventTimeFilter 1 1 13983880 13983880 51.324
> 21.711
> >> > > 13989060 37.980 13989260 0
> >> > > filter 6 6 9103880 9091640 249.974 67.828
> 13990120
> >> > > 0.063 13989800 0
> >> > > filterEventsWithDimensions 3 3 4754520 4754520
> 233.222
> >> > > 143.596 8958840 234.242 8960180 0
> >> > > flappingDetection 8 8 2229860 2229860 0.003 0.020
> >> > > 2229820 0.017 2229880 0
> >> > >
> >> > > (sorry for the lost formating)
> >> > >
> >> > > we have capacity at 249.974 for filter?
> >> > >
> >> > >
> >> > > Is it a known issue?
> >> > >
> >> > > Best regards,
> >> > > Alexandre
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > 2017-12-01 22:38 GMT+01:00 Alexandre Vermeerbergen <
> >> > > avermeerbergen@gmail.com>:
> >> > >
> >> > >> Hello Junktaek,
> >> > >>
> >> > >> Thanks for your answer.
> >> > >>
> >> > >> Actually the sooner Storm 1.2.0 could be released (now that we
> have a
> >> > >> working storm-kafka-client with an impressible list of fixed
> issue),
> >> the
> >> > >> better.
> >> > >>
> >> > >> Is it realistic to have some maximum date for starting the RC on
> 15th
> >> of
> >> > >> December?
> >> > >>
> >> > >> Best regards,
> >> > >> Alexandre Vermeerbergen
> >> > >>
> >> > >>
> >> > >> 2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
> >> > >>
> >> > >>> We're getting closer to merge Metrics V2 in, so unless it will
> take
> >> > more
> >> > >>> than couple of weeks, we will include to 1.2.0 as well.
> >> > >>> I think STORM-2835 could come in couple of days, so that's not the
> >> > issue
> >> > >>> for releasing 1.2.0.
> >> > >>>
> >> > >>> -Jungtaek Lim (HeartSaVioR)
> >> > >>>
> >> > >>> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
> >> > >>> avermeerbergen@gmail.com>님이
> >> > >>> 작성:
> >> > >>>
> >> > >>> > Hello Storm Dev team,
> >> > >>> >
> >> > >>> > Our tests with Storm 1.2.0 preview on our (large) preproduction
> has
> >> > >>> been
> >> > >>> > running fine for 1 week.
> >> > >>> >
> >> > >>> > This sound like a good opportunity to ask you if a Storm 1.2.0
> >> > release
> >> > >>> > could come soon so that we could target it for our next
> production
> >> > >>> upgrade.
> >> > >>> >
> >> > >>> > Yet I noticed this new JIRA:
> >> > >>> > https://issues.apache.org/jira/browse/STORM-2835
> >> > >>> >
> >> > >>> > Could it be important enough that we need it to be included in
> >> Storm
> >> > >>> 1.2.0
> >> > >>> > ?
> >> > >>> >
> >> > >>> > best regards,
> >> > >>> > Alexandre Vermeerbergen
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
> >> > >>> > avermeerbergen@gmail.com
> >> > >>> > >:
> >> > >>> >
> >> > >>> > > Hello All,
> >> > >>> > >
> >> > >>> > > I seems that the "[Discuss] Release Storm 1.2.0" thread
> reached
> >> > some
> >> > >>> > > limits as my latest posts on it aren't being received.
> >> > >>> > >
> >> > >>> > > Whatever, here's the follow-up of my tests with storm-1.2.0
> >> preview
> >> > >>> with
> >> > >>> > > latest version of Stig's storm-kafka-client:
> >> > >>> > >
> >> > >>> > > It works !
> >> > >>> > >
> >> > >>> > > But... to make my topologies compatible with both 1.1.0 and
> >> 1.2.0,
> >> > I
> >> > >>> had
> >> > >>> > > to write an ugly method based on reflection so as to activate
> >> > "acks"
> >> > >>> on
> >> > >>> > the
> >> > >>> > > official Kafka spout when used in autocommit mode:
> >> > >>> > >
> >> > >>> > > /**
> >> > >>> > > * Toggle the mode that makes Kafka spout able to send
> acks,
> >> if
> >> > >>> it's
> >> > >>> > > supported.
> >> > >>> > > * The support for this toggle has been introduced in
> Storm
> >> > >>> 1.2.0, so
> >> > >>> > > we
> >> > >>> > > * internal use reflection in order to keep our topologies
> >> > >>> compatible
> >> > >>> > > with pre-1.2.0
> >> > >>> > > * @param builder A kafka spout config builder.
> >> > >>> > > * @return The spout config builder that was passed in
> >> > argument,
> >> > >>> with
> >> > >>> > > modified toggle if it's supported.
> >> > >>> > > */
> >> > >>> > > public static Builder<?, ?>
> fixKafkaSpoutAcking(Builder<?,
> >> > ?>
> >> > >>> > > builder) {
> >> > >>> > > try {
> >> > >>> > > Method m =
> >> > >>> > builder.getClass().getMethod("setTupleTrackingEnforced",
> >> > >>> > > boolean.class);
> >> > >>> > > m.invoke(builder, true);
> >> > >>> > > } catch (NoSuchMethodException | SecurityException |
> >> > >>> > > IllegalAccessException | IllegalArgumentException |
> >> > >>> > > InvocationTargetException e) {
> >> > >>> > > // Assume we're running with storm-kafka-client
> 1.1.0
> >> > >>> > > }
> >> > >>> > > return builder;
> >> > >>> > >
> >> > >>> > > Next step for me: now that basic functionnal tests are OK, I
> want
> >> > to
> >> > >>> run
> >> > >>> > > the tests on my PPD environments having real big data rate and
> >> > >>> compare it
> >> > >>> > > with my PRD one with same volume... stay tuned!
> >> > >>> > >
> >> > >>> > > Thanks a lot to Stig for his quite reactive fixes on
> >> > >>> storm-kafka-client !
> >> > >>> > >
> >> > >>> > > Best regards,
> >> > >>> > > Alexandre
> >> > >>> > >
> >> > >>> >
> >> > >>>
> >> > >>
> >> > >>
> >> > >
> >> >
> >>
>
>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
On Dec 28, 2017, at 2:51 PM, Arun Mahadevan <ar...@apache.org> wrote:
>
>>> New Feature
>> STORM-2153: New Metrics Reporting API [2]
>
> I think this is waiting for a final +1 from revans2.
>
Technically, we don’t need to wait for a +1 from any single committer, but in practice I like to wait until all those involved in reviewing a patch give their blessing, especially for large or large-impact changes. In this case both Bobby and Stig contributed significantly to the review.
If we can get to that point, I think we’re ready (or very close) for a 1.2 release. For STORM-2153 there’s already one +1, but I’m not comfortable merging it without solid consensus.
-Taylor
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Arun Mahadevan <ar...@apache.org>.
>STORM-2869: KafkaSpout discards all pending record when adjusting the
>consumer position after a commit [1]
Hope we could get it merged this week or early next week.
>>New Feature
>STORM-2153: New Metrics Reporting API [2]
I think this is waiting for a final +1 from revans2.
>STORM-2867: Add consumer lag metrics to KafkaSpout [3]
If required we can call out the kafka dependency since its a minor version change. It may not be an issue if we use the reflection workaround proposed in the PR ?
IMO, it will be ideal to start the release process for 1.2.0 in the first week of Jan after the above three are addressed.
Thanks,
Arun
On 12/27/17, 11:46 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>Looks like we got lost the chance to make release phase be started in 2017,
>but I think we are really close to be sure we could start the process in
>early Jan. 2018.
>
>We haven't had "feature freeze" before releasing, so typically we still
>have a chance to get more features in Storm 1.2.0.
>
>So far, what we have remaining issues for Storm 1.2.0:
>
>> Bug
>STORM-2869: KafkaSpout discards all pending record when adjusting the
>consumer position after a commit [1]
>
>The PR for master got +1, so once we have PR for 1.x-branch, we could go on
>merging. I expect this will be done in several days, unless Stig is going
>for long vacation.
>
>> New Feature
>STORM-2153: New Metrics Reporting API [2]
>
>This is likely waiting for final review, so I expect this patch to be
>finished within couple of weeks (early Jan. 2018), and if not I'd like to
>propose moving out of 1.2.0.
>
>STORM-2867: Add consumer lag metrics to KafkaSpout [3]
>
>The patch looks good, but it requires Kafka dependency to be updated from
>0.10.0.0 to 0.10.1.0 which might make Kafka 0.10.0.x user unable to use
>storm-kafka-client in Storm 1.2.0. Do we want to have a poll, or would it
>be not a big deal?
>
>STORM-2860: Add Kerberos support to Solr bolt [4]
>
>This patch breaks backward compatibility and we are discussing about
>alternative way to not break backward compatibility for 1.2.0. If we can
>get alternative in time, we could bring it to Storm 1.2.0, but if not, it
>should not block the release.
>
>Please participate reviewing, or provide any missing issues for Storm
>1.2.0, or give opinions on Storm 1.2.0.
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)
>
>1. https://issues.apache.org/jira/browse/STORM-2869
>2. https://issues.apache.org/jira/browse/STORM-2153
>3. https://issues.apache.org/jira/browse/STORM-2867
>4. https://issues.apache.org/jira/browse/STORM-2860
>
>2017년 12월 18일 (월) 오전 3:50, Stig Rohde Døssing <st...@gmail.com>님이 작성:
>
>> Alexandre,
>>
>> There are a couple more issues pending, see
>> https://issues.apache.org/jira/browse/STORM-2710. It might be easier if
>> you
>> build the code yourself. There's a guide at
>>
>> https://github.com/apache/storm/blob/master/DEVELOPER.md#create-a-storm-distribution-packaging
>> .
>> The only change you should need to make is to run "mvn package -Dgpg.skip"
>> instead of "mvn package" in the storm-dist directory to skip the GPG
>> signature part.
>>
>> 2017-12-17 15:09 GMT+01:00 Alexandre Vermeerbergen <
>> avermeerbergen@gmail.com
>> >:
>>
>> > Hello Storm developers,
>> >
>> > Now that I see that everything planned for Storm 1.2.0 release is done
>> (as
>> > I see at https://issues.apache.org/jira/projects/STORM/versions/12341047
>> ),
>> > would it be possible to have new binaries for us to assess this new
>> release
>> > non-regression?
>> >
>> > In particular, I would like to check whether or not the bizarre
>> capacities
>> > metrics I get with the 1+ month-old Storm 1.2.0 preview are still there.
>> >
>> > Best regards,
>> > Alexandre
>> >
>> >
>> > 2017-12-09 10:38 GMT+01:00 Alexandre Vermeerbergen <
>> > avermeerbergen@gmail.com
>> > >:
>> >
>> > > Hello Storm developers,
>> > >
>> > > It's been about 2 weeks that I running Storm 1.2.0 preview with 15
>> > > topologies, 6 Supervisors, 1 Nimbus, 3 Zookeepers, and Kafka 0.10 libs
>> > all
>> > > with Storm Kafka client spout instead of our own Kafka 0.10 spout.
>> > >
>> > > I noticed that statistics are going a bit nuts on bolts, with
>> capacities
>> > > reaching hundreds or more while everything seem to be running fine.
>> > > This look like this is tied to topologies relying on tick tuple - not
>> > sure
>> > > but just a guess.
>> > >
>> > > See what kind of ridiculous capacities I can reach:
>> > >
>> > > Id Executors Tasks Emitted Transferred Capacity (last
>> > > 10m) Execute latency (ms) Executed Process latency (ms)
>> > > Acked Failed Error Host Error Port Last error Error Time
>> > > aggregate 2 2 130660 130660 0.000 0.027 130620
>> > > 0.031 130580 0
>> > > alertsToKafka 3 3 0 0 0.010 0.054 2238580
>> > > 337.080 2238420 0
>> > > checkUnknown 20 20 145840 145840 120.391 52060.445
>> > > 19060 15062.337 18780 0
>> > > evaluateTriggers 17 17 2815520 2815520 130.115
>> > > 112.998 4755580 28.311 4756320 0
>> > > eventTimeFilter 1 1 13983880 13983880 51.324 21.711
>> > > 13989060 37.980 13989260 0
>> > > filter 6 6 9103880 9091640 249.974 67.828 13990120
>> > > 0.063 13989800 0
>> > > filterEventsWithDimensions 3 3 4754520 4754520 233.222
>> > > 143.596 8958840 234.242 8960180 0
>> > > flappingDetection 8 8 2229860 2229860 0.003 0.020
>> > > 2229820 0.017 2229880 0
>> > >
>> > > (sorry for the lost formating)
>> > >
>> > > we have capacity at 249.974 for filter?
>> > >
>> > >
>> > > Is it a known issue?
>> > >
>> > > Best regards,
>> > > Alexandre
>> > >
>> > >
>> > >
>> > >
>> > > 2017-12-01 22:38 GMT+01:00 Alexandre Vermeerbergen <
>> > > avermeerbergen@gmail.com>:
>> > >
>> > >> Hello Junktaek,
>> > >>
>> > >> Thanks for your answer.
>> > >>
>> > >> Actually the sooner Storm 1.2.0 could be released (now that we have a
>> > >> working storm-kafka-client with an impressible list of fixed issue),
>> the
>> > >> better.
>> > >>
>> > >> Is it realistic to have some maximum date for starting the RC on 15th
>> of
>> > >> December?
>> > >>
>> > >> Best regards,
>> > >> Alexandre Vermeerbergen
>> > >>
>> > >>
>> > >> 2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
>> > >>
>> > >>> We're getting closer to merge Metrics V2 in, so unless it will take
>> > more
>> > >>> than couple of weeks, we will include to 1.2.0 as well.
>> > >>> I think STORM-2835 could come in couple of days, so that's not the
>> > issue
>> > >>> for releasing 1.2.0.
>> > >>>
>> > >>> -Jungtaek Lim (HeartSaVioR)
>> > >>>
>> > >>> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
>> > >>> avermeerbergen@gmail.com>님이
>> > >>> 작성:
>> > >>>
>> > >>> > Hello Storm Dev team,
>> > >>> >
>> > >>> > Our tests with Storm 1.2.0 preview on our (large) preproduction has
>> > >>> been
>> > >>> > running fine for 1 week.
>> > >>> >
>> > >>> > This sound like a good opportunity to ask you if a Storm 1.2.0
>> > release
>> > >>> > could come soon so that we could target it for our next production
>> > >>> upgrade.
>> > >>> >
>> > >>> > Yet I noticed this new JIRA:
>> > >>> > https://issues.apache.org/jira/browse/STORM-2835
>> > >>> >
>> > >>> > Could it be important enough that we need it to be included in
>> Storm
>> > >>> 1.2.0
>> > >>> > ?
>> > >>> >
>> > >>> > best regards,
>> > >>> > Alexandre Vermeerbergen
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
>> > >>> > avermeerbergen@gmail.com
>> > >>> > >:
>> > >>> >
>> > >>> > > Hello All,
>> > >>> > >
>> > >>> > > I seems that the "[Discuss] Release Storm 1.2.0" thread reached
>> > some
>> > >>> > > limits as my latest posts on it aren't being received.
>> > >>> > >
>> > >>> > > Whatever, here's the follow-up of my tests with storm-1.2.0
>> preview
>> > >>> with
>> > >>> > > latest version of Stig's storm-kafka-client:
>> > >>> > >
>> > >>> > > It works !
>> > >>> > >
>> > >>> > > But... to make my topologies compatible with both 1.1.0 and
>> 1.2.0,
>> > I
>> > >>> had
>> > >>> > > to write an ugly method based on reflection so as to activate
>> > "acks"
>> > >>> on
>> > >>> > the
>> > >>> > > official Kafka spout when used in autocommit mode:
>> > >>> > >
>> > >>> > > /**
>> > >>> > > * Toggle the mode that makes Kafka spout able to send acks,
>> if
>> > >>> it's
>> > >>> > > supported.
>> > >>> > > * The support for this toggle has been introduced in Storm
>> > >>> 1.2.0, so
>> > >>> > > we
>> > >>> > > * internal use reflection in order to keep our topologies
>> > >>> compatible
>> > >>> > > with pre-1.2.0
>> > >>> > > * @param builder A kafka spout config builder.
>> > >>> > > * @return The spout config builder that was passed in
>> > argument,
>> > >>> with
>> > >>> > > modified toggle if it's supported.
>> > >>> > > */
>> > >>> > > public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?,
>> > ?>
>> > >>> > > builder) {
>> > >>> > > try {
>> > >>> > > Method m =
>> > >>> > builder.getClass().getMethod("setTupleTrackingEnforced",
>> > >>> > > boolean.class);
>> > >>> > > m.invoke(builder, true);
>> > >>> > > } catch (NoSuchMethodException | SecurityException |
>> > >>> > > IllegalAccessException | IllegalArgumentException |
>> > >>> > > InvocationTargetException e) {
>> > >>> > > // Assume we're running with storm-kafka-client 1.1.0
>> > >>> > > }
>> > >>> > > return builder;
>> > >>> > >
>> > >>> > > Next step for me: now that basic functionnal tests are OK, I want
>> > to
>> > >>> run
>> > >>> > > the tests on my PPD environments having real big data rate and
>> > >>> compare it
>> > >>> > > with my PRD one with same volume... stay tuned!
>> > >>> > >
>> > >>> > > Thanks a lot to Stig for his quite reactive fixes on
>> > >>> storm-kafka-client !
>> > >>> > >
>> > >>> > > Best regards,
>> > >>> > > Alexandre
>> > >>> > >
>> > >>> >
>> > >>>
>> > >>
>> > >>
>> > >
>> >
>>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Jungtaek Lim <ka...@gmail.com>.
Looks like we got lost the chance to make release phase be started in 2017,
but I think we are really close to be sure we could start the process in
early Jan. 2018.
We haven't had "feature freeze" before releasing, so typically we still
have a chance to get more features in Storm 1.2.0.
So far, what we have remaining issues for Storm 1.2.0:
> Bug
STORM-2869: KafkaSpout discards all pending record when adjusting the
consumer position after a commit [1]
The PR for master got +1, so once we have PR for 1.x-branch, we could go on
merging. I expect this will be done in several days, unless Stig is going
for long vacation.
> New Feature
STORM-2153: New Metrics Reporting API [2]
This is likely waiting for final review, so I expect this patch to be
finished within couple of weeks (early Jan. 2018), and if not I'd like to
propose moving out of 1.2.0.
STORM-2867: Add consumer lag metrics to KafkaSpout [3]
The patch looks good, but it requires Kafka dependency to be updated from
0.10.0.0 to 0.10.1.0 which might make Kafka 0.10.0.x user unable to use
storm-kafka-client in Storm 1.2.0. Do we want to have a poll, or would it
be not a big deal?
STORM-2860: Add Kerberos support to Solr bolt [4]
This patch breaks backward compatibility and we are discussing about
alternative way to not break backward compatibility for 1.2.0. If we can
get alternative in time, we could bring it to Storm 1.2.0, but if not, it
should not block the release.
Please participate reviewing, or provide any missing issues for Storm
1.2.0, or give opinions on Storm 1.2.0.
Thanks,
Jungtaek Lim (HeartSaVioR)
1. https://issues.apache.org/jira/browse/STORM-2869
2. https://issues.apache.org/jira/browse/STORM-2153
3. https://issues.apache.org/jira/browse/STORM-2867
4. https://issues.apache.org/jira/browse/STORM-2860
2017년 12월 18일 (월) 오전 3:50, Stig Rohde Døssing <st...@gmail.com>님이 작성:
> Alexandre,
>
> There are a couple more issues pending, see
> https://issues.apache.org/jira/browse/STORM-2710. It might be easier if
> you
> build the code yourself. There's a guide at
>
> https://github.com/apache/storm/blob/master/DEVELOPER.md#create-a-storm-distribution-packaging
> .
> The only change you should need to make is to run "mvn package -Dgpg.skip"
> instead of "mvn package" in the storm-dist directory to skip the GPG
> signature part.
>
> 2017-12-17 15:09 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com
> >:
>
> > Hello Storm developers,
> >
> > Now that I see that everything planned for Storm 1.2.0 release is done
> (as
> > I see at https://issues.apache.org/jira/projects/STORM/versions/12341047
> ),
> > would it be possible to have new binaries for us to assess this new
> release
> > non-regression?
> >
> > In particular, I would like to check whether or not the bizarre
> capacities
> > metrics I get with the 1+ month-old Storm 1.2.0 preview are still there.
> >
> > Best regards,
> > Alexandre
> >
> >
> > 2017-12-09 10:38 GMT+01:00 Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com
> > >:
> >
> > > Hello Storm developers,
> > >
> > > It's been about 2 weeks that I running Storm 1.2.0 preview with 15
> > > topologies, 6 Supervisors, 1 Nimbus, 3 Zookeepers, and Kafka 0.10 libs
> > all
> > > with Storm Kafka client spout instead of our own Kafka 0.10 spout.
> > >
> > > I noticed that statistics are going a bit nuts on bolts, with
> capacities
> > > reaching hundreds or more while everything seem to be running fine.
> > > This look like this is tied to topologies relying on tick tuple - not
> > sure
> > > but just a guess.
> > >
> > > See what kind of ridiculous capacities I can reach:
> > >
> > > Id Executors Tasks Emitted Transferred Capacity (last
> > > 10m) Execute latency (ms) Executed Process latency (ms)
> > > Acked Failed Error Host Error Port Last error Error Time
> > > aggregate 2 2 130660 130660 0.000 0.027 130620
> > > 0.031 130580 0
> > > alertsToKafka 3 3 0 0 0.010 0.054 2238580
> > > 337.080 2238420 0
> > > checkUnknown 20 20 145840 145840 120.391 52060.445
> > > 19060 15062.337 18780 0
> > > evaluateTriggers 17 17 2815520 2815520 130.115
> > > 112.998 4755580 28.311 4756320 0
> > > eventTimeFilter 1 1 13983880 13983880 51.324 21.711
> > > 13989060 37.980 13989260 0
> > > filter 6 6 9103880 9091640 249.974 67.828 13990120
> > > 0.063 13989800 0
> > > filterEventsWithDimensions 3 3 4754520 4754520 233.222
> > > 143.596 8958840 234.242 8960180 0
> > > flappingDetection 8 8 2229860 2229860 0.003 0.020
> > > 2229820 0.017 2229880 0
> > >
> > > (sorry for the lost formating)
> > >
> > > we have capacity at 249.974 for filter?
> > >
> > >
> > > Is it a known issue?
> > >
> > > Best regards,
> > > Alexandre
> > >
> > >
> > >
> > >
> > > 2017-12-01 22:38 GMT+01:00 Alexandre Vermeerbergen <
> > > avermeerbergen@gmail.com>:
> > >
> > >> Hello Junktaek,
> > >>
> > >> Thanks for your answer.
> > >>
> > >> Actually the sooner Storm 1.2.0 could be released (now that we have a
> > >> working storm-kafka-client with an impressible list of fixed issue),
> the
> > >> better.
> > >>
> > >> Is it realistic to have some maximum date for starting the RC on 15th
> of
> > >> December?
> > >>
> > >> Best regards,
> > >> Alexandre Vermeerbergen
> > >>
> > >>
> > >> 2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
> > >>
> > >>> We're getting closer to merge Metrics V2 in, so unless it will take
> > more
> > >>> than couple of weeks, we will include to 1.2.0 as well.
> > >>> I think STORM-2835 could come in couple of days, so that's not the
> > issue
> > >>> for releasing 1.2.0.
> > >>>
> > >>> -Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
> > >>> avermeerbergen@gmail.com>님이
> > >>> 작성:
> > >>>
> > >>> > Hello Storm Dev team,
> > >>> >
> > >>> > Our tests with Storm 1.2.0 preview on our (large) preproduction has
> > >>> been
> > >>> > running fine for 1 week.
> > >>> >
> > >>> > This sound like a good opportunity to ask you if a Storm 1.2.0
> > release
> > >>> > could come soon so that we could target it for our next production
> > >>> upgrade.
> > >>> >
> > >>> > Yet I noticed this new JIRA:
> > >>> > https://issues.apache.org/jira/browse/STORM-2835
> > >>> >
> > >>> > Could it be important enough that we need it to be included in
> Storm
> > >>> 1.2.0
> > >>> > ?
> > >>> >
> > >>> > best regards,
> > >>> > Alexandre Vermeerbergen
> > >>> >
> > >>> >
> > >>> >
> > >>> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
> > >>> > avermeerbergen@gmail.com
> > >>> > >:
> > >>> >
> > >>> > > Hello All,
> > >>> > >
> > >>> > > I seems that the "[Discuss] Release Storm 1.2.0" thread reached
> > some
> > >>> > > limits as my latest posts on it aren't being received.
> > >>> > >
> > >>> > > Whatever, here's the follow-up of my tests with storm-1.2.0
> preview
> > >>> with
> > >>> > > latest version of Stig's storm-kafka-client:
> > >>> > >
> > >>> > > It works !
> > >>> > >
> > >>> > > But... to make my topologies compatible with both 1.1.0 and
> 1.2.0,
> > I
> > >>> had
> > >>> > > to write an ugly method based on reflection so as to activate
> > "acks"
> > >>> on
> > >>> > the
> > >>> > > official Kafka spout when used in autocommit mode:
> > >>> > >
> > >>> > > /**
> > >>> > > * Toggle the mode that makes Kafka spout able to send acks,
> if
> > >>> it's
> > >>> > > supported.
> > >>> > > * The support for this toggle has been introduced in Storm
> > >>> 1.2.0, so
> > >>> > > we
> > >>> > > * internal use reflection in order to keep our topologies
> > >>> compatible
> > >>> > > with pre-1.2.0
> > >>> > > * @param builder A kafka spout config builder.
> > >>> > > * @return The spout config builder that was passed in
> > argument,
> > >>> with
> > >>> > > modified toggle if it's supported.
> > >>> > > */
> > >>> > > public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?,
> > ?>
> > >>> > > builder) {
> > >>> > > try {
> > >>> > > Method m =
> > >>> > builder.getClass().getMethod("setTupleTrackingEnforced",
> > >>> > > boolean.class);
> > >>> > > m.invoke(builder, true);
> > >>> > > } catch (NoSuchMethodException | SecurityException |
> > >>> > > IllegalAccessException | IllegalArgumentException |
> > >>> > > InvocationTargetException e) {
> > >>> > > // Assume we're running with storm-kafka-client 1.1.0
> > >>> > > }
> > >>> > > return builder;
> > >>> > >
> > >>> > > Next step for me: now that basic functionnal tests are OK, I want
> > to
> > >>> run
> > >>> > > the tests on my PPD environments having real big data rate and
> > >>> compare it
> > >>> > > with my PRD one with same volume... stay tuned!
> > >>> > >
> > >>> > > Thanks a lot to Stig for his quite reactive fixes on
> > >>> storm-kafka-client !
> > >>> > >
> > >>> > > Best regards,
> > >>> > > Alexandre
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Stig Rohde Døssing <st...@gmail.com>.
Alexandre,
There are a couple more issues pending, see
https://issues.apache.org/jira/browse/STORM-2710. It might be easier if you
build the code yourself. There's a guide at
https://github.com/apache/storm/blob/master/DEVELOPER.md#create-a-storm-distribution-packaging.
The only change you should need to make is to run "mvn package -Dgpg.skip"
instead of "mvn package" in the storm-dist directory to skip the GPG
signature part.
2017-12-17 15:09 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
> Hello Storm developers,
>
> Now that I see that everything planned for Storm 1.2.0 release is done (as
> I see at https://issues.apache.org/jira/projects/STORM/versions/12341047),
> would it be possible to have new binaries for us to assess this new release
> non-regression?
>
> In particular, I would like to check whether or not the bizarre capacities
> metrics I get with the 1+ month-old Storm 1.2.0 preview are still there.
>
> Best regards,
> Alexandre
>
>
> 2017-12-09 10:38 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com
> >:
>
> > Hello Storm developers,
> >
> > It's been about 2 weeks that I running Storm 1.2.0 preview with 15
> > topologies, 6 Supervisors, 1 Nimbus, 3 Zookeepers, and Kafka 0.10 libs
> all
> > with Storm Kafka client spout instead of our own Kafka 0.10 spout.
> >
> > I noticed that statistics are going a bit nuts on bolts, with capacities
> > reaching hundreds or more while everything seem to be running fine.
> > This look like this is tied to topologies relying on tick tuple - not
> sure
> > but just a guess.
> >
> > See what kind of ridiculous capacities I can reach:
> >
> > Id Executors Tasks Emitted Transferred Capacity (last
> > 10m) Execute latency (ms) Executed Process latency (ms)
> > Acked Failed Error Host Error Port Last error Error Time
> > aggregate 2 2 130660 130660 0.000 0.027 130620
> > 0.031 130580 0
> > alertsToKafka 3 3 0 0 0.010 0.054 2238580
> > 337.080 2238420 0
> > checkUnknown 20 20 145840 145840 120.391 52060.445
> > 19060 15062.337 18780 0
> > evaluateTriggers 17 17 2815520 2815520 130.115
> > 112.998 4755580 28.311 4756320 0
> > eventTimeFilter 1 1 13983880 13983880 51.324 21.711
> > 13989060 37.980 13989260 0
> > filter 6 6 9103880 9091640 249.974 67.828 13990120
> > 0.063 13989800 0
> > filterEventsWithDimensions 3 3 4754520 4754520 233.222
> > 143.596 8958840 234.242 8960180 0
> > flappingDetection 8 8 2229860 2229860 0.003 0.020
> > 2229820 0.017 2229880 0
> >
> > (sorry for the lost formating)
> >
> > we have capacity at 249.974 for filter?
> >
> >
> > Is it a known issue?
> >
> > Best regards,
> > Alexandre
> >
> >
> >
> >
> > 2017-12-01 22:38 GMT+01:00 Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com>:
> >
> >> Hello Junktaek,
> >>
> >> Thanks for your answer.
> >>
> >> Actually the sooner Storm 1.2.0 could be released (now that we have a
> >> working storm-kafka-client with an impressible list of fixed issue), the
> >> better.
> >>
> >> Is it realistic to have some maximum date for starting the RC on 15th of
> >> December?
> >>
> >> Best regards,
> >> Alexandre Vermeerbergen
> >>
> >>
> >> 2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
> >>
> >>> We're getting closer to merge Metrics V2 in, so unless it will take
> more
> >>> than couple of weeks, we will include to 1.2.0 as well.
> >>> I think STORM-2835 could come in couple of days, so that's not the
> issue
> >>> for releasing 1.2.0.
> >>>
> >>> -Jungtaek Lim (HeartSaVioR)
> >>>
> >>> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
> >>> avermeerbergen@gmail.com>님이
> >>> 작성:
> >>>
> >>> > Hello Storm Dev team,
> >>> >
> >>> > Our tests with Storm 1.2.0 preview on our (large) preproduction has
> >>> been
> >>> > running fine for 1 week.
> >>> >
> >>> > This sound like a good opportunity to ask you if a Storm 1.2.0
> release
> >>> > could come soon so that we could target it for our next production
> >>> upgrade.
> >>> >
> >>> > Yet I noticed this new JIRA:
> >>> > https://issues.apache.org/jira/browse/STORM-2835
> >>> >
> >>> > Could it be important enough that we need it to be included in Storm
> >>> 1.2.0
> >>> > ?
> >>> >
> >>> > best regards,
> >>> > Alexandre Vermeerbergen
> >>> >
> >>> >
> >>> >
> >>> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
> >>> > avermeerbergen@gmail.com
> >>> > >:
> >>> >
> >>> > > Hello All,
> >>> > >
> >>> > > I seems that the "[Discuss] Release Storm 1.2.0" thread reached
> some
> >>> > > limits as my latest posts on it aren't being received.
> >>> > >
> >>> > > Whatever, here's the follow-up of my tests with storm-1.2.0 preview
> >>> with
> >>> > > latest version of Stig's storm-kafka-client:
> >>> > >
> >>> > > It works !
> >>> > >
> >>> > > But... to make my topologies compatible with both 1.1.0 and 1.2.0,
> I
> >>> had
> >>> > > to write an ugly method based on reflection so as to activate
> "acks"
> >>> on
> >>> > the
> >>> > > official Kafka spout when used in autocommit mode:
> >>> > >
> >>> > > /**
> >>> > > * Toggle the mode that makes Kafka spout able to send acks, if
> >>> it's
> >>> > > supported.
> >>> > > * The support for this toggle has been introduced in Storm
> >>> 1.2.0, so
> >>> > > we
> >>> > > * internal use reflection in order to keep our topologies
> >>> compatible
> >>> > > with pre-1.2.0
> >>> > > * @param builder A kafka spout config builder.
> >>> > > * @return The spout config builder that was passed in
> argument,
> >>> with
> >>> > > modified toggle if it's supported.
> >>> > > */
> >>> > > public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?,
> ?>
> >>> > > builder) {
> >>> > > try {
> >>> > > Method m =
> >>> > builder.getClass().getMethod("setTupleTrackingEnforced",
> >>> > > boolean.class);
> >>> > > m.invoke(builder, true);
> >>> > > } catch (NoSuchMethodException | SecurityException |
> >>> > > IllegalAccessException | IllegalArgumentException |
> >>> > > InvocationTargetException e) {
> >>> > > // Assume we're running with storm-kafka-client 1.1.0
> >>> > > }
> >>> > > return builder;
> >>> > >
> >>> > > Next step for me: now that basic functionnal tests are OK, I want
> to
> >>> run
> >>> > > the tests on my PPD environments having real big data rate and
> >>> compare it
> >>> > > with my PRD one with same volume... stay tuned!
> >>> > >
> >>> > > Thanks a lot to Stig for his quite reactive fixes on
> >>> storm-kafka-client !
> >>> > >
> >>> > > Best regards,
> >>> > > Alexandre
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello Storm developers,
Now that I see that everything planned for Storm 1.2.0 release is done (as
I see at https://issues.apache.org/jira/projects/STORM/versions/12341047),
would it be possible to have new binaries for us to assess this new release
non-regression?
In particular, I would like to check whether or not the bizarre capacities
metrics I get with the 1+ month-old Storm 1.2.0 preview are still there.
Best regards,
Alexandre
2017-12-09 10:38 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
> Hello Storm developers,
>
> It's been about 2 weeks that I running Storm 1.2.0 preview with 15
> topologies, 6 Supervisors, 1 Nimbus, 3 Zookeepers, and Kafka 0.10 libs all
> with Storm Kafka client spout instead of our own Kafka 0.10 spout.
>
> I noticed that statistics are going a bit nuts on bolts, with capacities
> reaching hundreds or more while everything seem to be running fine.
> This look like this is tied to topologies relying on tick tuple - not sure
> but just a guess.
>
> See what kind of ridiculous capacities I can reach:
>
> Id Executors Tasks Emitted Transferred Capacity (last
> 10m) Execute latency (ms) Executed Process latency (ms)
> Acked Failed Error Host Error Port Last error Error Time
> aggregate 2 2 130660 130660 0.000 0.027 130620
> 0.031 130580 0
> alertsToKafka 3 3 0 0 0.010 0.054 2238580
> 337.080 2238420 0
> checkUnknown 20 20 145840 145840 120.391 52060.445
> 19060 15062.337 18780 0
> evaluateTriggers 17 17 2815520 2815520 130.115
> 112.998 4755580 28.311 4756320 0
> eventTimeFilter 1 1 13983880 13983880 51.324 21.711
> 13989060 37.980 13989260 0
> filter 6 6 9103880 9091640 249.974 67.828 13990120
> 0.063 13989800 0
> filterEventsWithDimensions 3 3 4754520 4754520 233.222
> 143.596 8958840 234.242 8960180 0
> flappingDetection 8 8 2229860 2229860 0.003 0.020
> 2229820 0.017 2229880 0
>
> (sorry for the lost formating)
>
> we have capacity at 249.974 for filter?
>
>
> Is it a known issue?
>
> Best regards,
> Alexandre
>
>
>
>
> 2017-12-01 22:38 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>:
>
>> Hello Junktaek,
>>
>> Thanks for your answer.
>>
>> Actually the sooner Storm 1.2.0 could be released (now that we have a
>> working storm-kafka-client with an impressible list of fixed issue), the
>> better.
>>
>> Is it realistic to have some maximum date for starting the RC on 15th of
>> December?
>>
>> Best regards,
>> Alexandre Vermeerbergen
>>
>>
>> 2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
>>
>>> We're getting closer to merge Metrics V2 in, so unless it will take more
>>> than couple of weeks, we will include to 1.2.0 as well.
>>> I think STORM-2835 could come in couple of days, so that's not the issue
>>> for releasing 1.2.0.
>>>
>>> -Jungtaek Lim (HeartSaVioR)
>>>
>>> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
>>> avermeerbergen@gmail.com>님이
>>> 작성:
>>>
>>> > Hello Storm Dev team,
>>> >
>>> > Our tests with Storm 1.2.0 preview on our (large) preproduction has
>>> been
>>> > running fine for 1 week.
>>> >
>>> > This sound like a good opportunity to ask you if a Storm 1.2.0 release
>>> > could come soon so that we could target it for our next production
>>> upgrade.
>>> >
>>> > Yet I noticed this new JIRA:
>>> > https://issues.apache.org/jira/browse/STORM-2835
>>> >
>>> > Could it be important enough that we need it to be included in Storm
>>> 1.2.0
>>> > ?
>>> >
>>> > best regards,
>>> > Alexandre Vermeerbergen
>>> >
>>> >
>>> >
>>> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
>>> > avermeerbergen@gmail.com
>>> > >:
>>> >
>>> > > Hello All,
>>> > >
>>> > > I seems that the "[Discuss] Release Storm 1.2.0" thread reached some
>>> > > limits as my latest posts on it aren't being received.
>>> > >
>>> > > Whatever, here's the follow-up of my tests with storm-1.2.0 preview
>>> with
>>> > > latest version of Stig's storm-kafka-client:
>>> > >
>>> > > It works !
>>> > >
>>> > > But... to make my topologies compatible with both 1.1.0 and 1.2.0, I
>>> had
>>> > > to write an ugly method based on reflection so as to activate "acks"
>>> on
>>> > the
>>> > > official Kafka spout when used in autocommit mode:
>>> > >
>>> > > /**
>>> > > * Toggle the mode that makes Kafka spout able to send acks, if
>>> it's
>>> > > supported.
>>> > > * The support for this toggle has been introduced in Storm
>>> 1.2.0, so
>>> > > we
>>> > > * internal use reflection in order to keep our topologies
>>> compatible
>>> > > with pre-1.2.0
>>> > > * @param builder A kafka spout config builder.
>>> > > * @return The spout config builder that was passed in argument,
>>> with
>>> > > modified toggle if it's supported.
>>> > > */
>>> > > public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?, ?>
>>> > > builder) {
>>> > > try {
>>> > > Method m =
>>> > builder.getClass().getMethod("setTupleTrackingEnforced",
>>> > > boolean.class);
>>> > > m.invoke(builder, true);
>>> > > } catch (NoSuchMethodException | SecurityException |
>>> > > IllegalAccessException | IllegalArgumentException |
>>> > > InvocationTargetException e) {
>>> > > // Assume we're running with storm-kafka-client 1.1.0
>>> > > }
>>> > > return builder;
>>> > >
>>> > > Next step for me: now that basic functionnal tests are OK, I want to
>>> run
>>> > > the tests on my PPD environments having real big data rate and
>>> compare it
>>> > > with my PRD one with same volume... stay tuned!
>>> > >
>>> > > Thanks a lot to Stig for his quite reactive fixes on
>>> storm-kafka-client !
>>> > >
>>> > > Best regards,
>>> > > Alexandre
>>> > >
>>> >
>>>
>>
>>
>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello Storm developers,
It's been about 2 weeks that I running Storm 1.2.0 preview with 15
topologies, 6 Supervisors, 1 Nimbus, 3 Zookeepers, and Kafka 0.10 libs all
with Storm Kafka client spout instead of our own Kafka 0.10 spout.
I noticed that statistics are going a bit nuts on bolts, with capacities
reaching hundreds or more while everything seem to be running fine.
This look like this is tied to topologies relying on tick tuple - not sure
but just a guess.
See what kind of ridiculous capacities I can reach:
Id Executors Tasks Emitted Transferred Capacity (last
10m) Execute latency (ms) Executed Process latency (ms)
Acked Failed Error Host Error Port Last error Error Time
aggregate 2 2 130660 130660 0.000 0.027 130620
0.031 130580 0
alertsToKafka 3 3 0 0 0.010 0.054 2238580
337.080 2238420 0
checkUnknown 20 20 145840 145840 120.391 52060.445
19060 15062.337 18780 0
evaluateTriggers 17 17 2815520 2815520 130.115 112.998
4755580 28.311 4756320 0
eventTimeFilter 1 1 13983880 13983880 51.324 21.711
13989060 37.980 13989260 0
filter 6 6 9103880 9091640 249.974 67.828 13990120
0.063 13989800 0
filterEventsWithDimensions 3 3 4754520 4754520 233.222
143.596 8958840 234.242 8960180 0
flappingDetection 8 8 2229860 2229860 0.003 0.020
2229820 0.017 2229880 0
(sorry for the lost formating)
we have capacity at 249.974 for filter?
Is it a known issue?
Best regards,
Alexandre
2017-12-01 22:38 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
> Hello Junktaek,
>
> Thanks for your answer.
>
> Actually the sooner Storm 1.2.0 could be released (now that we have a
> working storm-kafka-client with an impressible list of fixed issue), the
> better.
>
> Is it realistic to have some maximum date for starting the RC on 15th of
> December?
>
> Best regards,
> Alexandre Vermeerbergen
>
>
> 2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
>
>> We're getting closer to merge Metrics V2 in, so unless it will take more
>> than couple of weeks, we will include to 1.2.0 as well.
>> I think STORM-2835 could come in couple of days, so that's not the issue
>> for releasing 1.2.0.
>>
>> -Jungtaek Lim (HeartSaVioR)
>>
>> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
>> avermeerbergen@gmail.com>님이
>> 작성:
>>
>> > Hello Storm Dev team,
>> >
>> > Our tests with Storm 1.2.0 preview on our (large) preproduction has been
>> > running fine for 1 week.
>> >
>> > This sound like a good opportunity to ask you if a Storm 1.2.0 release
>> > could come soon so that we could target it for our next production
>> upgrade.
>> >
>> > Yet I noticed this new JIRA:
>> > https://issues.apache.org/jira/browse/STORM-2835
>> >
>> > Could it be important enough that we need it to be included in Storm
>> 1.2.0
>> > ?
>> >
>> > best regards,
>> > Alexandre Vermeerbergen
>> >
>> >
>> >
>> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
>> > avermeerbergen@gmail.com
>> > >:
>> >
>> > > Hello All,
>> > >
>> > > I seems that the "[Discuss] Release Storm 1.2.0" thread reached some
>> > > limits as my latest posts on it aren't being received.
>> > >
>> > > Whatever, here's the follow-up of my tests with storm-1.2.0 preview
>> with
>> > > latest version of Stig's storm-kafka-client:
>> > >
>> > > It works !
>> > >
>> > > But... to make my topologies compatible with both 1.1.0 and 1.2.0, I
>> had
>> > > to write an ugly method based on reflection so as to activate "acks"
>> on
>> > the
>> > > official Kafka spout when used in autocommit mode:
>> > >
>> > > /**
>> > > * Toggle the mode that makes Kafka spout able to send acks, if
>> it's
>> > > supported.
>> > > * The support for this toggle has been introduced in Storm
>> 1.2.0, so
>> > > we
>> > > * internal use reflection in order to keep our topologies
>> compatible
>> > > with pre-1.2.0
>> > > * @param builder A kafka spout config builder.
>> > > * @return The spout config builder that was passed in argument,
>> with
>> > > modified toggle if it's supported.
>> > > */
>> > > public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?, ?>
>> > > builder) {
>> > > try {
>> > > Method m =
>> > builder.getClass().getMethod("setTupleTrackingEnforced",
>> > > boolean.class);
>> > > m.invoke(builder, true);
>> > > } catch (NoSuchMethodException | SecurityException |
>> > > IllegalAccessException | IllegalArgumentException |
>> > > InvocationTargetException e) {
>> > > // Assume we're running with storm-kafka-client 1.1.0
>> > > }
>> > > return builder;
>> > >
>> > > Next step for me: now that basic functionnal tests are OK, I want to
>> run
>> > > the tests on my PPD environments having real big data rate and
>> compare it
>> > > with my PRD one with same volume... stay tuned!
>> > >
>> > > Thanks a lot to Stig for his quite reactive fixes on
>> storm-kafka-client !
>> > >
>> > > Best regards,
>> > > Alexandre
>> > >
>> >
>>
>
>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello Junktaek,
Thanks for your answer.
Actually the sooner Storm 1.2.0 could be released (now that we have a
working storm-kafka-client with an impressible list of fixed issue), the
better.
Is it realistic to have some maximum date for starting the RC on 15th of
December?
Best regards,
Alexandre Vermeerbergen
2017-11-30 0:33 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
> We're getting closer to merge Metrics V2 in, so unless it will take more
> than couple of weeks, we will include to 1.2.0 as well.
> I think STORM-2835 could come in couple of days, so that's not the issue
> for releasing 1.2.0.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>님이
> 작성:
>
> > Hello Storm Dev team,
> >
> > Our tests with Storm 1.2.0 preview on our (large) preproduction has been
> > running fine for 1 week.
> >
> > This sound like a good opportunity to ask you if a Storm 1.2.0 release
> > could come soon so that we could target it for our next production
> upgrade.
> >
> > Yet I noticed this new JIRA:
> > https://issues.apache.org/jira/browse/STORM-2835
> >
> > Could it be important enough that we need it to be included in Storm
> 1.2.0
> > ?
> >
> > best regards,
> > Alexandre Vermeerbergen
> >
> >
> >
> > 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
> > avermeerbergen@gmail.com
> > >:
> >
> > > Hello All,
> > >
> > > I seems that the "[Discuss] Release Storm 1.2.0" thread reached some
> > > limits as my latest posts on it aren't being received.
> > >
> > > Whatever, here's the follow-up of my tests with storm-1.2.0 preview
> with
> > > latest version of Stig's storm-kafka-client:
> > >
> > > It works !
> > >
> > > But... to make my topologies compatible with both 1.1.0 and 1.2.0, I
> had
> > > to write an ugly method based on reflection so as to activate "acks" on
> > the
> > > official Kafka spout when used in autocommit mode:
> > >
> > > /**
> > > * Toggle the mode that makes Kafka spout able to send acks, if
> it's
> > > supported.
> > > * The support for this toggle has been introduced in Storm 1.2.0,
> so
> > > we
> > > * internal use reflection in order to keep our topologies
> compatible
> > > with pre-1.2.0
> > > * @param builder A kafka spout config builder.
> > > * @return The spout config builder that was passed in argument,
> with
> > > modified toggle if it's supported.
> > > */
> > > public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?, ?>
> > > builder) {
> > > try {
> > > Method m =
> > builder.getClass().getMethod("setTupleTrackingEnforced",
> > > boolean.class);
> > > m.invoke(builder, true);
> > > } catch (NoSuchMethodException | SecurityException |
> > > IllegalAccessException | IllegalArgumentException |
> > > InvocationTargetException e) {
> > > // Assume we're running with storm-kafka-client 1.1.0
> > > }
> > > return builder;
> > >
> > > Next step for me: now that basic functionnal tests are OK, I want to
> run
> > > the tests on my PPD environments having real big data rate and compare
> it
> > > with my PRD one with same volume... stay tuned!
> > >
> > > Thanks a lot to Stig for his quite reactive fixes on
> storm-kafka-client !
> > >
> > > Best regards,
> > > Alexandre
> > >
> >
>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Jungtaek Lim <ka...@gmail.com>.
We're getting closer to merge Metrics V2 in, so unless it will take more
than couple of weeks, we will include to 1.2.0 as well.
I think STORM-2835 could come in couple of days, so that's not the issue
for releasing 1.2.0.
-Jungtaek Lim (HeartSaVioR)
2017년 11월 30일 (목) 오전 1:01, Alexandre Vermeerbergen <av...@gmail.com>님이
작성:
> Hello Storm Dev team,
>
> Our tests with Storm 1.2.0 preview on our (large) preproduction has been
> running fine for 1 week.
>
> This sound like a good opportunity to ask you if a Storm 1.2.0 release
> could come soon so that we could target it for our next production upgrade.
>
> Yet I noticed this new JIRA:
> https://issues.apache.org/jira/browse/STORM-2835
>
> Could it be important enough that we need it to be included in Storm 1.2.0
> ?
>
> best regards,
> Alexandre Vermeerbergen
>
>
>
> 2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com
> >:
>
> > Hello All,
> >
> > I seems that the "[Discuss] Release Storm 1.2.0" thread reached some
> > limits as my latest posts on it aren't being received.
> >
> > Whatever, here's the follow-up of my tests with storm-1.2.0 preview with
> > latest version of Stig's storm-kafka-client:
> >
> > It works !
> >
> > But... to make my topologies compatible with both 1.1.0 and 1.2.0, I had
> > to write an ugly method based on reflection so as to activate "acks" on
> the
> > official Kafka spout when used in autocommit mode:
> >
> > /**
> > * Toggle the mode that makes Kafka spout able to send acks, if it's
> > supported.
> > * The support for this toggle has been introduced in Storm 1.2.0, so
> > we
> > * internal use reflection in order to keep our topologies compatible
> > with pre-1.2.0
> > * @param builder A kafka spout config builder.
> > * @return The spout config builder that was passed in argument, with
> > modified toggle if it's supported.
> > */
> > public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?, ?>
> > builder) {
> > try {
> > Method m =
> builder.getClass().getMethod("setTupleTrackingEnforced",
> > boolean.class);
> > m.invoke(builder, true);
> > } catch (NoSuchMethodException | SecurityException |
> > IllegalAccessException | IllegalArgumentException |
> > InvocationTargetException e) {
> > // Assume we're running with storm-kafka-client 1.1.0
> > }
> > return builder;
> >
> > Next step for me: now that basic functionnal tests are OK, I want to run
> > the tests on my PPD environments having real big data rate and compare it
> > with my PRD one with same volume... stay tuned!
> >
> > Thanks a lot to Stig for his quite reactive fixes on storm-kafka-client !
> >
> > Best regards,
> > Alexandre
> >
>
Re: [Discuss] Release Storm 1.2.0 (cont.)
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello Storm Dev team,
Our tests with Storm 1.2.0 preview on our (large) preproduction has been
running fine for 1 week.
This sound like a good opportunity to ask you if a Storm 1.2.0 release
could come soon so that we could target it for our next production upgrade.
Yet I noticed this new JIRA:
https://issues.apache.org/jira/browse/STORM-2835
Could it be important enough that we need it to be included in Storm 1.2.0 ?
best regards,
Alexandre Vermeerbergen
2017-11-22 17:43 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
> Hello All,
>
> I seems that the "[Discuss] Release Storm 1.2.0" thread reached some
> limits as my latest posts on it aren't being received.
>
> Whatever, here's the follow-up of my tests with storm-1.2.0 preview with
> latest version of Stig's storm-kafka-client:
>
> It works !
>
> But... to make my topologies compatible with both 1.1.0 and 1.2.0, I had
> to write an ugly method based on reflection so as to activate "acks" on the
> official Kafka spout when used in autocommit mode:
>
> /**
> * Toggle the mode that makes Kafka spout able to send acks, if it's
> supported.
> * The support for this toggle has been introduced in Storm 1.2.0, so
> we
> * internal use reflection in order to keep our topologies compatible
> with pre-1.2.0
> * @param builder A kafka spout config builder.
> * @return The spout config builder that was passed in argument, with
> modified toggle if it's supported.
> */
> public static Builder<?, ?> fixKafkaSpoutAcking(Builder<?, ?>
> builder) {
> try {
> Method m = builder.getClass().getMethod("setTupleTrackingEnforced",
> boolean.class);
> m.invoke(builder, true);
> } catch (NoSuchMethodException | SecurityException |
> IllegalAccessException | IllegalArgumentException |
> InvocationTargetException e) {
> // Assume we're running with storm-kafka-client 1.1.0
> }
> return builder;
>
> Next step for me: now that basic functionnal tests are OK, I want to run
> the tests on my PPD environments having real big data rate and compare it
> with my PRD one with same volume... stay tuned!
>
> Thanks a lot to Stig for his quite reactive fixes on storm-kafka-client !
>
> Best regards,
> Alexandre
>