You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Justin Leet <ju...@gmail.com> on 2016/10/11 13:54:51 UTC
[DISCUSS] Upgrading to Storm 1.0.x
Hi all,
I wanted to start a thread around upgrading our Storm version from 0.10.x
to 1.0.x. I created a Jira at
https://issues.apache.org/jira/browse/METRON-495 to mirror this discussion
and the results and opinions.
As listed at https://storm.apache.org/2016/04/12/storm100-released.html,
there's a variety of improvements with Storm 1.0. The obvious and likely
most important improvement is to performance, but a variety of other
improvements are noted on that link.
There are several changes that have to occur in order to make this upgrade.
As noted in http://storm.apache.org/releases/current/index.html, Storm's
code packages moved from backtype.storm to org.apache.storm, meaning all
topologies have to be recompiled if that change. There is a runtime
converter to run things in place with backtype.storm, but this doesn't
appear to be enough for our case because a couple interfaces change from
byte[] to ByteBuffer (somewhat, but not entirely, related to
https://issues.apache.org/jira/browse/STORM-1449). Even without this issue,
the long term solution is to use the new package naming.
In addition, our dev instances right now spin up an HDP 2.4 instance, which
matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to
migrate to Storm 1.0.x, I'd prefer to match that version, rather than going
to 1.0.2.
If there's anything I missed, or anything the group should be aware of in
this transition (Anybody lived done this upgrade on another project and
have input?), I'd love to hear it.
Thanks,
Justin
Re: [DISCUSS] Upgrading to Storm 1.0.x
Posted by Justin Leet <ju...@gmail.com>.
@Carolyn I'm inclined to call that a separate ticket since there's a
separate effort there. I'd link METRON-495 as a blocker to that ticket
@Casey 100% agree that we'll need to be pretty on top of testing, and
specific components to make sure we hit are absolutely a good topic for
this discussion if anyone wants to add onto your list.
Justin
On Tue, Oct 11, 2016 at 3:07 PM, Casey Stella <ce...@gmail.com> wrote:
> I think this would be good (+1 for me), but we should be quite careful that
> the testing for this PR is...exhaustive as it'll touch so much.
> At the very least, we'll need to ensure that:
>
> - Ansible full-dev and quick-dev continue to work
> - Ambari install continues to function
> - Data flows through the topologies and into kibana
> - Pcap ingestion continues to function
>
> Also, just to point out that with new versions come new dependencies (and
> versions), so you may experience some classpath issues.
>
> Casey
>
> On Tue, Oct 11, 2016 at 10:35 AM, Carolyn Duby <cd...@hortonworks.com>
> wrote:
>
> > Storm 1.0 will allow config of kafka with SSL. Should I comment on
> > METRON-495 or maybe it is better as a separate lira?
> >
> > Thanks
> > Carolyn
> >
> >
> >
> > On 10/11/16, 10:07 AM, "David Lyle" <dl...@gmail.com> wrote:
> >
> > >I'm +1 on this.
> > >
> > >On Tue, Oct 11, 2016 at 9:54 AM, Justin Leet <ju...@gmail.com>
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I wanted to start a thread around upgrading our Storm version from
> > 0.10.x
> > >> to 1.0.x. I created a Jira at
> > >> https://issues.apache.org/jira/browse/METRON-495 to mirror this
> > discussion
> > >> and the results and opinions.
> > >>
> > >> As listed at https://storm.apache.org/2016/
> 04/12/storm100-released.html
> > ,
> > >> there's a variety of improvements with Storm 1.0. The obvious and
> > likely
> > >> most important improvement is to performance, but a variety of other
> > >> improvements are noted on that link.
> > >>
> > >> There are several changes that have to occur in order to make this
> > upgrade.
> > >>
> > >>
> > >> As noted in http://storm.apache.org/releases/current/index.html,
> > Storm's
> > >> code packages moved from backtype.storm to org.apache.storm, meaning
> all
> > >> topologies have to be recompiled if that change. There is a runtime
> > >> converter to run things in place with backtype.storm, but this doesn't
> > >> appear to be enough for our case because a couple interfaces change
> from
> > >> byte[] to ByteBuffer (somewhat, but not entirely, related to
> > >> https://issues.apache.org/jira/browse/STORM-1449). Even without this
> > >> issue,
> > >> the long term solution is to use the new package naming.
> > >>
> > >> In addition, our dev instances right now spin up an HDP 2.4 instance,
> > which
> > >> matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to
> > >> migrate to Storm 1.0.x, I'd prefer to match that version, rather than
> > going
> > >> to 1.0.2.
> > >>
> > >> If there's anything I missed, or anything the group should be aware of
> > in
> > >> this transition (Anybody lived done this upgrade on another project
> and
> > >> have input?), I'd love to hear it.
> > >>
> > >> Thanks,
> > >> Justin
> > >>
> >
>
Re: [DISCUSS] Upgrading to Storm 1.0.x
Posted by Casey Stella <ce...@gmail.com>.
I think this would be good (+1 for me), but we should be quite careful that
the testing for this PR is...exhaustive as it'll touch so much.
At the very least, we'll need to ensure that:
- Ansible full-dev and quick-dev continue to work
- Ambari install continues to function
- Data flows through the topologies and into kibana
- Pcap ingestion continues to function
Also, just to point out that with new versions come new dependencies (and
versions), so you may experience some classpath issues.
Casey
On Tue, Oct 11, 2016 at 10:35 AM, Carolyn Duby <cd...@hortonworks.com>
wrote:
> Storm 1.0 will allow config of kafka with SSL. Should I comment on
> METRON-495 or maybe it is better as a separate lira?
>
> Thanks
> Carolyn
>
>
>
> On 10/11/16, 10:07 AM, "David Lyle" <dl...@gmail.com> wrote:
>
> >I'm +1 on this.
> >
> >On Tue, Oct 11, 2016 at 9:54 AM, Justin Leet <ju...@gmail.com>
> wrote:
> >
> >> Hi all,
> >>
> >> I wanted to start a thread around upgrading our Storm version from
> 0.10.x
> >> to 1.0.x. I created a Jira at
> >> https://issues.apache.org/jira/browse/METRON-495 to mirror this
> discussion
> >> and the results and opinions.
> >>
> >> As listed at https://storm.apache.org/2016/04/12/storm100-released.html
> ,
> >> there's a variety of improvements with Storm 1.0. The obvious and
> likely
> >> most important improvement is to performance, but a variety of other
> >> improvements are noted on that link.
> >>
> >> There are several changes that have to occur in order to make this
> upgrade.
> >>
> >>
> >> As noted in http://storm.apache.org/releases/current/index.html,
> Storm's
> >> code packages moved from backtype.storm to org.apache.storm, meaning all
> >> topologies have to be recompiled if that change. There is a runtime
> >> converter to run things in place with backtype.storm, but this doesn't
> >> appear to be enough for our case because a couple interfaces change from
> >> byte[] to ByteBuffer (somewhat, but not entirely, related to
> >> https://issues.apache.org/jira/browse/STORM-1449). Even without this
> >> issue,
> >> the long term solution is to use the new package naming.
> >>
> >> In addition, our dev instances right now spin up an HDP 2.4 instance,
> which
> >> matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to
> >> migrate to Storm 1.0.x, I'd prefer to match that version, rather than
> going
> >> to 1.0.2.
> >>
> >> If there's anything I missed, or anything the group should be aware of
> in
> >> this transition (Anybody lived done this upgrade on another project and
> >> have input?), I'd love to hear it.
> >>
> >> Thanks,
> >> Justin
> >>
>
Re: [DISCUSS] Upgrading to Storm 1.0.x
Posted by Carolyn Duby <cd...@hortonworks.com>.
Storm 1.0 will allow config of kafka with SSL. Should I comment on METRON-495 or maybe it is better as a separate lira?
Thanks
Carolyn
On 10/11/16, 10:07 AM, "David Lyle" <dl...@gmail.com> wrote:
>I'm +1 on this.
>
>On Tue, Oct 11, 2016 at 9:54 AM, Justin Leet <ju...@gmail.com> wrote:
>
>> Hi all,
>>
>> I wanted to start a thread around upgrading our Storm version from 0.10.x
>> to 1.0.x. I created a Jira at
>> https://issues.apache.org/jira/browse/METRON-495 to mirror this discussion
>> and the results and opinions.
>>
>> As listed at https://storm.apache.org/2016/04/12/storm100-released.html,
>> there's a variety of improvements with Storm 1.0. The obvious and likely
>> most important improvement is to performance, but a variety of other
>> improvements are noted on that link.
>>
>> There are several changes that have to occur in order to make this upgrade.
>>
>>
>> As noted in http://storm.apache.org/releases/current/index.html, Storm's
>> code packages moved from backtype.storm to org.apache.storm, meaning all
>> topologies have to be recompiled if that change. There is a runtime
>> converter to run things in place with backtype.storm, but this doesn't
>> appear to be enough for our case because a couple interfaces change from
>> byte[] to ByteBuffer (somewhat, but not entirely, related to
>> https://issues.apache.org/jira/browse/STORM-1449). Even without this
>> issue,
>> the long term solution is to use the new package naming.
>>
>> In addition, our dev instances right now spin up an HDP 2.4 instance, which
>> matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to
>> migrate to Storm 1.0.x, I'd prefer to match that version, rather than going
>> to 1.0.2.
>>
>> If there's anything I missed, or anything the group should be aware of in
>> this transition (Anybody lived done this upgrade on another project and
>> have input?), I'd love to hear it.
>>
>> Thanks,
>> Justin
>>
Re: [DISCUSS] Upgrading to Storm 1.0.x
Posted by David Lyle <dl...@gmail.com>.
I'm +1 on this.
On Tue, Oct 11, 2016 at 9:54 AM, Justin Leet <ju...@gmail.com> wrote:
> Hi all,
>
> I wanted to start a thread around upgrading our Storm version from 0.10.x
> to 1.0.x. I created a Jira at
> https://issues.apache.org/jira/browse/METRON-495 to mirror this discussion
> and the results and opinions.
>
> As listed at https://storm.apache.org/2016/04/12/storm100-released.html,
> there's a variety of improvements with Storm 1.0. The obvious and likely
> most important improvement is to performance, but a variety of other
> improvements are noted on that link.
>
> There are several changes that have to occur in order to make this upgrade.
>
>
> As noted in http://storm.apache.org/releases/current/index.html, Storm's
> code packages moved from backtype.storm to org.apache.storm, meaning all
> topologies have to be recompiled if that change. There is a runtime
> converter to run things in place with backtype.storm, but this doesn't
> appear to be enough for our case because a couple interfaces change from
> byte[] to ByteBuffer (somewhat, but not entirely, related to
> https://issues.apache.org/jira/browse/STORM-1449). Even without this
> issue,
> the long term solution is to use the new package naming.
>
> In addition, our dev instances right now spin up an HDP 2.4 instance, which
> matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to
> migrate to Storm 1.0.x, I'd prefer to match that version, rather than going
> to 1.0.2.
>
> If there's anything I missed, or anything the group should be aware of in
> this transition (Anybody lived done this upgrade on another project and
> have input?), I'd love to hear it.
>
> Thanks,
> Justin
>