You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Michael Miklavcic <mi...@gmail.com> on 2019/09/17 17:13:00 UTC
[DISCUSS] Curator client upgrade
Hey all,
While working through the feature branch upgrade for HDP 3.1, we came
across some classpath related issues conflicting with Storm and Guava while
testing out Kerberos. Initially, this seemed reasonable to roll into the
Kerberos fix PR, but the scope has expanded a bit because we found
additional impacts while working through the Hadoop upgrade branch as well.
We're now looking at splitting this out into a separate PR in the feature
branch. Curator 4.0.0 is backwards compatible and will work with Zookeeper
3.4.6. Strictly speaking, this *could* be done against master first, but
there may be some duplicate testing effort there, and I'm really not it's
worth it as there is absolutely no issue currently with Curator 2.12.0 in
master - I call this out purely for transparency/information share with the
community. I'm leaning towards us making this change against the feature
branch directly as that is the only place we currently have compatibility
issues.
Best,
Mike
Re: [DISCUSS] Curator client upgrade
Posted by Michael Miklavcic <mi...@gmail.com>.
FYI - we were able to move forward on getting Kerberos working without
requiring a Curator upgrade. We probably want to consider some architecture
changes around how we manage this queue in MaaS at some point, however
right now we will be continuing with the current version of Curator. The
Zookeeper version is the same from HDP 2.5 to 2.6 to 3.1, so there should
be no added risk.
On Fri, Sep 20, 2019 at 1:52 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> Discussed this a bit further with Nick and Ryan offline. The original
> issue with Curator appears to have come up during Kerberos testing. Storm
> brings in a whole host of extra libs when Kerberos is enabled (some
> preliminary testing was started in parallel to the HBase and Hadoop
> upgrades). I'm told the lib in conflict is Guava. There had also been a
> minor version bump in the Hadoop upgrade branch of Ryan's that I've been
> basing some of my current work off of - I'm unclear of the need for that
> change, but it looks like we might be able to avoid it.
>
> I'm investigating whether we can get away with simply shading and
> relocating Guava in this case and leave a Curator upgrade as a follow-on
> for a point release. It appears that Nick was able to get maas tests
> running with an upgraded Hadoop without having to change the curator
> versions. There are probably more classpath issues with a Hadoop upgrade
> and Curator whether or not we rev the version, so that should be a wash in
> risk and effort. However, the one added big risk with upgrading Curator is
> that it's 1. a couple major version changes we're upgrading, and 2. we
> already ran into one change in functionality based on the issues seen in
> https://github.com/apache/metron/pull/1516. We still have more work to do
> on upgrading Hadoop as well as Kerberos once Hadoop is finished being
> upgraded. This means we could have a much higher probability of more
> functionality changes as we get further into testing. It's possible there
> are other reasons to motivate a Curator upgrade now, but this may be
> premature without more concrete details and having tried a smaller scope
> change with dependency relocation first.
>
> Mike
>
> On Tue, Sep 17, 2019 at 4:04 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
>> +1
>>
>>
>>
>>
>> On September 17, 2019 at 14:53:13, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> Also, caught this come through the HBase dev list -
>> https://issues.apache.org/jira/browse/HBASE-23032. Might be worth a look
>> at
>> 4.2.0 while we're at it and possibly also avoid this issue that I was
>> running into before when trying to use curator-tests 4.x
>> https://issues.apache.org/jira/browse/CURATOR-446. Fixed in 4.0.1, which
>> would naturally be included with 4.2.0 assuming we don't run into any
>> other
>> classpath or API compatibility issues.
>>
>> On Tue, Sep 17, 2019 at 11:20 AM Nick Allen <ni...@nickallen.org> wrote:
>>
>> > +1 to making the change on the feature branch.
>> >
>> > We don't really know how this might affect master which is still
>> building
>> > against HDP 2.6, nor is it strictly needed there. Going to Curator
>> 4.0.0
>> > is only needed due to the HDP 3.1 upgrade. This is also likely to get
>> > more focused testing cycles in the feature branch before it has a
>> chance to
>> > break anything in master.
>> >
>> > On Tue, Sep 17, 2019 at 1:13 PM Michael Miklavcic <
>> > michael.miklavcic@gmail.com> wrote:
>> >
>> > > Hey all,
>> > >
>> > > While working through the feature branch upgrade for HDP 3.1, we came
>> > > across some classpath related issues conflicting with Storm and Guava
>> > while
>> > > testing out Kerberos. Initially, this seemed reasonable to roll into
>> the
>> > > Kerberos fix PR, but the scope has expanded a bit because we found
>> > > additional impacts while working through the Hadoop upgrade branch as
>> > well.
>> > > We're now looking at splitting this out into a separate PR in the
>> feature
>> > > branch. Curator 4.0.0 is backwards compatible and will work with
>> > Zookeeper
>> > > 3.4.6. Strictly speaking, this *could* be done against master first,
>> but
>> > > there may be some duplicate testing effort there, and I'm really not
>> it's
>> > > worth it as there is absolutely no issue currently with Curator
>> 2.12.0 in
>> > > master - I call this out purely for transparency/information share
>> with
>> > the
>> > > community. I'm leaning towards us making this change against the
>> feature
>> > > branch directly as that is the only place we currently have
>> compatibility
>> > > issues.
>> > >
>> > > Best,
>> > > Mike
>> > >
>> >
>>
>>
Re: [DISCUSS] Curator client upgrade
Posted by Michael Miklavcic <mi...@gmail.com>.
Discussed this a bit further with Nick and Ryan offline. The original issue
with Curator appears to have come up during Kerberos testing. Storm brings
in a whole host of extra libs when Kerberos is enabled (some preliminary
testing was started in parallel to the HBase and Hadoop upgrades). I'm told
the lib in conflict is Guava. There had also been a minor version bump in
the Hadoop upgrade branch of Ryan's that I've been basing some of my
current work off of - I'm unclear of the need for that change, but it looks
like we might be able to avoid it.
I'm investigating whether we can get away with simply shading and
relocating Guava in this case and leave a Curator upgrade as a follow-on
for a point release. It appears that Nick was able to get maas tests
running with an upgraded Hadoop without having to change the curator
versions. There are probably more classpath issues with a Hadoop upgrade
and Curator whether or not we rev the version, so that should be a wash in
risk and effort. However, the one added big risk with upgrading Curator is
that it's 1. a couple major version changes we're upgrading, and 2. we
already ran into one change in functionality based on the issues seen in
https://github.com/apache/metron/pull/1516. We still have more work to do
on upgrading Hadoop as well as Kerberos once Hadoop is finished being
upgraded. This means we could have a much higher probability of more
functionality changes as we get further into testing. It's possible there
are other reasons to motivate a Curator upgrade now, but this may be
premature without more concrete details and having tried a smaller scope
change with dependency relocation first.
Mike
On Tue, Sep 17, 2019 at 4:04 PM Otto Fowler <ot...@gmail.com> wrote:
> +1
>
>
>
>
> On September 17, 2019 at 14:53:13, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> Also, caught this come through the HBase dev list -
> https://issues.apache.org/jira/browse/HBASE-23032. Might be worth a look
> at
> 4.2.0 while we're at it and possibly also avoid this issue that I was
> running into before when trying to use curator-tests 4.x
> https://issues.apache.org/jira/browse/CURATOR-446. Fixed in 4.0.1, which
> would naturally be included with 4.2.0 assuming we don't run into any
> other
> classpath or API compatibility issues.
>
> On Tue, Sep 17, 2019 at 11:20 AM Nick Allen <ni...@nickallen.org> wrote:
>
> > +1 to making the change on the feature branch.
> >
> > We don't really know how this might affect master which is still
> building
> > against HDP 2.6, nor is it strictly needed there. Going to Curator 4.0.0
> > is only needed due to the HDP 3.1 upgrade. This is also likely to get
> > more focused testing cycles in the feature branch before it has a chance
> to
> > break anything in master.
> >
> > On Tue, Sep 17, 2019 at 1:13 PM Michael Miklavcic <
> > michael.miklavcic@gmail.com> wrote:
> >
> > > Hey all,
> > >
> > > While working through the feature branch upgrade for HDP 3.1, we came
> > > across some classpath related issues conflicting with Storm and Guava
> > while
> > > testing out Kerberos. Initially, this seemed reasonable to roll into
> the
> > > Kerberos fix PR, but the scope has expanded a bit because we found
> > > additional impacts while working through the Hadoop upgrade branch as
> > well.
> > > We're now looking at splitting this out into a separate PR in the
> feature
> > > branch. Curator 4.0.0 is backwards compatible and will work with
> > Zookeeper
> > > 3.4.6. Strictly speaking, this *could* be done against master first,
> but
> > > there may be some duplicate testing effort there, and I'm really not
> it's
> > > worth it as there is absolutely no issue currently with Curator 2.12.0
> in
> > > master - I call this out purely for transparency/information share
> with
> > the
> > > community. I'm leaning towards us making this change against the
> feature
> > > branch directly as that is the only place we currently have
> compatibility
> > > issues.
> > >
> > > Best,
> > > Mike
> > >
> >
>
>
Re: [DISCUSS] Curator client upgrade
Posted by Otto Fowler <ot...@gmail.com>.
+1
On September 17, 2019 at 14:53:13, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:
Also, caught this come through the HBase dev list -
https://issues.apache.org/jira/browse/HBASE-23032. Might be worth a look at
4.2.0 while we're at it and possibly also avoid this issue that I was
running into before when trying to use curator-tests 4.x
https://issues.apache.org/jira/browse/CURATOR-446. Fixed in 4.0.1, which
would naturally be included with 4.2.0 assuming we don't run into any other
classpath or API compatibility issues.
On Tue, Sep 17, 2019 at 11:20 AM Nick Allen <ni...@nickallen.org> wrote:
> +1 to making the change on the feature branch.
>
> We don't really know how this might affect master which is still building
> against HDP 2.6, nor is it strictly needed there. Going to Curator 4.0.0
> is only needed due to the HDP 3.1 upgrade. This is also likely to get
> more focused testing cycles in the feature branch before it has a chance
to
> break anything in master.
>
> On Tue, Sep 17, 2019 at 1:13 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > Hey all,
> >
> > While working through the feature branch upgrade for HDP 3.1, we came
> > across some classpath related issues conflicting with Storm and Guava
> while
> > testing out Kerberos. Initially, this seemed reasonable to roll into
the
> > Kerberos fix PR, but the scope has expanded a bit because we found
> > additional impacts while working through the Hadoop upgrade branch as
> well.
> > We're now looking at splitting this out into a separate PR in the
feature
> > branch. Curator 4.0.0 is backwards compatible and will work with
> Zookeeper
> > 3.4.6. Strictly speaking, this *could* be done against master first,
but
> > there may be some duplicate testing effort there, and I'm really not
it's
> > worth it as there is absolutely no issue currently with Curator 2.12.0
in
> > master - I call this out purely for transparency/information share with
> the
> > community. I'm leaning towards us making this change against the
feature
> > branch directly as that is the only place we currently have
compatibility
> > issues.
> >
> > Best,
> > Mike
> >
>
Re: [DISCUSS] Curator client upgrade
Posted by Michael Miklavcic <mi...@gmail.com>.
Also, caught this come through the HBase dev list -
https://issues.apache.org/jira/browse/HBASE-23032. Might be worth a look at
4.2.0 while we're at it and possibly also avoid this issue that I was
running into before when trying to use curator-tests 4.x
https://issues.apache.org/jira/browse/CURATOR-446. Fixed in 4.0.1, which
would naturally be included with 4.2.0 assuming we don't run into any other
classpath or API compatibility issues.
On Tue, Sep 17, 2019 at 11:20 AM Nick Allen <ni...@nickallen.org> wrote:
> +1 to making the change on the feature branch.
>
> We don't really know how this might affect master which is still building
> against HDP 2.6, nor is it strictly needed there. Going to Curator 4.0.0
> is only needed due to the HDP 3.1 upgrade. This is also likely to get
> more focused testing cycles in the feature branch before it has a chance to
> break anything in master.
>
> On Tue, Sep 17, 2019 at 1:13 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > Hey all,
> >
> > While working through the feature branch upgrade for HDP 3.1, we came
> > across some classpath related issues conflicting with Storm and Guava
> while
> > testing out Kerberos. Initially, this seemed reasonable to roll into the
> > Kerberos fix PR, but the scope has expanded a bit because we found
> > additional impacts while working through the Hadoop upgrade branch as
> well.
> > We're now looking at splitting this out into a separate PR in the feature
> > branch. Curator 4.0.0 is backwards compatible and will work with
> Zookeeper
> > 3.4.6. Strictly speaking, this *could* be done against master first, but
> > there may be some duplicate testing effort there, and I'm really not it's
> > worth it as there is absolutely no issue currently with Curator 2.12.0 in
> > master - I call this out purely for transparency/information share with
> the
> > community. I'm leaning towards us making this change against the feature
> > branch directly as that is the only place we currently have compatibility
> > issues.
> >
> > Best,
> > Mike
> >
>
Re: [DISCUSS] Curator client upgrade
Posted by Nick Allen <ni...@nickallen.org>.
+1 to making the change on the feature branch.
We don't really know how this might affect master which is still building
against HDP 2.6, nor is it strictly needed there. Going to Curator 4.0.0
is only needed due to the HDP 3.1 upgrade. This is also likely to get
more focused testing cycles in the feature branch before it has a chance to
break anything in master.
On Tue, Sep 17, 2019 at 1:13 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> Hey all,
>
> While working through the feature branch upgrade for HDP 3.1, we came
> across some classpath related issues conflicting with Storm and Guava while
> testing out Kerberos. Initially, this seemed reasonable to roll into the
> Kerberos fix PR, but the scope has expanded a bit because we found
> additional impacts while working through the Hadoop upgrade branch as well.
> We're now looking at splitting this out into a separate PR in the feature
> branch. Curator 4.0.0 is backwards compatible and will work with Zookeeper
> 3.4.6. Strictly speaking, this *could* be done against master first, but
> there may be some duplicate testing effort there, and I'm really not it's
> worth it as there is absolutely no issue currently with Curator 2.12.0 in
> master - I call this out purely for transparency/information share with the
> community. I'm leaning towards us making this change against the feature
> branch directly as that is the only place we currently have compatibility
> issues.
>
> Best,
> Mike
>