You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Sean Mackrory <ma...@gmail.com> on 2016/09/28 16:56:19 UTC
Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
HBase folks,
I'd like to work on upgrading some of the dependencies in Hadoop that are
starting to lag behind:
jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
commons-configuration (https://issues.apache.org/jira/browse/HADOOP-13660)
htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
Some of these (chiefly jackson) are known to be high risk upgrades and have
caused problems for HBase in the past. I wanted to give you a heads up
about this work and coordinate with you to make sure I do all the testing
you think is necessary to discover any potential problems well in advance
of the breakages.
I was going to look at just running HBase's unit tests when built against
my Hadoop's trunk with my changes (Should I expect any problems here? Is
there a branch already focused on Hadoop 3 compatibility or anything?) and
Bigtop's integration tests too. Anything else you would add to this?
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Ted Yu <yu...@gmail.com>.
Looking at HADOOP-12705, it upgrades jackson2.version.
However, hbase only depends on jackson (1.9.13).
It seems the impact of upgrading jackson2 should be minimal on hbase.
Cheers
On Wed, Sep 28, 2016 at 9:56 AM, Sean Mackrory <ma...@gmail.com> wrote:
> HBase folks,
>
> I'd like to work on upgrading some of the dependencies in Hadoop that are
> starting to lag behind:
>
> jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
> jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
> commons-configuration (https://issues.apache.org/jira/browse/HADOOP-13660)
> htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
>
> Some of these (chiefly jackson) are known to be high risk upgrades and have
> caused problems for HBase in the past. I wanted to give you a heads up
> about this work and coordinate with you to make sure I do all the testing
> you think is necessary to discover any potential problems well in advance
> of the breakages.
>
> I was going to look at just running HBase's unit tests when built against
> my Hadoop's trunk with my changes (Should I expect any problems here? Is
> there a branch already focused on Hadoop 3 compatibility or anything?) and
> Bigtop's integration tests too. Anything else you would add to this?
>
Re: Upgrading Hadoop dependencies and catching potential incompatibilities for HBase
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 28, 2016, at 9:56 AM, Sean Mackrory <ma...@gmail.com> wrote:
> and
> Bigtop's integration tests too. Anything else you would add to this?
Be aware that bigtop does things to the jar layout that will break parts of hadoop 3.
Re: Upgrading Hadoop dependencies and catching potential incompatibilities for HBase
Posted by Andrew Purtell <an...@gmail.com>.
Great suggestion!
> On Sep 28, 2016, at 11:55 AM, Enis Söztutar <en...@gmail.com> wrote:
>
> Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3
> so that we do not have this mess going forward.
>
> Enis
>
>> On Wed, Sep 28, 2016 at 11:19 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>>
>> You can build hbase using the profile by adding this -D setting.
>>
>> mvn clean test install -Dhadoop.profile=3.0
>>
>> At the moment the hbase's hadoop 3 profile has fails at mvn compile,
>> install, and test phases. There are some issues that are patch available
>> addressing these in HBase currently.
>>
>> compile https://issues.apache.org/jira/browse/HBASE-16711
>> test https://issues.apache.org/jira/browse/HBASE-6581
>> install https://issues.apache.org/jira/browse/HBASE-16712
>>
>> IIRC, currently hbase's hadoop3 profile currently tracks the hadoop-3.0
>> SNAPSHOT artifacts. Sean, would these dependencies changes be targeted for
>> this branch and land in the next hadoop 3 alpha/beta?
>> Jon.
>>
>> On Wed, Sep 28, 2016 at 9:56 AM, Sean Mackrory <ma...@gmail.com>
>> wrote:
>>
>>> HBase folks,
>>>
>>> I'd like to work on upgrading some of the dependencies in Hadoop that are
>>> starting to lag behind:
>>>
>>> jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
>>> jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
>>> commons-configuration (https://issues.apache.org/
>> jira/browse/HADOOP-13660)
>>> htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
>>>
>>> Some of these (chiefly jackson) are known to be high risk upgrades and
>> have
>>> caused problems for HBase in the past. I wanted to give you a heads up
>>> about this work and coordinate with you to make sure I do all the testing
>>> you think is necessary to discover any potential problems well in advance
>>> of the breakages.
>>>
>>> I was going to look at just running HBase's unit tests when built against
>>> my Hadoop's trunk with my changes (Should I expect any problems here? Is
>>> there a branch already focused on Hadoop 3 compatibility or anything?)
>> and
>>> Bigtop's integration tests too. Anything else you would add to this?
>>>
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // HBase Tech Lead, Software Engineer, Cloudera
>> // jon@cloudera.com // @jmhsieh
>>
Re: Upgrading Hadoop dependencies and catching potential incompatibilities for HBase
Posted by Andrew Purtell <an...@gmail.com>.
Great suggestion!
> On Sep 28, 2016, at 11:55 AM, Enis Söztutar <en...@gmail.com> wrote:
>
> Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3
> so that we do not have this mess going forward.
>
> Enis
>
>> On Wed, Sep 28, 2016 at 11:19 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>>
>> You can build hbase using the profile by adding this -D setting.
>>
>> mvn clean test install -Dhadoop.profile=3.0
>>
>> At the moment the hbase's hadoop 3 profile has fails at mvn compile,
>> install, and test phases. There are some issues that are patch available
>> addressing these in HBase currently.
>>
>> compile https://issues.apache.org/jira/browse/HBASE-16711
>> test https://issues.apache.org/jira/browse/HBASE-6581
>> install https://issues.apache.org/jira/browse/HBASE-16712
>>
>> IIRC, currently hbase's hadoop3 profile currently tracks the hadoop-3.0
>> SNAPSHOT artifacts. Sean, would these dependencies changes be targeted for
>> this branch and land in the next hadoop 3 alpha/beta?
>> Jon.
>>
>> On Wed, Sep 28, 2016 at 9:56 AM, Sean Mackrory <ma...@gmail.com>
>> wrote:
>>
>>> HBase folks,
>>>
>>> I'd like to work on upgrading some of the dependencies in Hadoop that are
>>> starting to lag behind:
>>>
>>> jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
>>> jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
>>> commons-configuration (https://issues.apache.org/
>> jira/browse/HADOOP-13660)
>>> htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
>>>
>>> Some of these (chiefly jackson) are known to be high risk upgrades and
>> have
>>> caused problems for HBase in the past. I wanted to give you a heads up
>>> about this work and coordinate with you to make sure I do all the testing
>>> you think is necessary to discover any potential problems well in advance
>>> of the breakages.
>>>
>>> I was going to look at just running HBase's unit tests when built against
>>> my Hadoop's trunk with my changes (Should I expect any problems here? Is
>>> there a branch already focused on Hadoop 3 compatibility or anything?)
>> and
>>> Bigtop's integration tests too. Anything else you would add to this?
>>>
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // HBase Tech Lead, Software Engineer, Cloudera
>> // jon@cloudera.com // @jmhsieh
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Enis Söztutar <en...@gmail.com>.
> I'm working on this under HADOOP-11804, using HBase as my test
application.
Cool, glad to see progress on this.
Enis
On Wed, Sep 28, 2016 at 12:15 PM, Sean Busbey <bu...@cloudera.com> wrote:
> On Wed, Sep 28, 2016 at 1:55 PM, Enis Söztutar <en...@gmail.com> wrote:
> > Can Hadoop please shade ALL of the dependencies (including PB) in
> Hadoop-3
> > so that we do not have this mess going forward.
> >
> > Enis
>
>
> I'm working on this under HADOOP-11804, using HBase as my test application.
>
> --
> busbey
>
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Enis Söztutar <en...@gmail.com>.
> I'm working on this under HADOOP-11804, using HBase as my test
application.
Cool, glad to see progress on this.
Enis
On Wed, Sep 28, 2016 at 12:15 PM, Sean Busbey <bu...@cloudera.com> wrote:
> On Wed, Sep 28, 2016 at 1:55 PM, Enis Söztutar <en...@gmail.com> wrote:
> > Can Hadoop please shade ALL of the dependencies (including PB) in
> Hadoop-3
> > so that we do not have this mess going forward.
> >
> > Enis
>
>
> I'm working on this under HADOOP-11804, using HBase as my test application.
>
> --
> busbey
>
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Sep 28, 2016 at 1:55 PM, Enis Söztutar <en...@gmail.com> wrote:
> Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3
> so that we do not have this mess going forward.
>
> Enis
I'm working on this under HADOOP-11804, using HBase as my test application.
--
busbey
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Sep 28, 2016 at 1:55 PM, Enis Söztutar <en...@gmail.com> wrote:
> Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3
> so that we do not have this mess going forward.
>
> Enis
I'm working on this under HADOOP-11804, using HBase as my test application.
--
busbey
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Enis Söztutar <en...@gmail.com>.
Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3
so that we do not have this mess going forward.
Enis
On Wed, Sep 28, 2016 at 11:19 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> You can build hbase using the profile by adding this -D setting.
>
> mvn clean test install -Dhadoop.profile=3.0
>
> At the moment the hbase's hadoop 3 profile has fails at mvn compile,
> install, and test phases. There are some issues that are patch available
> addressing these in HBase currently.
>
> compile https://issues.apache.org/jira/browse/HBASE-16711
> test https://issues.apache.org/jira/browse/HBASE-6581
> install https://issues.apache.org/jira/browse/HBASE-16712
>
> IIRC, currently hbase's hadoop3 profile currently tracks the hadoop-3.0
> SNAPSHOT artifacts. Sean, would these dependencies changes be targeted for
> this branch and land in the next hadoop 3 alpha/beta?
> Jon.
>
> On Wed, Sep 28, 2016 at 9:56 AM, Sean Mackrory <ma...@gmail.com>
> wrote:
>
> > HBase folks,
> >
> > I'd like to work on upgrading some of the dependencies in Hadoop that are
> > starting to lag behind:
> >
> > jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
> > jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
> > commons-configuration (https://issues.apache.org/
> jira/browse/HADOOP-13660)
> > htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
> >
> > Some of these (chiefly jackson) are known to be high risk upgrades and
> have
> > caused problems for HBase in the past. I wanted to give you a heads up
> > about this work and coordinate with you to make sure I do all the testing
> > you think is necessary to discover any potential problems well in advance
> > of the breakages.
> >
> > I was going to look at just running HBase's unit tests when built against
> > my Hadoop's trunk with my changes (Should I expect any problems here? Is
> > there a branch already focused on Hadoop 3 compatibility or anything?)
> and
> > Bigtop's integration tests too. Anything else you would add to this?
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Enis Söztutar <en...@gmail.com>.
Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3
so that we do not have this mess going forward.
Enis
On Wed, Sep 28, 2016 at 11:19 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> You can build hbase using the profile by adding this -D setting.
>
> mvn clean test install -Dhadoop.profile=3.0
>
> At the moment the hbase's hadoop 3 profile has fails at mvn compile,
> install, and test phases. There are some issues that are patch available
> addressing these in HBase currently.
>
> compile https://issues.apache.org/jira/browse/HBASE-16711
> test https://issues.apache.org/jira/browse/HBASE-6581
> install https://issues.apache.org/jira/browse/HBASE-16712
>
> IIRC, currently hbase's hadoop3 profile currently tracks the hadoop-3.0
> SNAPSHOT artifacts. Sean, would these dependencies changes be targeted for
> this branch and land in the next hadoop 3 alpha/beta?
> Jon.
>
> On Wed, Sep 28, 2016 at 9:56 AM, Sean Mackrory <ma...@gmail.com>
> wrote:
>
> > HBase folks,
> >
> > I'd like to work on upgrading some of the dependencies in Hadoop that are
> > starting to lag behind:
> >
> > jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
> > jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
> > commons-configuration (https://issues.apache.org/
> jira/browse/HADOOP-13660)
> > htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
> >
> > Some of these (chiefly jackson) are known to be high risk upgrades and
> have
> > caused problems for HBase in the past. I wanted to give you a heads up
> > about this work and coordinate with you to make sure I do all the testing
> > you think is necessary to discover any potential problems well in advance
> > of the breakages.
> >
> > I was going to look at just running HBase's unit tests when built against
> > my Hadoop's trunk with my changes (Should I expect any problems here? Is
> > there a branch already focused on Hadoop 3 compatibility or anything?)
> and
> > Bigtop's integration tests too. Anything else you would add to this?
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
Re: Upgrading Hadoop dependencies and catching potential
incompatibilities for HBase
Posted by Jonathan Hsieh <jo...@cloudera.com>.
You can build hbase using the profile by adding this -D setting.
mvn clean test install -Dhadoop.profile=3.0
At the moment the hbase's hadoop 3 profile has fails at mvn compile,
install, and test phases. There are some issues that are patch available
addressing these in HBase currently.
compile https://issues.apache.org/jira/browse/HBASE-16711
test https://issues.apache.org/jira/browse/HBASE-6581
install https://issues.apache.org/jira/browse/HBASE-16712
IIRC, currently hbase's hadoop3 profile currently tracks the hadoop-3.0
SNAPSHOT artifacts. Sean, would these dependencies changes be targeted for
this branch and land in the next hadoop 3 alpha/beta?
Jon.
On Wed, Sep 28, 2016 at 9:56 AM, Sean Mackrory <ma...@gmail.com> wrote:
> HBase folks,
>
> I'd like to work on upgrading some of the dependencies in Hadoop that are
> starting to lag behind:
>
> jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
> jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
> commons-configuration (https://issues.apache.org/jira/browse/HADOOP-13660)
> htrace (https://issues.apache.org/jira/browse/HADOOP-13661)
>
> Some of these (chiefly jackson) are known to be high risk upgrades and have
> caused problems for HBase in the past. I wanted to give you a heads up
> about this work and coordinate with you to make sure I do all the testing
> you think is necessary to discover any potential problems well in advance
> of the breakages.
>
> I was going to look at just running HBase's unit tests when built against
> my Hadoop's trunk with my changes (Should I expect any problems here? Is
> there a branch already focused on Hadoop 3 compatibility or anything?) and
> Bigtop's integration tests too. Anything else you would add to this?
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: Upgrading Hadoop dependencies and catching potential incompatibilities for HBase
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 28, 2016, at 9:56 AM, Sean Mackrory <ma...@gmail.com> wrote:
> and
> Bigtop's integration tests too. Anything else you would add to this?
Be aware that bigtop does things to the jar layout that will break parts of hadoop 3.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org