You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Liren Ding <sk...@gmail.com> on 2016/01/28 20:33:35 UTC
Hadoop library version issues and conflict
My MR job use following libs:
- commons-codec-1.9,
- commons-compress-1.7,
- guava-12.0.1,
- httpclient-4.3.1
- httpcore-4.
But the versions in hadoop libs are:
- commons-codec-1.4
- commons-compress-1.4.1
- guava-11.0.2
- httpclient-4.2.5
- httpcore-4.2.5
This conflict causes a "java.lang.NoSuchMethodError" error. One insane
solution is replacing all libs allover the cluster with new versions. But
the approach is not practical in production environment. Is there a decent
way to solve the hadoop libraries version conflict issue?
Re: Hadoop library version issues and conflict
Posted by Liren Ding <sk...@gmail.com>.
I saw a few posts mentioned the property "mapreduce.job.user.classpath.first"
could be used to handle this situation. But it seems to have been
deprecated in yarn. Does anyone know any alternative?
On Thu, Jan 28, 2016 at 12:19 PM, Solomon Duskis <sd...@gmail.com> wrote:
> I'm not sure about the solution here, but it would be nice to have much
> newer dependencies of everything. There are a host of issues with newer
> libraries running in the same VM as hbase and hadoop. Even the newer hbase
> libraries are 3+ year old.
>
> On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
> wrote:
>
>> My MR job use following libs:
>>
>> - commons-codec-1.9,
>> - commons-compress-1.7,
>> - guava-12.0.1,
>> - httpclient-4.3.1
>> - httpcore-4.
>>
>> But the versions in hadoop libs are:
>>
>> - commons-codec-1.4
>> - commons-compress-1.4.1
>> - guava-11.0.2
>> - httpclient-4.2.5
>> - httpcore-4.2.5
>>
>> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
>> solution is replacing all libs allover the cluster with new versions. But
>> the approach is not practical in production environment. Is there a decent
>> way to solve the hadoop libraries version conflict issue?
>>
>
>
Re: Hadoop library version issues and conflict
Posted by Liren Ding <sk...@gmail.com>.
I saw a few posts mentioned the property "mapreduce.job.user.classpath.first"
could be used to handle this situation. But it seems to have been
deprecated in yarn. Does anyone know any alternative?
On Thu, Jan 28, 2016 at 12:19 PM, Solomon Duskis <sd...@gmail.com> wrote:
> I'm not sure about the solution here, but it would be nice to have much
> newer dependencies of everything. There are a host of issues with newer
> libraries running in the same VM as hbase and hadoop. Even the newer hbase
> libraries are 3+ year old.
>
> On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
> wrote:
>
>> My MR job use following libs:
>>
>> - commons-codec-1.9,
>> - commons-compress-1.7,
>> - guava-12.0.1,
>> - httpclient-4.3.1
>> - httpcore-4.
>>
>> But the versions in hadoop libs are:
>>
>> - commons-codec-1.4
>> - commons-compress-1.4.1
>> - guava-11.0.2
>> - httpclient-4.2.5
>> - httpcore-4.2.5
>>
>> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
>> solution is replacing all libs allover the cluster with new versions. But
>> the approach is not practical in production environment. Is there a decent
>> way to solve the hadoop libraries version conflict issue?
>>
>
>
Re: Hadoop library version issues and conflict
Posted by Liren Ding <sk...@gmail.com>.
I saw a few posts mentioned the property "mapreduce.job.user.classpath.first"
could be used to handle this situation. But it seems to have been
deprecated in yarn. Does anyone know any alternative?
On Thu, Jan 28, 2016 at 12:19 PM, Solomon Duskis <sd...@gmail.com> wrote:
> I'm not sure about the solution here, but it would be nice to have much
> newer dependencies of everything. There are a host of issues with newer
> libraries running in the same VM as hbase and hadoop. Even the newer hbase
> libraries are 3+ year old.
>
> On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
> wrote:
>
>> My MR job use following libs:
>>
>> - commons-codec-1.9,
>> - commons-compress-1.7,
>> - guava-12.0.1,
>> - httpclient-4.3.1
>> - httpcore-4.
>>
>> But the versions in hadoop libs are:
>>
>> - commons-codec-1.4
>> - commons-compress-1.4.1
>> - guava-11.0.2
>> - httpclient-4.2.5
>> - httpcore-4.2.5
>>
>> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
>> solution is replacing all libs allover the cluster with new versions. But
>> the approach is not practical in production environment. Is there a decent
>> way to solve the hadoop libraries version conflict issue?
>>
>
>
Re: Hadoop library version issues and conflict
Posted by Liren Ding <sk...@gmail.com>.
I saw a few posts mentioned the property "mapreduce.job.user.classpath.first"
could be used to handle this situation. But it seems to have been
deprecated in yarn. Does anyone know any alternative?
On Thu, Jan 28, 2016 at 12:19 PM, Solomon Duskis <sd...@gmail.com> wrote:
> I'm not sure about the solution here, but it would be nice to have much
> newer dependencies of everything. There are a host of issues with newer
> libraries running in the same VM as hbase and hadoop. Even the newer hbase
> libraries are 3+ year old.
>
> On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
> wrote:
>
>> My MR job use following libs:
>>
>> - commons-codec-1.9,
>> - commons-compress-1.7,
>> - guava-12.0.1,
>> - httpclient-4.3.1
>> - httpcore-4.
>>
>> But the versions in hadoop libs are:
>>
>> - commons-codec-1.4
>> - commons-compress-1.4.1
>> - guava-11.0.2
>> - httpclient-4.2.5
>> - httpcore-4.2.5
>>
>> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
>> solution is replacing all libs allover the cluster with new versions. But
>> the approach is not practical in production environment. Is there a decent
>> way to solve the hadoop libraries version conflict issue?
>>
>
>
Re: Hadoop library version issues and conflict
Posted by Solomon Duskis <sd...@gmail.com>.
I'm not sure about the solution here, but it would be nice to have much
newer dependencies of everything. There are a host of issues with newer
libraries running in the same VM as hbase and hadoop. Even the newer hbase
libraries are 3+ year old.
On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
wrote:
> My MR job use following libs:
>
> - commons-codec-1.9,
> - commons-compress-1.7,
> - guava-12.0.1,
> - httpclient-4.3.1
> - httpcore-4.
>
> But the versions in hadoop libs are:
>
> - commons-codec-1.4
> - commons-compress-1.4.1
> - guava-11.0.2
> - httpclient-4.2.5
> - httpcore-4.2.5
>
> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
> solution is replacing all libs allover the cluster with new versions. But
> the approach is not practical in production environment. Is there a decent
> way to solve the hadoop libraries version conflict issue?
>
Re: Hadoop library version issues and conflict
Posted by Solomon Duskis <sd...@gmail.com>.
I'm not sure about the solution here, but it would be nice to have much
newer dependencies of everything. There are a host of issues with newer
libraries running in the same VM as hbase and hadoop. Even the newer hbase
libraries are 3+ year old.
On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
wrote:
> My MR job use following libs:
>
> - commons-codec-1.9,
> - commons-compress-1.7,
> - guava-12.0.1,
> - httpclient-4.3.1
> - httpcore-4.
>
> But the versions in hadoop libs are:
>
> - commons-codec-1.4
> - commons-compress-1.4.1
> - guava-11.0.2
> - httpclient-4.2.5
> - httpcore-4.2.5
>
> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
> solution is replacing all libs allover the cluster with new versions. But
> the approach is not practical in production environment. Is there a decent
> way to solve the hadoop libraries version conflict issue?
>
Re: Hadoop library version issues and conflict
Posted by Solomon Duskis <sd...@gmail.com>.
I'm not sure about the solution here, but it would be nice to have much
newer dependencies of everything. There are a host of issues with newer
libraries running in the same VM as hbase and hadoop. Even the newer hbase
libraries are 3+ year old.
On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
wrote:
> My MR job use following libs:
>
> - commons-codec-1.9,
> - commons-compress-1.7,
> - guava-12.0.1,
> - httpclient-4.3.1
> - httpcore-4.
>
> But the versions in hadoop libs are:
>
> - commons-codec-1.4
> - commons-compress-1.4.1
> - guava-11.0.2
> - httpclient-4.2.5
> - httpcore-4.2.5
>
> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
> solution is replacing all libs allover the cluster with new versions. But
> the approach is not practical in production environment. Is there a decent
> way to solve the hadoop libraries version conflict issue?
>
Re: Hadoop library version issues and conflict
Posted by Solomon Duskis <sd...@gmail.com>.
I'm not sure about the solution here, but it would be nice to have much
newer dependencies of everything. There are a host of issues with newer
libraries running in the same VM as hbase and hadoop. Even the newer hbase
libraries are 3+ year old.
On Thu, Jan 28, 2016 at 2:33 PM, Liren Ding <sk...@gmail.com>
wrote:
> My MR job use following libs:
>
> - commons-codec-1.9,
> - commons-compress-1.7,
> - guava-12.0.1,
> - httpclient-4.3.1
> - httpcore-4.
>
> But the versions in hadoop libs are:
>
> - commons-codec-1.4
> - commons-compress-1.4.1
> - guava-11.0.2
> - httpclient-4.2.5
> - httpcore-4.2.5
>
> This conflict causes a "java.lang.NoSuchMethodError" error. One insane
> solution is replacing all libs allover the cluster with new versions. But
> the approach is not practical in production environment. Is there a decent
> way to solve the hadoop libraries version conflict issue?
>