You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by David <da...@gmail.com> on 2021/02/05 18:56:42 UTC
Hadoop Thirdparty
Hello Gang,
I'm not sure I understand that "Hadoop Thirdparty" project. I mean, I
understand that it's a collection of shaded third party libraries, but the
way it is setup currently seems self-defeating. What if I have multiple
versions of the Hadoop Thirdparty library on my classpath?
It seems to me that the "relocation" space should be different for each
version.
<shaded.prefix>org.apache.hadoop.thirdparty</shaded.prefix>
<relocation>
<pattern>com/google/</pattern>
<shadedPattern>${shaded.prefix}/com/google/</shadedPattern>
</relocation>
Wouldn't it make more sense perhaps to do
<shaded.prefix>org.apache.hadoop.thirdparty{$version}</shaded.prefix>
So that even between versions of the thirdparty JAR, there are no clashing
namespaces?
Thanks.
Re: Hadoop Thirdparty
Posted by Ayush Saxena <ay...@gmail.com>.
If the application has multiple versions of hadoop as dependency why only thirdparty jar will conflict? Will the other hadoop jars not conflict?
Well appending the version in the package name won’t be a good idea, for every thirdparty release we need to change the imports in the hadoop project, and I don’t think this version number stuff is practised anywhere widely.
Looks like something to be handled at application end.
-Ayush
> On 06-Feb-2021, at 4:13 AM, Wei-Chiu Chuang <we...@cloudera.com.invalid> wrote:
>
> it's the concept borrowed from HBase. (there's a hbase-thirdparty and it
> works quite well there )
>
> it was meant to prevent class name clashes with downstream applications if
> they use the same dependency but incompatible versions. (e.g. guava 11 vs
> guava 27)
>
> Your scenario could happen if the application has multiple hadoop versions
> in the dependency. Given how many layers of applications are built on top
> of Hadoop, it seems possible.
>
>> On Sat, Feb 6, 2021 at 2:57 AM David <da...@gmail.com> wrote:
>>
>> Hello Gang,
>>
>> I'm not sure I understand that "Hadoop Thirdparty" project. I mean, I
>> understand that it's a collection of shaded third party libraries, but the
>> way it is setup currently seems self-defeating. What if I have multiple
>> versions of the Hadoop Thirdparty library on my classpath?
>>
>> It seems to me that the "relocation" space should be different for each
>> version.
>>
>> <shaded.prefix>org.apache.hadoop.thirdparty</shaded.prefix>
>>
>> <relocation>
>> <pattern>com/google/</pattern>
>>
>> <shadedPattern>${shaded.prefix}/com/google/</shadedPattern>
>> </relocation>
>>
>> Wouldn't it make more sense perhaps to do
>> <shaded.prefix>org.apache.hadoop.thirdparty{$version}</shaded.prefix>
>>
>> So that even between versions of the thirdparty JAR, there are no clashing
>> namespaces?
>>
>> Thanks.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Hadoop Thirdparty
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
it's the concept borrowed from HBase. (there's a hbase-thirdparty and it
works quite well there )
it was meant to prevent class name clashes with downstream applications if
they use the same dependency but incompatible versions. (e.g. guava 11 vs
guava 27)
Your scenario could happen if the application has multiple hadoop versions
in the dependency. Given how many layers of applications are built on top
of Hadoop, it seems possible.
On Sat, Feb 6, 2021 at 2:57 AM David <da...@gmail.com> wrote:
> Hello Gang,
>
> I'm not sure I understand that "Hadoop Thirdparty" project. I mean, I
> understand that it's a collection of shaded third party libraries, but the
> way it is setup currently seems self-defeating. What if I have multiple
> versions of the Hadoop Thirdparty library on my classpath?
>
> It seems to me that the "relocation" space should be different for each
> version.
>
> <shaded.prefix>org.apache.hadoop.thirdparty</shaded.prefix>
>
> <relocation>
> <pattern>com/google/</pattern>
>
> <shadedPattern>${shaded.prefix}/com/google/</shadedPattern>
> </relocation>
>
> Wouldn't it make more sense perhaps to do
> <shaded.prefix>org.apache.hadoop.thirdparty{$version}</shaded.prefix>
>
> So that even between versions of the thirdparty JAR, there are no clashing
> namespaces?
>
> Thanks.
>