You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Xuefu Zhang <xz...@cloudera.com> on 2015/10/13 05:54:24 UTC
Dependency version on Kryo
Hi all,
It was found in PIG-4693 (https://issues.apache.org/jira/browse/PIG-4693)
that Pig is currently dependent on Kryo 2.22. However, Spark depends on
2.21. The two versions are not completely compatible. We tried several ways
to solve the problem but unfortunately none worked. This is mainly because
Spark doesn't give user an opportunity to provide their own kryo library
(SPARK-10910). Please refer to the full discussions in PIG-4693.
It seems that Pig brought in kryo dependency for ORC. I'm wondering if
there is any specific reasons for kryo 2.22 and if not, whether we can
downgrade the dependency to 2.21 instead. Our initial test shows that kryo
2.21 works just fine for ORC. This obviously solve our problem as well.
Your input to this is greatly appreciated.
Thanks,
Xuefu
Re: Dependency version on Kryo
Posted by Rohini Palaniswamy <ro...@gmail.com>.
That should be fine. We wanted to get rid of the kryo dependency in ORC and
use the shaded one that hive uses. But that is in hive-exec jar which is
huge and has too many other jars packed in and did not want to add that as
dependency to Pig.
On Mon, Oct 12, 2015 at 8:54 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> Hi all,
>
> It was found in PIG-4693 (https://issues.apache.org/jira/browse/PIG-4693)
> that Pig is currently dependent on Kryo 2.22. However, Spark depends on
> 2.21. The two versions are not completely compatible. We tried several ways
> to solve the problem but unfortunately none worked. This is mainly because
> Spark doesn't give user an opportunity to provide their own kryo library
> (SPARK-10910). Please refer to the full discussions in PIG-4693.
>
> It seems that Pig brought in kryo dependency for ORC. I'm wondering if
> there is any specific reasons for kryo 2.22 and if not, whether we can
> downgrade the dependency to 2.21 instead. Our initial test shows that kryo
> 2.21 works just fine for ORC. This obviously solve our problem as well.
>
> Your input to this is greatly appreciated.
>
> Thanks,
> Xuefu
>