You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Srikanth Sundarrajan (JIRA)" <ji...@apache.org> on 2015/10/06 06:47:27 UTC
[jira] [Commented] (PIG-4693) Class conflicts: Kryo bundled in
spark vs kryo bundled with pig
[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944494#comment-14944494 ]
Srikanth Sundarrajan commented on PIG-4693:
-------------------------------------------
Running the following simple pig script
{code}
IN = LOAD 'test-data' USING PigStorage('');
G = GROUP IN BY $11;
R = FOREACH G GENERATE group, SUM(IN.$10);
STORE R INTO 'test-out' USING PigStorage(',');
{code}
results in
{noformat}
ERROR 2998: Unhandled internal error. com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V
java.lang.NoSuchMethodError: com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V
at com.twitter.chill.KryoBase.setInstantiatorStrategy(KryoBase.scala:86)
at com.twitter.chill.EmptyScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:59)
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:80)
at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:227)
at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:212)
at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:128)
at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:85)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1291)
at org.apache.spark.rdd.NewHadoopRDD.<init>(NewHadoopRDD.scala:77)
at org.apache.spark.SparkContext$$anonfun$newAPIHadoopRDD$1.apply(SparkContext.scala:1099)
at org.apache.spark.SparkContext$$anonfun$newAPIHadoopRDD$1.apply(SparkContext.scala:1094)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:681)
at org.apache.spark.SparkContext.newAPIHadoopRDD(SparkContext.scala:1094)
at org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:88)
at org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:58)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:636)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.sparkOperToRDD(SparkLauncher.java:555)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.sparkPlanToRDD(SparkLauncher.java:504)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:206)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:301)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
at org.apache.pig.PigServer.execute(PigServer.java:1364)
at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:624)
at org.apache.pig.Main.main(Main.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
================================================================================
{noformat}
Here is the logs from the class-loader:
{noformat}
[Loaded org.apache.spark.broadcast.TorrentBroadcast$$anonfun$5 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded org.apache.spark.serializer.KryoSerializationStream from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded org.apache.spark.serializer.KryoDeserializationStream from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.twitter.chill.KryoInstantiator from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.twitter.chill.EmptyScalaKryoInstantiator from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.twitter.chill.KryoInstantiator$3 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.twitter.chill.KryoInstantiator$2 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.twitter.chill.KryoInstantiator$1 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.twitter.chill.KryoInstantiator$4 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded org.objenesis.strategy.InstantiatorStrategy from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.esotericsoftware.kryo.KryoException from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.ClassResolver from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.StreamFactory from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.factories.SerializerFactory from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$IntSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$BooleanSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$ByteSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$CharSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$ShortSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$LongSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$DoubleSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$VoidSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.kryo.ReferenceResolver from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.esotericsoftware.shaded.org.objenesis.instantiator.ObjectInstantiator from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
[Loaded com.twitter.chill.ObjectSerializer from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar]
[Loaded com.esotericsoftware.kryo.serializers.FieldSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-2.6.0.2.2.0.0-2041.jar]
[Loaded com.esotericsoftware.kryo.util.DefaultClassResolver from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar]
{noformat}
org.objenesis.strategy.InstantiatorStrategy is used to set the InstantiatorStrategy in Kryo, where Kryo in 2.22 requires shaded com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy.class and hence the error.
I guess Kryo is pulled in as a dependency for OrcStorage.
> Class conflicts: Kryo bundled in spark vs kryo bundled with pig
> ---------------------------------------------------------------
>
> Key: PIG-4693
> URL: https://issues.apache.org/jira/browse/PIG-4693
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Affects Versions: spark-branch
> Reporter: Srikanth Sundarrajan
> Assignee: Srikanth Sundarrajan
> Labels: spork
> Fix For: spark-branch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)