You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Mohit Sabharwal (JIRA)" <ji...@apache.org> on 2016/04/04 22:49:25 UTC
[jira] [Commented] (PIG-4859) Need upgrade snappy-java.version to
1.1.1.3
[ https://issues.apache.org/jira/browse/PIG-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225017#comment-15225017 ]
Mohit Sabharwal commented on PIG-4859:
--------------------------------------
+1 (non-binding)
> Need upgrade snappy-java.version to 1.1.1.3
> -------------------------------------------
>
> Key: PIG-4859
> URL: https://issues.apache.org/jira/browse/PIG-4859
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4859.patch
>
>
> run pig on spark on yarn-client env as following:
> export SPARK_MASTER="yarn-client"
> ./pig -x spark xxx.pig
> Throw error like following:
> {code}
> main] 2016-03-30 16:52:26,115 INFO scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Job 0 failed: saveAsNewAPIHadoopDataset at StoreConverter.java:101, took 73.980147 s
> 19895 [main] 2016-03-30 16:52:26,119 ERROR spark.JobGraphBuilder (JobGraphBuilder.java:sparkOperToRDD(166)) - throw exception in sparkOperToRDD:
> 19896 org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, zly1.sh.intel.com): java .lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
> 19897 at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> 19898 at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541)
> 19899 at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350)
> 19900 at org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158)
> 19901 at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
> 19902 at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313)
> 19903 at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326)
> 19904 at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
> 19905 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
> 19906 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
> 19907 at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:64)
> 19908 at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:64)
> 19909 at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:103)
> 19910 at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216)
> 19911 at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)