You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Antonio Jesus Navarro (JIRA)" <ji...@apache.org> on 2014/10/30 09:18:33 UTC
[jira] [Updated] (SPARK-4133) PARSING_ERROR(2) when upgrading
issues from 1.0.2 to 1.1.0
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Antonio Jesus Navarro updated SPARK-4133:
-----------------------------------------
Description:
Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0.
We can not run an existing 1.0.2 spark app if upgraded to 1.1.0
> IOException is thrown by snappy (parsing_error(2))
> Only spark version changed
As far as we have checked, snappy will throw this error when dealing with zero bytes length arrays.
We have tried:
> Changing from snappy to LZF,
> Changing broadcast.compression false
> Changing from TorrentBroadcast to HTTPBroadcast.
but with no luck for the moment.
{code}
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Level for block broadcast_0 is StorageLevel(true, true, false, true, 1)
Executor task launch worker-0 DEBUG storage.BlockManager - Getting block broadcast_0 from memory
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0
Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0
Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 not registered locally
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started reading broadcast variable 0
sparkDriver-akka.actor.default-dispatcher-4 INFO receiver.ReceiverSupervisorImpl - Registered receiver 0
Executor task launch worker-0 INFO util.RecurringTimer - Started timer for BlockGenerator at time 1414656492400
Executor task launch worker-0 INFO receiver.BlockGenerator - Started BlockGenerator
Thread-87 INFO receiver.BlockGenerator - Started block pushing thread
Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting receiver
sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - Registered receiver for stream 0 from akka://sparkDriver
Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka Consumer Stream with group: stratioStreaming
Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to Zookeeper: node.stratio.com:2181
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.442354 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.412421 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] handled message (8.385471 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying properties
Executor task launch worker-0 INFO utils.VerifiableProperties - Property group.id is overridden to stratioStreaming
Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connect is overridden to node.stratio.com:2181
Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connection.timeout.ms is overridden to 10000
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading broadcast variable 0 took 0.033998997 s
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to zookeeper instance at node.stratio.com:2181
Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new ZookKeeper instance to connect to node.stratio.com:2181.
ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread - Starting ZkClient event thread.
Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client connection, connectString=node.stratio.com:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81
Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection to Zookeeper server
Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper state SyncConnected
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Opening socket connection to server node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using SASL (unknown error)
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Socket connection established to node.stratio.com/172.19.0.96:2181, initiating session
Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG zookeeper.ClientCnxn - Session establishment request sent on node.stratio.com/172.19.0.96:2181
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Session establishment complete on server node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710002, negotiated timeout = 6000
Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received event: WatchedEvent state:SyncConnected type:None path:null
Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper state changed (SyncConnected)
Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving process event
Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected
RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for BlockGenerator called at time 1414656492400
Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task scheduler.
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], starting auto committer every 60000 ms
Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms.
Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to node.stratio.com:2181
Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], entering consume
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], begin registering consumer stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a in ZK
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 not registered locally
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started reading broadcast variable 0
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading broadcast variable 0 took 5.5676E-5 s
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl - parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl - parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (1.213476 ms) StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (1.543991 ms) StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 failed 1 times; aborting job
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 failed 1 times; aborting job
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet 0.0, whose tasks have all completed, from pool
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet 0.0, whose tasks have all completed, from pool
sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl - Cancelling stage 0
sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl - Cancelling stage 0
Thread-84 INFO scheduler.DAGScheduler - Failed to run runJob at ReceiverTracker.scala:275
Thread-85 INFO scheduler.DAGScheduler - Failed to run runJob at ReceiverTracker.scala:275
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - Removing running stage 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - Removing running stage 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - After removal of stage 0, remaining stages = 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - After removal of stage 0, remaining stages = 0
Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710002, packet:: clientPath:null serverPath:null finished:false header:: 1,1 replyHeader:: 1,25,-101 request:: '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f616374696f6e223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363536343932343737227d,v{s{31,s{'world,'anyone}}},1 response::
{code}
was:
Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0.
We can not run an existing 1.0.2 spark app if upgraded to 1.1.0
> IOException is thrown by snappy (parsing_error(2))
> Only spark version changed
As far as we have checked, snappy will throw this error when dealing with zero bytes length arrays.
We have tried:
> Changing from snappy to LZF,
> Changing broadcast.compression false
> Changing from TorrentBroadcast to HTTPBroadcast.
but with no luck for the moment.
{code}
[ERROR] 2014-10-29 11:23:26,396 [Executor task launch worker-0] org.apache.spark.executor.Executor logError - Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
> PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0
> ----------------------------------------------------------
>
> Key: SPARK-4133
> URL: https://issues.apache.org/jira/browse/SPARK-4133
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.1.0
> Reporter: Antonio Jesus Navarro
> Priority: Blocker
>
> Snappy related problems found when trying to upgrade existing Spark Streaming App from 1.0.2 to 1.1.0.
> We can not run an existing 1.0.2 spark app if upgraded to 1.1.0
> > IOException is thrown by snappy (parsing_error(2))
> > Only spark version changed
> As far as we have checked, snappy will throw this error when dealing with zero bytes length arrays.
> We have tried:
> > Changing from snappy to LZF,
> > Changing broadcast.compression false
> > Changing from TorrentBroadcast to HTTPBroadcast.
> but with no luck for the moment.
> {code}
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0
> Executor task launch worker-0 DEBUG storage.BlockManager - Level for block broadcast_0 is StorageLevel(true, true, false, true, 1)
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting block broadcast_0 from memory
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0
> Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0
> Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 not registered locally
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started reading broadcast variable 0
> sparkDriver-akka.actor.default-dispatcher-4 INFO receiver.ReceiverSupervisorImpl - Registered receiver 0
> Executor task launch worker-0 INFO util.RecurringTimer - Started timer for BlockGenerator at time 1414656492400
> Executor task launch worker-0 INFO receiver.BlockGenerator - Started BlockGenerator
> Thread-87 INFO receiver.BlockGenerator - Started block pushing thread
> Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting receiver
> sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker - Registered receiver for stream 0 from akka://sparkDriver
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka Consumer Stream with group: stratioStreaming
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to Zookeeper: node.stratio.com:2181
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.442354 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (8.412421 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor] handled message (8.385471 ms) StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from Actor[akka://sparkDriver/deadLetters]
> Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying properties
> Executor task launch worker-0 INFO utils.VerifiableProperties - Property group.id is overridden to stratioStreaming
> Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connect is overridden to node.stratio.com:2181
> Executor task launch worker-0 INFO utils.VerifiableProperties - Property zookeeper.connection.timeout.ms is overridden to 10000
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading broadcast variable 0 took 0.033998997 s
> Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to zookeeper instance at node.stratio.com:2181
> Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new ZookKeeper instance to connect to node.stratio.com:2181.
> ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread - Starting ZkClient event thread.
> Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client connection, connectString=node.stratio.com:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81
> Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection to Zookeeper server
> Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper state SyncConnected
> Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Opening socket connection to server node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using SASL (unknown error)
> Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Socket connection established to node.stratio.com/172.19.0.96:2181, initiating session
> Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG zookeeper.ClientCnxn - Session establishment request sent on node.stratio.com/172.19.0.96:2181
> Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO zookeeper.ClientCnxn - Session establishment complete on server node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710002, negotiated timeout = 6000
> Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received event: WatchedEvent state:SyncConnected type:None path:null
> Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper state changed (SyncConnected)
> Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving process event
> Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected
> RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for BlockGenerator called at time 1414656492400
> Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task scheduler.
> Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], starting auto committer every 60000 ms
> Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms.
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to node.stratio.com:2181
> Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], entering consume
> Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector - [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], begin registering consumer stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a in ZK
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block broadcast_0
> Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0 not registered locally
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started reading broadcast variable 0
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading broadcast variable 0 took 5.5676E-5 s
> Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)
> java.io.IOException: PARSING_ERROR(2)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
> at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
> at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
> at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)
> java.io.IOException: PARSING_ERROR(2)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
> at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
> at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
> at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl - parentName: , name: TaskSet_0, runningTasks: 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl - parentName: , name: TaskSet_0, runningTasks: 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (1.213476 ms) StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor] handled message (1.543991 ms) StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from Actor[akka://sparkDriver/deadLetters]
> Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
> org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:606)
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
> org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:606)
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 failed 1 times; aborting job
> Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0 failed 1 times; aborting job
> Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet 0.0, whose tasks have all completed, from pool
> Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet 0.0, whose tasks have all completed, from pool
> sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl - Cancelling stage 0
> sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl - Cancelling stage 0
> Thread-84 INFO scheduler.DAGScheduler - Failed to run runJob at ReceiverTracker.scala:275
> Thread-85 INFO scheduler.DAGScheduler - Failed to run runJob at ReceiverTracker.scala:275
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - Removing running stage 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - Removing running stage 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - After removal of stage 0, remaining stages = 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler - After removal of stage 0, remaining stages = 0
> Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710002, packet:: clientPath:null serverPath:null finished:false header:: 1,1 replyHeader:: 1,25,-101 request:: '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f616374696f6e223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363536343932343737227d,v{s{31,s{'world,'anyone}}},1 response::
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org