[jira] [Created] (SPARK-27598) DStreams checkpointing does not work with Scala 2.12

["Stavros Kontopoulos (JIRA)" <ji...@apache.org>]

Stavros Kontopoulos created SPARK-27598:
-------------------------------------------

             Summary: DStreams checkpointing does not work with Scala 2.12
                 Key: SPARK-27598
                 URL: https://issues.apache.org/jira/browse/SPARK-27598
             Project: Spark
          Issue Type: Bug
          Components: DStreams
    Affects Versions: 2.4.2, 2.4.1, 2.4.0, 3.0.0
            Reporter: Stavros Kontopoulos


When I restarted a stream with checkpointing enabled I got this:
{quote}19/04/29 22:45:06 WARN CheckpointReader: Error reading checkpoint from file file:/tmp/checkpoint/checkpoint-1556566950000.bk
java.io.IOException: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.streaming.dstream.FileInputDStream.filter of type scala.Function1 in instance of org.apache.spark.streaming.dstream.FileInputDStream
 at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1322)
 at org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:314)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{quote}
It seems that the closure is stored in the Serialized format and cannot be assigned back to a scala function1

Details of how to reproduce it here:

https://gist.github.com/skonto/87d5b2368b0bf7786d9dd992a710e4e6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org