You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jia Li (JIRA)" <ji...@apache.org> on 2015/10/11 12:00:07 UTC
[jira] [Commented] (SPARK-11051) NullPointerException when action
called on localCheckpointed RDD (that was checkpointed before)
[ https://issues.apache.org/jira/browse/SPARK-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952242#comment-14952242 ]
Jia Li commented on SPARK-11051:
--------------------------------
I would like to work on this issue.
> NullPointerException when action called on localCheckpointed RDD (that was checkpointed before)
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-11051
> URL: https://issues.apache.org/jira/browse/SPARK-11051
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.6.0
> Environment: Spark version 1.6.0-SNAPSHOT built from the sources as of today - Oct, 10th
> Reporter: Jacek Laskowski
>
> While toying with {{RDD.checkpoint}} and {{RDD.localCheckpoint}} methods, the following NullPointerException was thrown:
> {code}
> scala> lines.count
> java.lang.NullPointerException
> at org.apache.spark.rdd.RDD.firstParent(RDD.scala:1587)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1927)
> at org.apache.spark.rdd.RDD.count(RDD.scala:1115)
> ... 48 elided
> {code}
> To reproduce the issue do the following:
> {code}
> $ ./bin/spark-shell
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 1.6.0-SNAPSHOT
> /_/
> Using Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> val lines = sc.textFile("README.md")
> lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:24
> scala> sc.setCheckpointDir("checkpoints")
> scala> lines.checkpoint
> scala> lines.count
> res2: Long = 98
> scala> lines.localCheckpoint
> 15/10/10 22:59:20 WARN MapPartitionsRDD: RDD was already marked for reliable checkpointing: overriding with local checkpoint.
> res4: lines.type = MapPartitionsRDD[1] at textFile at <console>:24
> scala> lines.count
> java.lang.NullPointerException
> at org.apache.spark.rdd.RDD.firstParent(RDD.scala:1587)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1927)
> at org.apache.spark.rdd.RDD.count(RDD.scala:1115)
> ... 48 elided
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org