Streaming | Partition count mismatch exception while saving data in RDD

[Aniket Bhatnagar <an...@gmail.com>]

I am using spark 1.1.0 running a streaming job that uses updateStateByKey
and then (after a bunch of maps/flatMaps) does a foreachRDD to save data in
each RDD by making HTTP calls. The issue is that each time I attempt to
save the RDD (using foreach on RDD), it gives me the following exception:

org.apache.spark.SparkException: Checkpoint RDD CheckpointRDD[467] at apply
at List.scala:318(0) has different number of partitions than original RDD
MapPartitionsRDD[461] at mapPartitions at StateDStream.scala:71(56)

I read through the code but couldn't understand why this exception is
happening. Any help would be appreciated!

Thanks,
Aniket

Re: Streaming | Partition count mismatch exception while saving data in RDD

[Aniket Bhatnagar <an...@gmail.com>]

It turns out that this happens when checkpoint is set to a local directory
path. I have opened a JIRA SPARK-4862 for Spark streaming to output better
error message.

Thanks,
Aniket

On Tue Dec 16 2014 at 20:08:13 Aniket Bhatnagar <an...@gmail.com>
wrote:

> I am using spark 1.1.0 running a streaming job that uses updateStateByKey
> and then (after a bunch of maps/flatMaps) does a foreachRDD to save data in
> each RDD by making HTTP calls. The issue is that each time I attempt to
> save the RDD (using foreach on RDD), it gives me the following exception:
>
> org.apache.spark.SparkException: Checkpoint RDD CheckpointRDD[467] at
> apply at List.scala:318(0) has different number of partitions than original
> RDD MapPartitionsRDD[461] at mapPartitions at StateDStream.scala:71(56)
>
> I read through the code but couldn't understand why this exception is
> happening. Any help would be appreciated!
>
> Thanks,
> Aniket
>