Out of scope RDDs not getting cleaned up

[jainbhavya53 <ja...@gmail.com>]

Hi,

I am using spark 2.1 and I am leveraging spark streaming for my data
pipeline. Now, in my case the batch size is 3 minutes and we persist couple
of RDDs while processing a batch and after processing we rely on Spark's
ContextCleaner to clean out RDDs which are no longer in scope.

So we have set "spark.cleaner.periodicGC.interval" = "15s" and
"spark.network.timeout" = "20s".

Now sometimes, when GC is triggered and it tries to clean all the out of
scope RDDs, then a Future timeout occurs(File attached -->
FuturesTimeout.txt) and it says "Failed to remove RDD 14254".

FuturesTImeoutException.txt
<http://apache-spark-user-list.1001560.n3.nabble.com/file/t10974/FuturesTImeoutException.txt>  

So, when I go and check for this particular RDD under storage tab, then I
can see the same RDD is still there(verified using RddId -> 14254).

That's fine unless it is queued for cleanup in subsequent GC cycle !
But that does not happen and I could see this RDD under the storage tab.
This happened for couple of more RDDs.

So, I tried looking into it and it seems once ContextCleaner sends a request
for cleaning the RDD and then if an error occurs while cleaning this RDD
then it does not re-queue the given RDD for cleaning.

It seems like a bug to me. Can you please look into it !

Thanks
Bhavya Jain



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org