saveAsTextFile makes no progress without caching RDD

[jerryye <je...@gmail.com>]

Hi,
I'm running on branch-1.1 and trying to do a simple transformation to a
relatively small dataset of 64GB and saveAsTextFile essentially hangs and
tasks are stuck in running mode with the following code:

// Stalls with tasks running for over an hour with no tasks finishing.
Smallest partition is 10MB
val data = sc.textFile("s3n://input")
val reformatted = data.map(t =>
t.replace("Test(","").replace(")","").replaceAll(",","\t"))
reformatted.saveAsTextFile("s3n://transformed")

// This runs but stalls doing GC after filling up 150% of 650GB of memory
val data = sc.textFile("s3n://input")
val reformatted = data.map(t =>
t.replace("Test(","").replace(")","").replaceAll(",","\t")).cache
reformatted.saveAsTextFile("s3n://transformed")

Any idea if this is a parameter issue and there is something I should try
out?

Thanks!

- jerry




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/saveAsTextFile-makes-no-progress-without-caching-RDD-tp12613.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: saveAsTextFile makes no progress without caching RDD

[jerryye <je...@gmail.com>]

As an update. I'm still getting the same issue. I ended up doing a coalesce
instead of a cache to get around the memory issue but saveAsTextFile still
won't proceed without the coalesce or cache first.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/saveAsTextFile-makes-no-progress-without-caching-RDD-tp12613p13283.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org