You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by anny9699 <an...@gmail.com> on 2014/10/15 02:28:13 UTC

Spark output to s3 extremely slow

Hi,

I found writing output back to s3 using rdd.saveAsTextFile() is extremely
slow, much slower than reading from s3. Is there a way to make it faster?
The rdd has 150 partitions so parallelism is enough I assume.

Thanks a lot!
Anny



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-output-to-s3-extremely-slow-tp16447.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark output to s3 extremely slow

Posted by Anny Chen <an...@gmail.com>.
Hi Rafal,

Thanks for the explanation and solution! I need to write maybe 100 GB to
s3. I will try your way and see whether it works for me.

Thanks again!

On Wed, Oct 15, 2014 at 1:44 AM, Rafal Kwasny <ma...@entropy.be> wrote:

> Hi,
> How large is the dataset you're saving into S3?
>
> Actually saving to S3 is done in two steps:
> 1) writing temporary files
> 2) commiting them to proper directory
> Step 2) could be slow because S3 do not have a quick atomic "move"
> operation, you have to copy (server side but still takes time) and then
> delete the original.
>
> I've overcome this but using a jobconf with NullOutputCommitter
>       jobConf.setOutputCommitter(classOf[NullOutputCommitter])
>
> Where NullOutputCommiter is a Class that doesn't do anything:
>
>   class NullOutputCommitter extends OutputCommitter {
>     def abortTask(taskContext: TaskAttemptContext) =  { }
>     override  def cleanupJob(jobContext: JobContext ) = { }
>     def commitTask(taskContext: TaskAttemptContext ) = { }
>     def needsTaskCommit(taskContext: TaskAttemptContext ) = {  false  }
>     def setupJob(jobContext: JobContext) { }
>     def setupTask(taskContext: TaskAttemptContext) { }
>   }
>
> This works but maybe someone has a better solution.
>
> /Raf
>
> anny9699 wrote:
> > Hi,
> >
> > I found writing output back to s3 using rdd.saveAsTextFile() is extremely
> > slow, much slower than reading from s3. Is there a way to make it faster?
> > The rdd has 150 partitions so parallelism is enough I assume.
> >
> > Thanks a lot!
> > Anny
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-output-to-s3-extremely-slow-tp16447.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
>

Re: Spark output to s3 extremely slow

Posted by Rafal Kwasny <ma...@entropy.be>.
Hi,
How large is the dataset you're saving into S3?

Actually saving to S3 is done in two steps:
1) writing temporary files
2) commiting them to proper directory
Step 2) could be slow because S3 do not have a quick atomic "move"
operation, you have to copy (server side but still takes time) and then
delete the original.

I've overcome this but using a jobconf with NullOutputCommitter
      jobConf.setOutputCommitter(classOf[NullOutputCommitter])

Where NullOutputCommiter is a Class that doesn't do anything:

  class NullOutputCommitter extends OutputCommitter {
    def abortTask(taskContext: TaskAttemptContext) =  { }
    override  def cleanupJob(jobContext: JobContext ) = { }
    def commitTask(taskContext: TaskAttemptContext ) = { }
    def needsTaskCommit(taskContext: TaskAttemptContext ) = {  false  }
    def setupJob(jobContext: JobContext) { }
    def setupTask(taskContext: TaskAttemptContext) { }
  }

This works but maybe someone has a better solution.

/Raf

anny9699 wrote:
> Hi,
>
> I found writing output back to s3 using rdd.saveAsTextFile() is extremely
> slow, much slower than reading from s3. Is there a way to make it faster?
> The rdd has 150 partitions so parallelism is enough I assume.
>
> Thanks a lot!
> Anny
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-output-to-s3-extremely-slow-tp16447.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org