You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Макар Красноперов <co...@gmail.com> on 2014/09/17 16:37:49 UTC
Spark and disk usage.
Hello everyone.
The problem is that spark write data to the disk very hard, even if
application has a lot of free memory (about 3.8g).
So, I've noticed that folder with name like
"spark-local-20140917165839-f58c" contains a lot of other folders with
files like "shuffle_446_0_1". The total size of files in the dir
"spark-local-20140917165839-f58c" can reach 1.1g.
Sometimes its size decreases (are there only temp files in that folder?),
so the totally amount of data written to the disk is greater than 1.1g.
The question is what kind of data Spark store there and can I make spark
not to write it on the disk and just keep it in the memory if there is
enough RAM free space?
I run my job locally with Spark 1.0.1:
./bin/spark-submit --driver-memory 12g --master local[3] --properties-file
conf/spark-defaults.conf --class my.company.Main /path/to/jar/myJob.jar
spark-defaults.conf :
spark.shuffle.spill false
spark.reducer.maxMbInFlight 1024
spark.shuffle.file.buffer.kb 2048
spark.storage.memoryFraction 0.7
The situation with disk usage is common for many jobs. I had also used ALS
from MLIB and saw the similar things.
I had reached no success by playing with spark configuration and i hope
someone can help me :)
Re: Spark and disk usage.
Posted by Andrew Ash <an...@andrewash.com>.
Thanks for the info Burak!
I filed a bug on myself at https://issues.apache.org/jira/browse/SPARK-3631
to turn this information into a new section on the programming guide.
Thanks for the explanation it's very helpful.
Andrew
On Wed, Sep 17, 2014 at 12:08 PM, Burak Yavuz <by...@stanford.edu> wrote:
> Yes, writing to HDFS is more expensive, but I feel it is still a small
> price to pay when compared to having a Disk Space Full error three hours in
> and having to start from scratch.
>
> The main goal of checkpointing is to truncate the lineage. Clearing up
> shuffle writes come as a bonus to checkpointing, it is not the main goal.
> The
> subtlety here is that .checkpoint() is just like .cache(). Until you call
> an action, nothing happens. Therefore, if you're going to do 1000 maps in a
> row and you don't want to checkpoint in the meantime until a shuffle
> happens, you will still get a StackOverflowError, because the lineage is
> too long.
>
> I went through some of the code for checkpointing. As far as I can tell,
> it materializes the data in HDFS, and resets all its dependencies, so you
> start
> a fresh lineage. My understanding would be that checkpointing still should
> be done every N operations to reset the lineage. However, an action must be
> performed before the lineage grows too long.
>
> I believe it would be nice to write up checkpointing in the programming
> guide. The reason that it's not there yet I believe is that most
> applications don't
> grow such a long lineage, except in Spark Streaming, and some MLlib
> algorithms. If you can help with the guide, I think it would be a nice
> feature to have!
>
> Burak
>
>
> ----- Original Message -----
> From: "Andrew Ash" <an...@andrewash.com>
> To: "Burak Yavuz" <by...@stanford.edu>
> Cc: "Макар Красноперов" <co...@gmail.com>, "user" <
> user@spark.apache.org>
> Sent: Wednesday, September 17, 2014 11:04:02 AM
> Subject: Re: Spark and disk usage.
>
> Thanks for the info!
>
> Are there performance impacts with writing to HDFS instead of local disk?
> I'm assuming that's why ALS checkpoints every third iteration instead of
> every iteration.
>
> Also I can imagine that checkpointing should be done every N shuffles
> instead of every N operations (counting maps), since only the shuffle
> leaves data on disk. Do you have any suggestions on this?
>
> We should write up some guidance on the use of checkpointing in the
> programming
> guide <https://spark.apache.org/docs/latest/programming-guide.html> - I
> can
> help with this
>
> Andrew
>
>
Re: Spark and disk usage.
Posted by Burak Yavuz <by...@stanford.edu>.
Yes, writing to HDFS is more expensive, but I feel it is still a small price to pay when compared to having a Disk Space Full error three hours in
and having to start from scratch.
The main goal of checkpointing is to truncate the lineage. Clearing up shuffle writes come as a bonus to checkpointing, it is not the main goal. The
subtlety here is that .checkpoint() is just like .cache(). Until you call an action, nothing happens. Therefore, if you're going to do 1000 maps in a
row and you don't want to checkpoint in the meantime until a shuffle happens, you will still get a StackOverflowError, because the lineage is too long.
I went through some of the code for checkpointing. As far as I can tell, it materializes the data in HDFS, and resets all its dependencies, so you start
a fresh lineage. My understanding would be that checkpointing still should be done every N operations to reset the lineage. However, an action must be
performed before the lineage grows too long.
I believe it would be nice to write up checkpointing in the programming guide. The reason that it's not there yet I believe is that most applications don't
grow such a long lineage, except in Spark Streaming, and some MLlib algorithms. If you can help with the guide, I think it would be a nice feature to have!
Burak
----- Original Message -----
From: "Andrew Ash" <an...@andrewash.com>
To: "Burak Yavuz" <by...@stanford.edu>
Cc: "Макар Красноперов" <co...@gmail.com>, "user" <us...@spark.apache.org>
Sent: Wednesday, September 17, 2014 11:04:02 AM
Subject: Re: Spark and disk usage.
Thanks for the info!
Are there performance impacts with writing to HDFS instead of local disk?
I'm assuming that's why ALS checkpoints every third iteration instead of
every iteration.
Also I can imagine that checkpointing should be done every N shuffles
instead of every N operations (counting maps), since only the shuffle
leaves data on disk. Do you have any suggestions on this?
We should write up some guidance on the use of checkpointing in the programming
guide <https://spark.apache.org/docs/latest/programming-guide.html> - I can
help with this
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark and disk usage.
Posted by Andrew Ash <an...@andrewash.com>.
Thanks for the info!
Are there performance impacts with writing to HDFS instead of local disk?
I'm assuming that's why ALS checkpoints every third iteration instead of
every iteration.
Also I can imagine that checkpointing should be done every N shuffles
instead of every N operations (counting maps), since only the shuffle
leaves data on disk. Do you have any suggestions on this?
We should write up some guidance on the use of checkpointing in the programming
guide <https://spark.apache.org/docs/latest/programming-guide.html> - I can
help with this
Andrew
Re: Spark and disk usage.
Posted by Burak Yavuz <by...@stanford.edu>.
Hi Andrew,
Yes, I'm referring to sparkContext.setCheckpointDir(). It has the same effect as in Spark Streaming.
For example, in an algorithm like ALS, the RDDs go through many transformations and the lineage of the RDD starts to grow drastically just like
the lineage of DStreams do in Spark Streaming. You may observe StackOverflowErrors in ALS if you set the number of iterations to be very high.
If you set the checkpointing directory however, the intermediate state of the RDDs will be saved in HDFS, and the lineage will pick off from there.
You won't need to keep the shuffle data before the checkpointed state, therefore those can be safely removed (will be removed automatically).
However, checkpoint must be called explicitly as in https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala#L291 ,just setting the directory will not be enough.
Best,
Burak
----- Original Message -----
From: "Andrew Ash" <an...@andrewash.com>
To: "Burak Yavuz" <by...@stanford.edu>
Cc: "Макар Красноперов" <co...@gmail.com>, "user" <us...@spark.apache.org>
Sent: Wednesday, September 17, 2014 10:19:42 AM
Subject: Re: Spark and disk usage.
Hi Burak,
Most discussions of checkpointing in the docs is related to Spark
streaming. Are you talking about the sparkContext.setCheckpointDir()?
What effect does that have?
https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing
On Wed, Sep 17, 2014 at 7:44 AM, Burak Yavuz <by...@stanford.edu> wrote:
> Hi,
>
> The files you mentioned are temporary files written by Spark during
> shuffling. ALS will write a LOT of those files as it is a shuffle heavy
> algorithm.
> Those files will be deleted after your program completes as Spark looks
> for those files in case a fault occurs. Having those files ready allows
> Spark to
> continue from the stage the shuffle left off, instead of starting from the
> very beginning.
>
> Long story short, it's to your benefit that Spark writes those files to
> disk. If you don't want Spark writing to disk, you can specify a checkpoint
> directory in
> HDFS, where Spark will write the current status instead and will clean up
> files from disk.
>
> Best,
> Burak
>
> ----- Original Message -----
> From: "Макар Красноперов" <co...@gmail.com>
> To: user@spark.apache.org
> Sent: Wednesday, September 17, 2014 7:37:49 AM
> Subject: Spark and disk usage.
>
> Hello everyone.
>
> The problem is that spark write data to the disk very hard, even if
> application has a lot of free memory (about 3.8g).
> So, I've noticed that folder with name like
> "spark-local-20140917165839-f58c" contains a lot of other folders with
> files like "shuffle_446_0_1". The total size of files in the dir
> "spark-local-20140917165839-f58c" can reach 1.1g.
> Sometimes its size decreases (are there only temp files in that folder?),
> so the totally amount of data written to the disk is greater than 1.1g.
>
> The question is what kind of data Spark store there and can I make spark
> not to write it on the disk and just keep it in the memory if there is
> enough RAM free space?
>
> I run my job locally with Spark 1.0.1:
> ./bin/spark-submit --driver-memory 12g --master local[3] --properties-file
> conf/spark-defaults.conf --class my.company.Main /path/to/jar/myJob.jar
>
> spark-defaults.conf :
> spark.shuffle.spill false
> spark.reducer.maxMbInFlight 1024
> spark.shuffle.file.buffer.kb 2048
> spark.storage.memoryFraction 0.7
>
> The situation with disk usage is common for many jobs. I had also used ALS
> from MLIB and saw the similar things.
>
> I had reached no success by playing with spark configuration and i hope
> someone can help me :)
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark and disk usage.
Posted by Andrew Ash <an...@andrewash.com>.
Hi Burak,
Most discussions of checkpointing in the docs is related to Spark
streaming. Are you talking about the sparkContext.setCheckpointDir()?
What effect does that have?
https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing
On Wed, Sep 17, 2014 at 7:44 AM, Burak Yavuz <by...@stanford.edu> wrote:
> Hi,
>
> The files you mentioned are temporary files written by Spark during
> shuffling. ALS will write a LOT of those files as it is a shuffle heavy
> algorithm.
> Those files will be deleted after your program completes as Spark looks
> for those files in case a fault occurs. Having those files ready allows
> Spark to
> continue from the stage the shuffle left off, instead of starting from the
> very beginning.
>
> Long story short, it's to your benefit that Spark writes those files to
> disk. If you don't want Spark writing to disk, you can specify a checkpoint
> directory in
> HDFS, where Spark will write the current status instead and will clean up
> files from disk.
>
> Best,
> Burak
>
> ----- Original Message -----
> From: "Макар Красноперов" <co...@gmail.com>
> To: user@spark.apache.org
> Sent: Wednesday, September 17, 2014 7:37:49 AM
> Subject: Spark and disk usage.
>
> Hello everyone.
>
> The problem is that spark write data to the disk very hard, even if
> application has a lot of free memory (about 3.8g).
> So, I've noticed that folder with name like
> "spark-local-20140917165839-f58c" contains a lot of other folders with
> files like "shuffle_446_0_1". The total size of files in the dir
> "spark-local-20140917165839-f58c" can reach 1.1g.
> Sometimes its size decreases (are there only temp files in that folder?),
> so the totally amount of data written to the disk is greater than 1.1g.
>
> The question is what kind of data Spark store there and can I make spark
> not to write it on the disk and just keep it in the memory if there is
> enough RAM free space?
>
> I run my job locally with Spark 1.0.1:
> ./bin/spark-submit --driver-memory 12g --master local[3] --properties-file
> conf/spark-defaults.conf --class my.company.Main /path/to/jar/myJob.jar
>
> spark-defaults.conf :
> spark.shuffle.spill false
> spark.reducer.maxMbInFlight 1024
> spark.shuffle.file.buffer.kb 2048
> spark.storage.memoryFraction 0.7
>
> The situation with disk usage is common for many jobs. I had also used ALS
> from MLIB and saw the similar things.
>
> I had reached no success by playing with spark configuration and i hope
> someone can help me :)
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark and disk usage.
Posted by Burak Yavuz <by...@stanford.edu>.
Hi,
The files you mentioned are temporary files written by Spark during shuffling. ALS will write a LOT of those files as it is a shuffle heavy algorithm.
Those files will be deleted after your program completes as Spark looks for those files in case a fault occurs. Having those files ready allows Spark to
continue from the stage the shuffle left off, instead of starting from the very beginning.
Long story short, it's to your benefit that Spark writes those files to disk. If you don't want Spark writing to disk, you can specify a checkpoint directory in
HDFS, where Spark will write the current status instead and will clean up files from disk.
Best,
Burak
----- Original Message -----
From: "Макар Красноперов" <co...@gmail.com>
To: user@spark.apache.org
Sent: Wednesday, September 17, 2014 7:37:49 AM
Subject: Spark and disk usage.
Hello everyone.
The problem is that spark write data to the disk very hard, even if
application has a lot of free memory (about 3.8g).
So, I've noticed that folder with name like
"spark-local-20140917165839-f58c" contains a lot of other folders with
files like "shuffle_446_0_1". The total size of files in the dir
"spark-local-20140917165839-f58c" can reach 1.1g.
Sometimes its size decreases (are there only temp files in that folder?),
so the totally amount of data written to the disk is greater than 1.1g.
The question is what kind of data Spark store there and can I make spark
not to write it on the disk and just keep it in the memory if there is
enough RAM free space?
I run my job locally with Spark 1.0.1:
./bin/spark-submit --driver-memory 12g --master local[3] --properties-file
conf/spark-defaults.conf --class my.company.Main /path/to/jar/myJob.jar
spark-defaults.conf :
spark.shuffle.spill false
spark.reducer.maxMbInFlight 1024
spark.shuffle.file.buffer.kb 2048
spark.storage.memoryFraction 0.7
The situation with disk usage is common for many jobs. I had also used ALS
from MLIB and saw the similar things.
I had reached no success by playing with spark configuration and i hope
someone can help me :)
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org