You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by luca_guerra <lg...@bitbang.com> on 2016/04/15 17:15:56 UTC

How many disks for spark_local_dirs?

Hi,
I'm looking for a solution to improve my Spark cluster performances, I have
read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
"We recommend having 4-8 disks per node", I have tried both with one and two
disks but I have seen that with 2 disks the execution time is doubled. Any
explanations about this?

This is my configuration:
1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
configuration) and on this I have a standalone Spark cluster with 1 Worker.

Thank you very much for the help.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: How many disks for spark_local_dirs?

Posted by Mich Talebzadeh <mi...@gmail.com>.
.. to force spills to disk ...

That is pretty smart as at some point you are inevitable going to run out
of memory.

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 18 April 2016 at 08:17, Luca Guerra <lg...@bitbang.com> wrote:

> Hi Mich,
>
> I have only 32 cores, I have tested with 2 GB of memory per worker to
> force spills to disk. My application had 12 cores and 3 cores per executor.
>
>
>
> Thank you very much.
>
> Luca
>
>
>
> *Da:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Inviato:* venerdì 15 aprile 2016 18:56
> *A:* Luca Guerra <lg...@bitbang.com>
> *Cc:* user @spark <us...@spark.apache.org>
> *Oggetto:* Re: How many disks for spark_local_dirs?
>
>
>
> Is that 32 CPUs or 32 cores?
>
>
>
> So in this configuration assuming 32 cores you have I worker with how much
> memory (deducting memory for OS etc) and 32 cores.
>
>
>
> What is the ratio  of memory per core in this case?
>
>
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn  *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
>
> On 15 April 2016 at 16:15, luca_guerra <lg...@bitbang.com> wrote:
>
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I have
> read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one and
> two
> disks but I have seen that with 2 disks the execution time is doubled. Any
> explanations about this?
>
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
>
> Thank you very much for the help.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>

R: How many disks for spark_local_dirs?

Posted by Luca Guerra <lg...@bitbang.com>.
Hi Mich,
I have only 32 cores, I have tested with 2 GB of memory per worker to force spills to disk. My application had 12 cores and 3 cores per executor.

Thank you very much.
Luca

Da: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
Inviato: venerdì 15 aprile 2016 18:56
A: Luca Guerra <lg...@bitbang.com>
Cc: user @spark <us...@spark.apache.org>
Oggetto: Re: How many disks for spark_local_dirs?

Is that 32 CPUs or 32 cores?

So in this configuration assuming 32 cores you have I worker with how much memory (deducting memory for OS etc) and 32 cores.

What is the ratio  of memory per core in this case?

HTH


Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>



On 15 April 2016 at 16:15, luca_guerra <lg...@bitbang.com>> wrote:
Hi,
I'm looking for a solution to improve my Spark cluster performances, I have
read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
"We recommend having 4-8 disks per node", I have tried both with one and two
disks but I have seen that with 2 disks the execution time is doubled. Any
explanations about this?

This is my configuration:
1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
configuration) and on this I have a standalone Spark cluster with 1 Worker.

Thank you very much for the help.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>


Re: How many disks for spark_local_dirs?

Posted by Mich Talebzadeh <mi...@gmail.com>.
Is that 32 CPUs or 32 cores?

So in this configuration assuming 32 cores you have I worker with how much
memory (deducting memory for OS etc) and 32 cores.

What is the ratio  of memory per core in this case?

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 15 April 2016 at 16:15, luca_guerra <lg...@bitbang.com> wrote:

> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I have
> read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one and
> two
> disks but I have seen that with 2 disks the execution time is doubled. Any
> explanations about this?
>
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
>
> Thank you very much for the help.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

R: How many disks for spark_local_dirs?

Posted by Luca Guerra <lg...@bitbang.com>.
Hi Jan,
It's a physical server, I have launched the application with:
	- "spark.cores.max": "12",
	- "spark.executor.cores": "3"
	- 2 GB RAM per worker
Spark version is 1.6.0, I don't use Hadoop.

Thanks,
Luca

-----Messaggio originale-----
Da: Jan Rock [mailto:rock@do-hadoop.com] 
Inviato: venerdì 15 aprile 2016 18:04
A: Luca Guerra <lg...@bitbang.com>
Cc: user@spark.apache.org
Oggetto: Re: How many disks for spark_local_dirs?

Hi, 

is it physical server or AWS/Azure? What are the executed parameters for spark-shell command? Hadoop distro/version and Spark version?

Kind Regards,
Jan


> On 15 Apr 2016, at 16:15, luca_guerra <lg...@bitbang.com> wrote:
> 
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I 
> have read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one 
> and two disks but I have seen that with 2 disks the execution time is 
> doubled. Any explanations about this?
> 
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an 
> unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
> 
> Thank you very much for the help.
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for
> -spark-local-dirs-tp26790.html Sent from the Apache Spark User List 
> mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For 
> additional commands, e-mail: user-help@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: How many disks for spark_local_dirs?

Posted by Jan Rock <ro...@do-hadoop.com>.
Hi, 

is it physical server or AWS/Azure? What are the executed parameters for spark-shell command? Hadoop distro/version and Spark version?

Kind Regards,
Jan


> On 15 Apr 2016, at 16:15, luca_guerra <lg...@bitbang.com> wrote:
> 
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I have
> read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one and two
> disks but I have seen that with 2 disks the execution time is doubled. Any
> explanations about this?
> 
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
> 
> Thank you very much for the help.
> 
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org