You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by luca_guerra <lg...@bitbang.com> on 2016/04/15 17:15:56 UTC
How many disks for spark_local_dirs?
Hi,
I'm looking for a solution to improve my Spark cluster performances, I have
read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
"We recommend having 4-8 disks per node", I have tried both with one and two
disks but I have seen that with 2 disks the execution time is doubled. Any
explanations about this?
This is my configuration:
1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
configuration) and on this I have a standalone Spark cluster with 1 Worker.
Thank you very much for the help.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How many disks for spark_local_dirs?
Posted by Mich Talebzadeh <mi...@gmail.com>.
.. to force spills to disk ...
That is pretty smart as at some point you are inevitable going to run out
of memory.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 18 April 2016 at 08:17, Luca Guerra <lg...@bitbang.com> wrote:
> Hi Mich,
>
> I have only 32 cores, I have tested with 2 GB of memory per worker to
> force spills to disk. My application had 12 cores and 3 cores per executor.
>
>
>
> Thank you very much.
>
> Luca
>
>
>
> *Da:* Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
> *Inviato:* venerdì 15 aprile 2016 18:56
> *A:* Luca Guerra <lg...@bitbang.com>
> *Cc:* user @spark <us...@spark.apache.org>
> *Oggetto:* Re: How many disks for spark_local_dirs?
>
>
>
> Is that 32 CPUs or 32 cores?
>
>
>
> So in this configuration assuming 32 cores you have I worker with how much
> memory (deducting memory for OS etc) and 32 cores.
>
>
>
> What is the ratio of memory per core in this case?
>
>
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
>
> On 15 April 2016 at 16:15, luca_guerra <lg...@bitbang.com> wrote:
>
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I have
> read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one and
> two
> disks but I have seen that with 2 disks the execution time is doubled. Any
> explanations about this?
>
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
>
> Thank you very much for the help.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
R: How many disks for spark_local_dirs?
Posted by Luca Guerra <lg...@bitbang.com>.
Hi Mich,
I have only 32 cores, I have tested with 2 GB of memory per worker to force spills to disk. My application had 12 cores and 3 cores per executor.
Thank you very much.
Luca
Da: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
Inviato: venerdì 15 aprile 2016 18:56
A: Luca Guerra <lg...@bitbang.com>
Cc: user @spark <us...@spark.apache.org>
Oggetto: Re: How many disks for spark_local_dirs?
Is that 32 CPUs or 32 cores?
So in this configuration assuming 32 cores you have I worker with how much memory (deducting memory for OS etc) and 32 cores.
What is the ratio of memory per core in this case?
HTH
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>
On 15 April 2016 at 16:15, luca_guerra <lg...@bitbang.com>> wrote:
Hi,
I'm looking for a solution to improve my Spark cluster performances, I have
read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
"We recommend having 4-8 disks per node", I have tried both with one and two
disks but I have seen that with 2 disks the execution time is doubled. Any
explanations about this?
This is my configuration:
1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
configuration) and on this I have a standalone Spark cluster with 1 Worker.
Thank you very much for the help.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
Re: How many disks for spark_local_dirs?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Is that 32 CPUs or 32 cores?
So in this configuration assuming 32 cores you have I worker with how much
memory (deducting memory for OS etc) and 32 cores.
What is the ratio of memory per core in this case?
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 15 April 2016 at 16:15, luca_guerra <lg...@bitbang.com> wrote:
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I have
> read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one and
> two
> disks but I have seen that with 2 disks the execution time is doubled. Any
> explanations about this?
>
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
>
> Thank you very much for the help.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
R: How many disks for spark_local_dirs?
Posted by Luca Guerra <lg...@bitbang.com>.
Hi Jan,
It's a physical server, I have launched the application with:
- "spark.cores.max": "12",
- "spark.executor.cores": "3"
- 2 GB RAM per worker
Spark version is 1.6.0, I don't use Hadoop.
Thanks,
Luca
-----Messaggio originale-----
Da: Jan Rock [mailto:rock@do-hadoop.com]
Inviato: venerdì 15 aprile 2016 18:04
A: Luca Guerra <lg...@bitbang.com>
Cc: user@spark.apache.org
Oggetto: Re: How many disks for spark_local_dirs?
Hi,
is it physical server or AWS/Azure? What are the executed parameters for spark-shell command? Hadoop distro/version and Spark version?
Kind Regards,
Jan
> On 15 Apr 2016, at 16:15, luca_guerra <lg...@bitbang.com> wrote:
>
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I
> have read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one
> and two disks but I have seen that with 2 disks the execution time is
> doubled. Any explanations about this?
>
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an
> unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
>
> Thank you very much for the help.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for
> -spark-local-dirs-tp26790.html Sent from the Apache Spark User List
> mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For
> additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How many disks for spark_local_dirs?
Posted by Jan Rock <ro...@do-hadoop.com>.
Hi,
is it physical server or AWS/Azure? What are the executed parameters for spark-shell command? Hadoop distro/version and Spark version?
Kind Regards,
Jan
> On 15 Apr 2016, at 16:15, luca_guerra <lg...@bitbang.com> wrote:
>
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I have
> read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one and two
> disks but I have seen that with 2 disks the execution time is doubled. Any
> explanations about this?
>
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
>
> Thank you very much for the help.
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org