You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kevin Jung <it...@samsung.com> on 2014/10/20 06:38:35 UTC

default parallelism bug?

Hi,
I usually use file on hdfs to make PairRDD and analyze it by using
combineByKey,reduceByKey, etc.
But sometimes it hangs when I set spark.default.parallelism configuration,
though the size of file is small.
If I remove this configuration, all works fine.
Does anyone tell me why this occur?

Regards,
Kevin



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/default-parallelism-bug-tp16787.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: default parallelism bug?

Posted by Olivier Girardot <ss...@gmail.com>.
Hi,
what do you mean by pretty small ? How big is your file ?

Regards,

Olivier.

2014-10-21 6:01 GMT+02:00 Kevin Jung <it...@samsung.com>:

> I use Spark 1.1.0 and set these options to spark-defaults.conf
> spark.scheduler.mode FAIR
> spark.cores.max 48
> spark.default.parallelism 72
>
> Thanks,
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/default-parallelism-bug-tp16787p16894.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: default parallelism bug?

Posted by Kevin Jung <it...@samsung.com>.
I use Spark 1.1.0 and set these options to spark-defaults.conf
spark.scheduler.mode FAIR
spark.cores.max 48
spark.default.parallelism 72

Thanks,
Kevin



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/default-parallelism-bug-tp16787p16894.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: default parallelism bug?

Posted by Yi Tian <ti...@gmail.com>.
Could you show your spark version ?

And the value of `spark.default.parallelism` you are setting?

Best Regards,

Yi Tian
tianyi.asiainfo@gmail.com




On Oct 20, 2014, at 12:38, Kevin Jung <it...@samsung.com> wrote:

> Hi,
> I usually use file on hdfs to make PairRDD and analyze it by using
> combineByKey,reduceByKey, etc.
> But sometimes it hangs when I set spark.default.parallelism configuration,
> though the size of file is small.
> If I remove this configuration, all works fine.
> Does anyone tell me why this occur?
> 
> Regards,
> Kevin
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/default-parallelism-bug-tp16787.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org