You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by vantoniuk <vi...@jetlore.com> on 2015/01/21 23:35:07 UTC
Re: Exception in connection from worker to worker
I have temporary fix for my case. My sample file was 2G / 50M lines in size.
My initial configuration was 1000 splits.
Based on my understanding of distributed algorithms, number of splits can
affect the memory pressure in operations such as distinct and reduceByKey.
So i tried to reduce the number of splits from 1000 to 100. Now I can run
distinct and reduceByKey on files that are 2G / 50M lines.
Unfortunately it still doesn't scale well.
Thanks.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Exception-in-connection-from-worker-to-worker-tp20983p21302.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Exception in connection from worker to worker
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Can you try the following:
- Use Kryo Serializer
- Enable RDD Compression
- Repartition the data (Use hash partition, then all the similar keys will
go in the same partition)
Thanks
Best Regards
On Thu, Jan 22, 2015 at 4:05 AM, vantoniuk <vi...@jetlore.com> wrote:
> I have temporary fix for my case. My sample file was 2G / 50M lines in
> size.
> My initial configuration was 1000 splits.
>
> Based on my understanding of distributed algorithms, number of splits can
> affect the memory pressure in operations such as distinct and reduceByKey.
> So i tried to reduce the number of splits from 1000 to 100. Now I can run
> distinct and reduceByKey on files that are 2G / 50M lines.
>
> Unfortunately it still doesn't scale well.
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Exception-in-connection-from-worker-to-worker-tp20983p21302.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>