You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ankur Srivastava <an...@gmail.com> on 2016/10/12 06:16:10 UTC

Spark Shuffle Issue

Hi,

I am upgrading my jobs to Spark 1.6 and am running into shuffle issues. I
have tried all options and now am falling back to legacy memory model but
still running into same issue.

I have set spark.shuffle.blockTransferService to nio.

16/10/12 06:00:10 INFO MapOutputTrackerMaster: Size of output statuses for
shuffle 6 is 57910504 bytes

16/10/12 06:00:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map
output locations for shuffle 6 to dedwfprshd023.de.neustar.com:52510

16/10/12 06:00:10 ERROR TransportRequestHandler: Error sending result
RpcResponse{requestId=7357168459686146036,
body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=57910531
cap=57910531]}} to <hostname>; closing connection


Any other settings that I should tune?


Thanks

Ankur

Re: Spark Shuffle Issue

Posted by Ankur Srivastava <an...@gmail.com>.
Hi,

I was able to resolve the issue with increasing the timeout and reducing
the number of executors and increasing number of cores per executor.

The issue is resolved but I am still not sure why reducing the number of
executors and increasing number of cores per executor fixed issues related
to shuffle reads. I understand that now with more cores per executor the
executor threads are getting less memory but earlier too there were no GC
issues.

Can some one help me understand how this setting could help and what are
the things we should consider for selecting the right ratio of executors to
cores?

Thanks
Ankur

On Tue, Oct 11, 2016 at 11:16 PM, Ankur Srivastava <
ankur.srivastava@gmail.com> wrote:

> Hi,
>
> I am upgrading my jobs to Spark 1.6 and am running into shuffle issues. I
> have tried all options and now am falling back to legacy memory model but
> still running into same issue.
>
> I have set spark.shuffle.blockTransferService to nio.
>
> 16/10/12 06:00:10 INFO MapOutputTrackerMaster: Size of output statuses for
> shuffle 6 is 57910504 bytes
>
> 16/10/12 06:00:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map
> output locations for shuffle 6 to dedwfprshd023.de.neustar.com:52510
>
> 16/10/12 06:00:10 ERROR TransportRequestHandler: Error sending result
> RpcResponse{requestId=7357168459686146036, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0
> lim=57910531 cap=57910531]}} to <hostname>; closing connection
>
>
> Any other settings that I should tune?
>
>
> Thanks
>
> Ankur
>