You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Guillaume Pitel <gu...@exensa.com> on 2014/02/20 16:24:40 UTC
Problem with akka.frameSize
Hi,
I've sent a few emails to the user mailing list, but since I believe this is
bug, I think it's time to talk to the developpers.
So here is what happens: since we've migrated from 0.8.1 to 0.9, whatever the
value of spark.akka.frameSize I set, the Executors lock when a collect tries to
send more than 10MB of data to the driver.
Since 10MB is the spark default, I suspect it could be related to something in
the configuration. We still use System.setProperty to set the frameSize.
As a workaround, setting the frameSize back to 10 seems to work.
Guillaume
--
eXenSa
*Guillaume PITEL, Président*
+33(0)6 25 48 86 80
eXenSa S.A.S. <http://www.exensa.com/>
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05
Re: Problem with akka.frameSize
Posted by Patrick Wendell <pw...@gmail.com>.
Thanks for this bug report... we'll look into this!
On Thu, Feb 20, 2014 at 8:39 AM, Guillaume Pitel <guillaume.pitel@exensa.com
> wrote:
> Jira ticket created https://spark-project.atlassian.net/browse/SPARK-1112
>
> Guillaume
>
> Hi,
>
> I've sent a few emails to the user mailing list, but since I believe this
> is bug, I think it's time to talk to the developpers.
>
> So here is what happens: since we've migrated from 0.8.1 to 0.9, whatever
> the value of spark.akka.frameSize I set, the Executors lock when a collect
> tries to send more than 10MB of data to the driver.
>
> Since 10MB is the spark default, I suspect it could be related to
> something in the configuration. We still use System.setProperty to set the
> frameSize.
>
> As a workaround, setting the frameSize back to 10 seems to work.
>
> Guillaume
>
>
> --
> [image: eXenSa]
> *Guillaume PITEL, Président*
> +33(0)6 25 48 86 80
>
> eXenSa S.A.S. <http://www.exensa.com/>
> 41, rue Périer - 92120 Montrouge - FRANCE
> Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05
>
Re: Problem with akka.frameSize
Posted by Guillaume Pitel <gu...@exensa.com>.
Jira ticket created https://spark-project.atlassian.net/browse/SPARK-1112
Guillaume
> Hi,
>
> I've sent a few emails to the user mailing list, but since I believe this is
> bug, I think it's time to talk to the developpers.
>
> So here is what happens: since we've migrated from 0.8.1 to 0.9, whatever the
> value of spark.akka.frameSize I set, the Executors lock when a collect tries
> to send more than 10MB of data to the driver.
>
> Since 10MB is the spark default, I suspect it could be related to something in
> the configuration. We still use System.setProperty to set the frameSize.
>
> As a workaround, setting the frameSize back to 10 seems to work.
>
> Guillaume
--
eXenSa
*Guillaume PITEL, Président*
+33(0)6 25 48 86 80
eXenSa S.A.S. <http://www.exensa.com/>
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05