You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by klrmowse <kl...@gmail.com> on 2018/08/05 14:51:51 UTC
Broadcast variable size limit?
is it currently still ~2GB (Integer.MAX_VALUE) ??
or am i misinformed, since that's what google-search and scouring this
mailing list seem to say... ?
Thanks
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Broadcast variable size limit?
Posted by Vadim Semenov <va...@datadoghq.com>.
That’s the max size of a byte array in Java, limited by the length which is
defined as integer, and in most JVMS arrays can’t hold more than
Int.MaxValue - 8 elements. Other way to overcome this is to create multiple
broadcast variables
On Sunday, August 5, 2018, klrmowse <kl...@gmail.com> wrote:
> i don't need more, per se... i just need to watch the size of the variable;
> then, if it's within the size limit, go ahead and broadcast it; if not,
> then
> i won't broadcast...
>
> so, that would be a yes then? (2 GB, or which is it exactly?)
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
--
Sent from my iPhone
Re: Broadcast variable size limit?
Posted by klrmowse <kl...@gmail.com>.
i don't need more, per se... i just need to watch the size of the variable;
then, if it's within the size limit, go ahead and broadcast it; if not, then
i won't broadcast...
so, that would be a yes then? (2 GB, or which is it exactly?)
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Broadcast variable size limit?
Posted by Jörn Franke <jo...@gmail.com>.
I think if you need more then you should anyway think about something different than broadcast variable ...
> On 5. Aug 2018, at 16:51, klrmowse <kl...@gmail.com> wrote:
>
> is it currently still ~2GB (Integer.MAX_VALUE) ??
>
> or am i misinformed, since that's what google-search and scouring this
> mailing list seem to say... ?
>
>
>
> Thanks
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org