You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Bode, Meikel, NMA-CFD" <Me...@Bertelsmann.de> on 2021/05/03 12:54:02 UTC

Broadcast Variable

Hi all,

when broadcasting a large dict containing several million entries to executors what exactly happens when calling bc_var.value within a UDF like:

..
d = bc_var.value
..

Does d receives a copy of the dict inside value or is this handled like a pointer?

Thanks,
Meikel

Re: Broadcast Variable

Posted by Sean Owen <sr...@gmail.com>.

There is just one copy in memory. No different than if you have to
variables pointing to the same dict.

On Mon, May 3, 2021 at 7:54 AM Bode, Meikel, NMA-CFD <
Meikel.Bode@bertelsmann.de> wrote:

> Hi all,
>
>
>
> when broadcasting a large dict containing several million entries to
> executors what exactly happens when calling bc_var.value within a UDF like:
>
>
>
> ..
>
> d = bc_var.value
>
> ..
>
>
>
> Does d receives a copy of the dict inside value or is this handled like a
> pointer?
>
>
>
> Thanks,
> Meikel
>