You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ashic Mahtab <as...@live.com> on 2014/12/19 12:21:42 UTC

Scala Lazy values and partitions

Hi Guys,
Are scala lazy values instantiated once per executor, or once per partition? For example, if I have:

object Something = 
    val lazy context = create()

    def foo(item) = context.doSomething(item)

and I do

someRdd.foreach(Something.foo)

then will context get instantiated once per executor, or once per partition?

Thanks,
Ashic.

Re: Scala Lazy values and partitions

Posted by Sean Owen <so...@cloudera.com>.
A val in an object should be instantiated once per JVM (really,
ClassLoader, but probably won't make a difference here). Therefore I
expect it is going to live effectively as long as the executor, across
partitions but also across jobs.

On Fri, Dec 19, 2014 at 11:21 AM, Ashic Mahtab <as...@live.com> wrote:
> Hi Guys,
> Are scala lazy values instantiated once per executor, or once per partition?
> For example, if I have:
>
> object Something =
>     val lazy context = create()
>
>     def foo(item) = context.doSomething(item)
>
> and I do
>
> someRdd.foreach(Something.foo)
>
> then will context get instantiated once per executor, or once per partition?
>
> Thanks,
> Ashic.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

RE: Scala Lazy values and partitions

Posted by Ashic Mahtab <as...@live.com>.
Just to confirm, once per VM means that it'll be the same instance across all applications in a particular JVM instance (i.e. executor). So even if the spark application is terminated, the instance will live on, correct? I think that's what Sean said, and it seems logical.

From: gerard.maas@gmail.com
Date: Fri, 19 Dec 2014 12:52:23 +0100
Subject: Re: Scala Lazy values and partitions
To: ashic@live.com
CC: user@spark.apache.org

It will be instantiated once per VM, which translates to once per executor.
-kr, Gerard.
On Fri, Dec 19, 2014 at 12:21 PM, Ashic Mahtab <as...@live.com> wrote:


Hi Guys,
Are scala lazy values instantiated once per executor, or once per partition? For example, if I have:

object Something = 
    val lazy context = create()

    def foo(item) = context.doSomething(item)

and I do

someRdd.foreach(Something.foo)

then will context get instantiated once per executor, or once per partition?

Thanks,
Ashic.

Re: Scala Lazy values and partitions

Posted by Gerard Maas <ge...@gmail.com>.
It will be instantiated once per VM, which translates to once per executor.

-kr, Gerard.

On Fri, Dec 19, 2014 at 12:21 PM, Ashic Mahtab <as...@live.com> wrote:
>
> Hi Guys,
> Are scala lazy values instantiated once per executor, or once per
> partition? For example, if I have:
>
> object Something =
>     val lazy context = create()
>
>     def foo(item) = context.doSomething(item)
>
> and I do
>
> someRdd.foreach(Something.foo)
>
> then will context get instantiated once per executor, or once per
> partition?
>
> Thanks,
> Ashic.
>
>