spark : use the global config variables in executors

[hotdog <li...@163.com>]

I have a global config Object in my spark app. 

    Object Config {
     var lambda = 0.01
    }

and I will set the value of lambda according to user's input.

    Object MyApp {
       def main(args: String[]) {
         Config.lambda = args(0).toDouble
         ...
         rdd.map(_ * Config.lambda)
       }
    }

and I found that the modification does not take effect in executors. The
value of lambda is always 0.01. I guess the modification in driver's jvm
will not effect the executor's.


Do you have other solution ? 

I found a similar question in stackoverflow :

http://stackoverflow.com/questions/29685330/how-to-set-and-get-static-variables-from-spark

in @DanielL.  's answer, he gives three solutions:

1. Put the value inside a closure to be serialized to the executors to
perform a task. 

**But I wonder how to write the closure and how to serialized it to the
executors, could any one give me some code example?**


2.If the values are fixed or the configuration is available on the executor
nodes (lives inside the jar, etc), then you can have a lazy val,
guaranteeing initialization only once.

**what if I declare the lambda as a lazy val variable? the modification in
the driver will take effects in the executors? could you give me some code
example?** 


3.Create a broadcast variable with the data. I know this way, but it also
need a local Broadcast[] variable which wraps the Config Object right? for
example:

    val config = sc.broadcast(Config)

and use `config.value.lambda` in executors , right ?





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-use-the-global-config-variables-in-executors-tp22846.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: spark : use the global config variables in executors

[Marcelo Vanzin <va...@cloudera.com>]

Note that `object` is equivalent to a class full of static fields / methods
(in Java), so the data it holds will not be serialized, ever.

What you want is a config class instead, so you can instantiate it, and
that instance can be serialized. Then you can easily do (1) or (3).

On Mon, May 11, 2015 at 7:55 AM, hotdog <li...@163.com> wrote:

> I have a global config Object in my spark app.
>
>     Object Config {
>      var lambda = 0.01
>     }
>
> and I will set the value of lambda according to user's input.
>
>     Object MyApp {
>        def main(args: String[]) {
>          Config.lambda = args(0).toDouble
>          ...
>          rdd.map(_ * Config.lambda)
>        }
>     }
>
> and I found that the modification does not take effect in executors. The
> value of lambda is always 0.01. I guess the modification in driver's jvm
> will not effect the executor's.
>
>
> Do you have other solution ?
>
> I found a similar question in stackoverflow :
>
>
> http://stackoverflow.com/questions/29685330/how-to-set-and-get-static-variables-from-spark
>
> in @DanielL.  's answer, he gives three solutions:
>
> 1. Put the value inside a closure to be serialized to the executors to
> perform a task.
>
> **But I wonder how to write the closure and how to serialized it to the
> executors, could any one give me some code example?**
>
>
> 2.If the values are fixed or the configuration is available on the executor
> nodes (lives inside the jar, etc), then you can have a lazy val,
> guaranteeing initialization only once.
>
> **what if I declare the lambda as a lazy val variable? the modification in
> the driver will take effects in the executors? could you give me some code
> example?**
>
>
> 3.Create a broadcast variable with the data. I know this way, but it also
> need a local Broadcast[] variable which wraps the Config Object right? for
> example:
>
>     val config = sc.broadcast(Config)
>
> and use `config.value.lambda` in executors , right ?
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-use-the-global-config-variables-in-executors-tp22846.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
Marcelo