Recommended ways to pass functions

[Kevin Jung <it...@samsung.com>]

Hi all,
I read 'Passing Functions to Spark' section in Spark Programming Guide.
It recommends make function Anonymous or static to avoid transfer whole
class instance.
So, I wonder if I can pass functions like this.

case1: define func2 inside func1
class{
  def func1() = {
    def func2() = {...}
    rdd.map(x=> func2())
  }
}

case2: define inner object
class{
  def func1() = {
    rdd.map(x=>MyFunc.func2())
  }

  object MyFunc{
    def func2() = {...}
  }
}

Thanks in advance.

Kevin



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Recommended-ways-to-pass-functions-tp14875.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Recommended ways to pass functions

[Yanbo Liang <ya...@gmail.com>]

All these two kinds of function is OK but you need to make your class
extends Serializable.
But all these kinds of pass functions can not save data which will be send.

If you define a function which will not use member parameter of a class or
object, you can use val like definition method.
For example:
class T1 {
val func0: RDD[String] => RDD[String] = (x:RDD[String]) => x.map(y => y +
"123")
}

class T2 {
def func1(rdd: RDD[String]):RDD[String] = {
    def func2(args:String):String = {
      args + " 123"
    }
    rdd.map(func2)
  }
}

If you call func0 is more efficient method than func1 because that func0
only send the function code and func1 will send the whole class.

2014-09-23 15:29 GMT+08:00 Kevin Jung <it...@samsung.com>:

> Hi all,
> I read 'Passing Functions to Spark' section in Spark Programming Guide.
> It recommends make function Anonymous or static to avoid transfer whole
> class instance.
> So, I wonder if I can pass functions like this.
>
> case1: define func2 inside func1
> class{
>   def func1() = {
>     def func2() = {...}
>     rdd.map(x=> func2())
>   }
> }
>
> case2: define inner object
> class{
>   def func1() = {
>     rdd.map(x=>MyFunc.func2())
>   }
>
>   object MyFunc{
>     def func2() = {...}
>   }
> }
>
> Thanks in advance.
>
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Recommended-ways-to-pass-functions-tp14875.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>