broadcasting object issue

[Henry Hung <YT...@winbond.com>]

Hi All,

I have a problem with broadcasting a serialize class object that returned by another not-serialize class, here is the sample code:

class A extends java.io.Serializable {
    def halo(): String = "halo"
}

class B {
    def getA() = new A
}

val list = List(1)

val b = new B
val a = b.getA

val p = sc.parallelize(list)

// this will fail
val bcA = sc.broadcast(a)
p.map(x => {
    bcA.value.halo()
})

// this will success
val bcA = sc.broadcast(new A)
p.map(x => {
    bcA.value.halo()
})


A is a serializable class, where B is not-serialize.
If I create a new object A through B method getA(), the map process will failed with exception "org.apache.spark.SparkException: Task not serializable, Caused by: java.io.NotSerializableException: $iwC$$iwC$B"

I don't know why spark will check if the B class serializable or not, is there a way to code this?

Best regards,
Henry

________________________________
The privileged confidential information contained in this email is intended for use only by the addressees as indicated by the original sender of this email. If you are not the addressee indicated in this email or are not responsible for delivery of the email to such a person, please kindly reply to the sender indicating this fact and delete all copies of it from your computer and network server immediately. Your cooperation is highly appreciated. It is advised that any unauthorized use of confidential information of Winbond is strictly prohibited; and any information in this email irrelevant to the official business of Winbond shall be deemed as neither given nor endorsed by Winbond.

Re: broadcasting object issue

[madhu phatak <ph...@gmail.com>]

Hi,
 Just ran your code on spark-shell.  If you replace

 val bcA = sc.broadcast(a)


with

val bcA = sc.broadcast(new B().getA)


it seems to work. Not sure why.


On Tue, Dec 23, 2014 at 9:12 AM, Henry Hung <YT...@winbond.com> wrote:

>  Hi All,
>
>
>
> I have a problem with broadcasting a serialize class object that returned
> by another not-serialize class, here is the sample code:
>
>
>
> class A extends java.io.Serializable {
>
>     def halo(): String = "halo"
>
> }
>
>
>
> class B {
>
>     def getA() = new A
>
> }
>
>
>
> val list = List(1)
>
>
>
> val b = new B
>
> val a = b.getA
>
>
>
> val p = sc.parallelize(list)
>
>
>
> // this will fail
>
> val bcA = sc.broadcast(a)
>
> p.map(x => {
>
>     bcA.value.halo()
>
> })
>
>
>
> // this will success
>
> val bcA = sc.broadcast(new A)
>
> p.map(x => {
>
>     bcA.value.halo()
>
> })
>
>
>
>
>
> A is a serializable class, where B is not-serialize.
>
> If I create a new object A through B method getA(), the map process will
> failed with exception “org.apache.spark.SparkException: Task not
> serializable, Caused by: java.io.NotSerializableException: $iwC$$iwC$B”
>
>
>
> I don’t know why spark will check if the B class serializable or not, is
> there a way to code this?
>
>
>
> Best regards,
>
> Henry
>
> ------------------------------
> The privileged confidential information contained in this email is
> intended for use only by the addressees as indicated by the original sender
> of this email. If you are not the addressee indicated in this email or are
> not responsible for delivery of the email to such a person, please kindly
> reply to the sender indicating this fact and delete all copies of it from
> your computer and network server immediately. Your cooperation is highly
> appreciated. It is advised that any unauthorized use of confidential
> information of Winbond is strictly prohibited; and any information in this
> email irrelevant to the official business of Winbond shall be deemed as
> neither given nor endorsed by Winbond.
>



-- 
Regards,
Madhukara Phatak
http://www.madhukaraphatak.com