You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Daniel Haviv <da...@veracity-group.com> on 2015/09/25 19:06:44 UTC
java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema
Hi,
I'm getting a NotSerializableException even though I'm creating all the my
objects from within the closure:
import org.apache.avro.generic.GenericDatumReader
import java.io.File
import org.apache.avro._
val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
val READER = new GenericDatumReader[GenericRecord](schema)
val bd = sc.broadcast(orig_schema.toString)
val rdd=sc.binaryFiles("/daniel").map(zibi => {
val schema_obj = new Schema.Parser
val schema2 = schema_obj.parse(bd.value)
})
I think that the problem is that Schema itself is an abstract class with
static methods (such as Parser).
Am I correct?
How can I overcome it ?
Thank you.
Daniel
Re: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema
Posted by Daniel Haviv <da...@veracity-group.com>.
I tried but I'm getting the same error (task not serializable)
> On 25 בספט׳ 2015, at 20:10, Ted Yu <yu...@gmail.com> wrote:
>
> Is the Schema.parse() call expensive ?
>
> Can you call it in the closure ?
>
>> On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv <da...@veracity-group.com> wrote:
>> Hi,
>> I'm getting a NotSerializableException even though I'm creating all the my objects from within the closure:
>> import org.apache.avro.generic.GenericDatumReader
>> import java.io.File
>> import org.apache.avro._
>>
>> val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
>> val READER = new GenericDatumReader[GenericRecord](schema)
>>
>> val bd = sc.broadcast(orig_schema.toString)
>>
>>
>> val rdd=sc.binaryFiles("/daniel").map(zibi => {
>> val schema_obj = new Schema.Parser
>> val schema2 = schema_obj.parse(bd.value)
>> })
>>
>> I think that the problem is that Schema itself is an abstract class with static methods (such as Parser).
>>
>> Am I correct?
>> How can I overcome it ?
>>
>> Thank you.
>> Daniel
>
Re: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema
Posted by Ted Yu <yu...@gmail.com>.
Is the Schema.parse() call expensive ?
Can you call it in the closure ?
On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv <
daniel.haviv@veracity-group.com> wrote:
> Hi,
> I'm getting a NotSerializableException even though I'm creating all the my
> objects from within the closure:
> import org.apache.avro.generic.GenericDatumReader
> import java.io.File
> import org.apache.avro._
>
> val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
> val READER = new GenericDatumReader[GenericRecord](schema)
>
> val bd = sc.broadcast(orig_schema.toString)
>
>
> val rdd=sc.binaryFiles("/daniel").map(zibi => {
> val schema_obj = new Schema.Parser
> val schema2 = schema_obj.parse(bd.value)
> })
>
> I think that the problem is that Schema itself is an abstract class with
> static methods (such as Parser).
>
> Am I correct?
> How can I overcome it ?
>
> Thank you.
> Daniel
>