You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Daniel Haviv <da...@veracity-group.com> on 2015/09/25 19:06:44 UTC

java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema

Hi,
I'm getting a NotSerializableException even though I'm creating all the my
objects from within the closure:
 import org.apache.avro.generic.GenericDatumReader
 import   java.io.File
import org.apache.avro._

 val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
  val READER = new GenericDatumReader[GenericRecord](schema)

val bd = sc.broadcast(orig_schema.toString)


 val rdd=sc.binaryFiles("/daniel").map(zibi => {
 val schema_obj =  new Schema.Parser
 val schema2 = schema_obj.parse(bd.value)
})

I think that the problem is that Schema itself is an abstract class with
static methods (such as Parser).

Am I correct?
How can I overcome it ?

Thank you.
Daniel

Re: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema

Posted by Daniel Haviv <da...@veracity-group.com>.
I tried but I'm getting the same error (task not serializable)

> On 25 בספט׳ 2015, at 20:10, Ted Yu <yu...@gmail.com> wrote:
> 
> Is the Schema.parse() call expensive ?
> 
> Can you call it in the closure ?
> 
>> On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv <da...@veracity-group.com> wrote:
>> Hi,
>> I'm getting a NotSerializableException even though I'm creating all the my objects from within the closure:
>>  import org.apache.avro.generic.GenericDatumReader
>>  import   java.io.File
>> import org.apache.avro._
>> 
>>  val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
>>   val READER = new GenericDatumReader[GenericRecord](schema)
>> 
>> val bd = sc.broadcast(orig_schema.toString)
>> 
>> 
>>  val rdd=sc.binaryFiles("/daniel").map(zibi => {  
>>  val schema_obj =  new Schema.Parser
>>  val schema2 = schema_obj.parse(bd.value)
>> })
>> 
>> I think that the problem is that Schema itself is an abstract class with static methods (such as Parser).
>> 
>> Am I correct? 
>> How can I overcome it ?
>> 
>> Thank you.
>> Daniel
> 

Re: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema

Posted by Ted Yu <yu...@gmail.com>.
Is the Schema.parse() call expensive ?

Can you call it in the closure ?

On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv <
daniel.haviv@veracity-group.com> wrote:

> Hi,
> I'm getting a NotSerializableException even though I'm creating all the my
> objects from within the closure:
>  import org.apache.avro.generic.GenericDatumReader
>  import   java.io.File
> import org.apache.avro._
>
>  val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
>   val READER = new GenericDatumReader[GenericRecord](schema)
>
> val bd = sc.broadcast(orig_schema.toString)
>
>
>  val rdd=sc.binaryFiles("/daniel").map(zibi => {
>  val schema_obj =  new Schema.Parser
>  val schema2 = schema_obj.parse(bd.value)
> })
>
> I think that the problem is that Schema itself is an abstract class with
> static methods (such as Parser).
>
> Am I correct?
> How can I overcome it ?
>
> Thank you.
> Daniel
>