You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Zuhair Khayyat <zu...@gmail.com> on 2013/11/27 16:22:23 UTC
Modifying RDD.scala
Dear SPARK members,
I am trying to start developing on SPARK source code. I have added a new
dummy function in RDD.scala to test if it compiles and runs. The modified
Spark compiled correctly but when I execute my code I got the following
error:
java.io.InvalidClassException: spark.RDD; local class incompatible: stream
classdesc serialVersionUID = 5151096093324583655, local class
serialVersionUID = 9012954318378784201
at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
at
spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:54)
at
spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:111)
at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
at spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
at spark.executor.Executor$TaskRunner.run(Executor.scala:96)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
13/11/27 17:47:43 ERROR executor.StandaloneExecutorBackend: Driver or
worker disconnected! Shutting down.
Can you please help me to find out what went wrong? Thank you
Zuhair Khayyat
Re: Modifying RDD.scala
Posted by Zuhair Khayyat <zu...@gmail.com>.
Dear Gerard,
I don’t have the problem when running on a single machine.
I think I found the problem; since I build my project using maven, the old Spark copy is still cached in my home directory (~/.m2/repository/org…). Every time I modify my local spark and build it, I have to clean whatever Maven is caching in my home directory.
Thank you
Zuhair Khayyat
On Nov 28, 2013, at 2:29 PM, Gerard Maas <ge...@gmail.com> wrote:
> Hi Zuhair,
>
> Following the exception, you have two different versions somewhere. Do you
> get the same behavior if you use it in a single node?
> Maybe Spark veterans have more specific tips for you.
>
> kr, Gerard.
>
>
> On Wed, Nov 27, 2013 at 5:00 PM, Zuhair Khayyat <zu...@gmail.com>wrote:
>
>> Dear Gerard,
>>
>> All servers share the spark binaries through NFS; It is unlikly that other
>> servers contains the old class. I will test later with one server and see
>> if I got the same problem..
>>
>> Regards,
>> Zuhair Khayyat
>>
>> On Nov 27, 2013, at 6:29 PM, Gerard Maas <ge...@gmail.com> wrote:
>>
>>> From the looks of your exception, you modified your local class, but you
>>> forgot to deploy those local changes to the cluster. This error msg:
>>> classdesc serialVersionUID = 5151096093324583655, local class
>>> serialVersionUID = 9012954318378784201
>>>
>>> indicates that a version being de-serialized is different from the local
>>> version. Make sure you deploy your changes across your Spark cluster.
>>>
>>> -kr, Gerard.
>>>
>>>
>>> On Wed, Nov 27, 2013 at 4:22 PM, Zuhair Khayyat <
>> zuhair.khayyat@gmail.com>wrote:
>>>
>>>> Dear SPARK members,
>>>>
>>>> I am trying to start developing on SPARK source code. I have added a new
>>>> dummy function in RDD.scala to test if it compiles and runs. The
>> modified
>>>> Spark compiled correctly but when I execute my code I got the following
>>>> error:
>>>>
>>>> java.io.InvalidClassException: spark.RDD; local class incompatible:
>> stream
>>>> classdesc serialVersionUID = 5151096093324583655, local class
>>>> serialVersionUID = 9012954318378784201
>>>> at
>>>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>>>> at
>>>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
>>>> at
>>>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
>>>> at
>>>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
>>>> at
>>>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
>>>> at
>>>>
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>>>> at
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>>>> at
>>>> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
>>>> at
>>>> spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:54)
>>>> at
>>>> spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:111)
>>>> at
>>>> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
>>>> at
>>>>
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>>>> at
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>>>> at
>>>> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
>>>> at
>>>> spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
>>>> at spark.executor.Executor$TaskRunner.run(Executor.scala:96)
>>>> at
>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:724)
>>>> 13/11/27 17:47:43 ERROR executor.StandaloneExecutorBackend: Driver or
>>>> worker disconnected! Shutting down.
>>>>
>>>> Can you please help me to find out what went wrong? Thank you
>>>>
>>>> Zuhair Khayyat
>>>>
>>
>>
Re: Modifying RDD.scala
Posted by Gerard Maas <ge...@gmail.com>.
Hi Zuhair,
Following the exception, you have two different versions somewhere. Do you
get the same behavior if you use it in a single node?
Maybe Spark veterans have more specific tips for you.
kr, Gerard.
On Wed, Nov 27, 2013 at 5:00 PM, Zuhair Khayyat <zu...@gmail.com>wrote:
> Dear Gerard,
>
> All servers share the spark binaries through NFS; It is unlikly that other
> servers contains the old class. I will test later with one server and see
> if I got the same problem..
>
> Regards,
> Zuhair Khayyat
>
> On Nov 27, 2013, at 6:29 PM, Gerard Maas <ge...@gmail.com> wrote:
>
> > From the looks of your exception, you modified your local class, but you
> > forgot to deploy those local changes to the cluster. This error msg:
> > classdesc serialVersionUID = 5151096093324583655, local class
> > serialVersionUID = 9012954318378784201
> >
> > indicates that a version being de-serialized is different from the local
> > version. Make sure you deploy your changes across your Spark cluster.
> >
> > -kr, Gerard.
> >
> >
> > On Wed, Nov 27, 2013 at 4:22 PM, Zuhair Khayyat <
> zuhair.khayyat@gmail.com>wrote:
> >
> >> Dear SPARK members,
> >>
> >> I am trying to start developing on SPARK source code. I have added a new
> >> dummy function in RDD.scala to test if it compiles and runs. The
> modified
> >> Spark compiled correctly but when I execute my code I got the following
> >> error:
> >>
> >> java.io.InvalidClassException: spark.RDD; local class incompatible:
> stream
> >> classdesc serialVersionUID = 5151096093324583655, local class
> >> serialVersionUID = 9012954318378784201
> >> at
> >> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> >> at
> >> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
> >> at
> >> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
> >> at
> >> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
> >> at
> >> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
> >> at
> >>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
> >> at
> >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
> >> at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> >> at
> >> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
> >> at
> >> spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:54)
> >> at
> >> spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:111)
> >> at
> >> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
> >> at
> >>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
> >> at
> >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
> >> at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> >> at
> >> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
> >> at
> >> spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
> >> at spark.executor.Executor$TaskRunner.run(Executor.scala:96)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:724)
> >> 13/11/27 17:47:43 ERROR executor.StandaloneExecutorBackend: Driver or
> >> worker disconnected! Shutting down.
> >>
> >> Can you please help me to find out what went wrong? Thank you
> >>
> >> Zuhair Khayyat
> >>
>
>
Re: Modifying RDD.scala
Posted by Zuhair Khayyat <zu...@gmail.com>.
Dear Gerard,
All servers share the spark binaries through NFS; It is unlikly that other servers contains the old class. I will test later with one server and see if I got the same problem..
Regards,
Zuhair Khayyat
On Nov 27, 2013, at 6:29 PM, Gerard Maas <ge...@gmail.com> wrote:
> From the looks of your exception, you modified your local class, but you
> forgot to deploy those local changes to the cluster. This error msg:
> classdesc serialVersionUID = 5151096093324583655, local class
> serialVersionUID = 9012954318378784201
>
> indicates that a version being de-serialized is different from the local
> version. Make sure you deploy your changes across your Spark cluster.
>
> -kr, Gerard.
>
>
> On Wed, Nov 27, 2013 at 4:22 PM, Zuhair Khayyat <zu...@gmail.com>wrote:
>
>> Dear SPARK members,
>>
>> I am trying to start developing on SPARK source code. I have added a new
>> dummy function in RDD.scala to test if it compiles and runs. The modified
>> Spark compiled correctly but when I execute my code I got the following
>> error:
>>
>> java.io.InvalidClassException: spark.RDD; local class incompatible: stream
>> classdesc serialVersionUID = 5151096093324583655, local class
>> serialVersionUID = 9012954318378784201
>> at
>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>> at
>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
>> at
>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
>> at
>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
>> at
>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
>> at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
>> at
>> spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:54)
>> at
>> spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:111)
>> at
>> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
>> at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
>> at
>> spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
>> at spark.executor.Executor$TaskRunner.run(Executor.scala:96)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> 13/11/27 17:47:43 ERROR executor.StandaloneExecutorBackend: Driver or
>> worker disconnected! Shutting down.
>>
>> Can you please help me to find out what went wrong? Thank you
>>
>> Zuhair Khayyat
>>
Re: Modifying RDD.scala
Posted by Gerard Maas <ge...@gmail.com>.
>From the looks of your exception, you modified your local class, but you
forgot to deploy those local changes to the cluster. This error msg:
classdesc serialVersionUID = 5151096093324583655, local class
serialVersionUID = 9012954318378784201
indicates that a version being de-serialized is different from the local
version. Make sure you deploy your changes across your Spark cluster.
-kr, Gerard.
On Wed, Nov 27, 2013 at 4:22 PM, Zuhair Khayyat <zu...@gmail.com>wrote:
> Dear SPARK members,
>
> I am trying to start developing on SPARK source code. I have added a new
> dummy function in RDD.scala to test if it compiles and runs. The modified
> Spark compiled correctly but when I execute my code I got the following
> error:
>
> java.io.InvalidClassException: spark.RDD; local class incompatible: stream
> classdesc serialVersionUID = 5151096093324583655, local class
> serialVersionUID = 9012954318378784201
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
> at
> spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:54)
> at
> spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:111)
> at
> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
> at
> spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
> at spark.executor.Executor$TaskRunner.run(Executor.scala:96)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> 13/11/27 17:47:43 ERROR executor.StandaloneExecutorBackend: Driver or
> worker disconnected! Shutting down.
>
> Can you please help me to find out what went wrong? Thank you
>
> Zuhair Khayyat
>