You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by David Howell <da...@zipmoney.com.au> on 2017/06/27 04:46:33 UTC

Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Hi, 
I know this issue is resolved for reading from json, and tested for that use
case, but I'm seeing the exact same error message when writing to json.

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser;
local class incompatible: stream classdesc serialVersionUID = 2, local class
serialVersionUID = 3

Easy to reproduce on AWS (also happens writing to HDFS)

val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
            .toDF()
            .write
            .json(f"s3n://ReplaceWithBucketName/test/")


I know writing to json is not a common use case, so not an urgent issue. 

I see the git commit looks like it is mostly replacing the import of 
-import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang.StringUtils;

But the issue is with org.apache.commons.lang3.time.FastDateParser

I had a quick search in the codebase and couldn't find any imports of that
class directly or indirectly so I'm not really sure what would fix it.




--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/InvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-spark-cluster-tp4900p5854.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.

RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Posted by David Howell <da...@zipmoney.com.au>.
Hi Jeff, 
My apologies, I realise this was caused by an uber jar I had loaded as a spark dependency which contained a conflicting library. Removing the dependency fixed the issue.

 I've shaded & rebuilt the jar and no longer seeing the issue.

Sorry for the confusion!

-----Original Message-----
From: David Howell 
Sent: Tuesday, 27 June 2017 4:44 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Hi Jeff, 
The ticket says it is fixed from Zeppelin 0.7.0

I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.

See my repro example.


-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com] 
Sent: Tuesday, 27 June 2017 4:40 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster


It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977


Best Regard,
Jeff Zhang





On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:

>Hi,
>I know this issue is resolved for reading from json, and tested for 
>that use case, but I'm seeing the exact same error message when writing 
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local 
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>            .toDF()
>            .write
>            .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of 
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of 
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I 
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing 
>list archive at Nabble.com.
>


Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
It works for me in master branch.

Maybe the issue is due to ZEPPELIN-2375

Could you try 0.7.2 ?



Best Regard,
Jeff Zhang





On 6/27/17, 2:43 PM, "David Howell" <da...@zipmoney.com.au> wrote:

>Hi Jeff, 
>The ticket says it is fixed from Zeppelin 0.7.0
>
>I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json,
>but it still throws an error for writing to json.
>
>See my repro example.
>
>
>-----Original Message-----
>From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
>Sent: Tuesday, 27 June 2017 4:40 PM
>To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
>Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1
>on a standalone spark cluster
>
>
>It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
>
>
>Best Regard,
>Jeff Zhang
>
>
>
>
>
>On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>
>>Hi,
>>I know this issue is resolved for reading from json, and tested for
>>that use case, but I'm seeing the exact same error message when writing
>>to json.
>>
>>java.io.InvalidClassException:
>>org.apache.commons.lang3.time.FastDateParser;
>>local class incompatible: stream classdesc serialVersionUID = 2, local
>>class serialVersionUID = 3
>>
>>Easy to reproduce on AWS (also happens writing to HDFS)
>>
>>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>>            .toDF()
>>            .write
>>            .json(f"s3n://ReplaceWithBucketName/test/")
>>
>>
>>I know writing to json is not a common use case, so not an urgent issue.
>>
>>I see the git commit looks like it is mostly replacing the import of
>>-import org.apache.commons.lang3.StringUtils;
>>+import org.apache.commons.lang.StringUtils;
>>
>>But the issue is with org.apache.commons.lang3.time.FastDateParser
>>
>>I had a quick search in the codebase and couldn't find any imports of
>>that class directly or indirectly so I'm not really sure what would fix
>>it.
>>
>>
>>
>>
>>--
>>View this message in context:
>>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>>m/I 
>>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>>e-s
>>park-cluster-tp4900p5854.html
>>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>>list archive at Nabble.com.
>>
>


RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Posted by David Howell <da...@zipmoney.com.au>.
Hi Jeff, 
The ticket says it is fixed from Zeppelin 0.7.0

I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.

See my repro example.


-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com] 
Sent: Tuesday, 27 June 2017 4:40 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster


It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977


Best Regard,
Jeff Zhang





On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:

>Hi,
>I know this issue is resolved for reading from json, and tested for 
>that use case, but I'm seeing the exact same error message when writing 
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local 
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>            .toDF()
>            .write
>            .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of 
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of 
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I 
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing 
>list archive at Nabble.com.
>


Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977


Best Regard,
Jeff Zhang





On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:

>Hi, 
>I know this issue is resolved for reading from json, and tested for that
>use
>case, but I'm seeing the exact same error message when writing to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class
>serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>            .toDF()
>            .write
>            .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of that
>class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>