You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by David Howell <da...@zipmoney.com.au> on 2017/06/27 04:46:33 UTC
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on
a standalone spark cluster
Hi,
I know this issue is resolved for reading from json, and tested for that use
case, but I'm seeing the exact same error message when writing to json.
java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser;
local class incompatible: stream classdesc serialVersionUID = 2, local class
serialVersionUID = 3
Easy to reproduce on AWS (also happens writing to HDFS)
val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
.toDF()
.write
.json(f"s3n://ReplaceWithBucketName/test/")
I know writing to json is not a common use case, so not an urgent issue.
I see the git commit looks like it is mostly replacing the import of
-import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang.StringUtils;
But the issue is with org.apache.commons.lang3.time.FastDateParser
I had a quick search in the codebase and couldn't find any imports of that
class directly or indirectly so I'm not really sure what would fix it.
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/InvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-spark-cluster-tp4900p5854.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by David Howell <da...@zipmoney.com.au>.
Hi Jeff,
My apologies, I realise this was caused by an uber jar I had loaded as a spark dependency which contained a conflicting library. Removing the dependency fixed the issue.
I've shaded & rebuilt the jar and no longer seeing the issue.
Sorry for the confusion!
-----Original Message-----
From: David Howell
Sent: Tuesday, 27 June 2017 4:44 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster
Hi Jeff,
The ticket says it is fixed from Zeppelin 0.7.0
I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.
See my repro example.
-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
Sent: Tuesday, 27 June 2017 4:40 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster
It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
Best Regard,
Jeff Zhang
On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi,
>I know this issue is resolved for reading from json, and tested for
>that use case, but I'm seeing the exact same error message when writing
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
> .toDF()
> .write
> .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
It works for me in master branch.
Maybe the issue is due to ZEPPELIN-2375
Could you try 0.7.2 ?
Best Regard,
Jeff Zhang
On 6/27/17, 2:43 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi Jeff,
>The ticket says it is fixed from Zeppelin 0.7.0
>
>I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json,
>but it still throws an error for writing to json.
>
>See my repro example.
>
>
>-----Original Message-----
>From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
>Sent: Tuesday, 27 June 2017 4:40 PM
>To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
>Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1
>on a standalone spark cluster
>
>
>It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
>
>
>Best Regard,
>Jeff Zhang
>
>
>
>
>
>On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>
>>Hi,
>>I know this issue is resolved for reading from json, and tested for
>>that use case, but I'm seeing the exact same error message when writing
>>to json.
>>
>>java.io.InvalidClassException:
>>org.apache.commons.lang3.time.FastDateParser;
>>local class incompatible: stream classdesc serialVersionUID = 2, local
>>class serialVersionUID = 3
>>
>>Easy to reproduce on AWS (also happens writing to HDFS)
>>
>>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>> .toDF()
>> .write
>> .json(f"s3n://ReplaceWithBucketName/test/")
>>
>>
>>I know writing to json is not a common use case, so not an urgent issue.
>>
>>I see the git commit looks like it is mostly replacing the import of
>>-import org.apache.commons.lang3.StringUtils;
>>+import org.apache.commons.lang.StringUtils;
>>
>>But the issue is with org.apache.commons.lang3.time.FastDateParser
>>
>>I had a quick search in the codebase and couldn't find any imports of
>>that class directly or indirectly so I'm not really sure what would fix
>>it.
>>
>>
>>
>>
>>--
>>View this message in context:
>>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>>m/I
>>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>>e-s
>>park-cluster-tp4900p5854.html
>>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>>list archive at Nabble.com.
>>
>
RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by David Howell <da...@zipmoney.com.au>.
Hi Jeff,
The ticket says it is fixed from Zeppelin 0.7.0
I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.
See my repro example.
-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
Sent: Tuesday, 27 June 2017 4:40 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster
It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
Best Regard,
Jeff Zhang
On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi,
>I know this issue is resolved for reading from json, and tested for
>that use case, but I'm seeing the exact same error message when writing
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
> .toDF()
> .write
> .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
Best Regard,
Jeff Zhang
On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi,
>I know this issue is resolved for reading from json, and tested for that
>use
>case, but I'm seeing the exact same error message when writing to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class
>serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
> .toDF()
> .write
> .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of that
>class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>