You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Antoine <a....@gmail.com> on 2017/01/20 21:19:29 UTC
InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Hei,
I'm trying to use Zeppelin from the master git branch with spark-2.1 and I
get an invalid class exeption when I use a standalone spark cluster.
java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser;
local class incompatible: stream classdesc serialVersionUID = 2, local
class serialVersionUID = 3
To reproduce the error, I configure the spark interpreter to connect to a
standalone cluster, it works with a local spark, and to load a file. For
example, spark.read.json("/data/file.json").
I'm building and running Zeppelin with docker, based on the
dylanmei/docker-zeppelin docker image, but with a few changes to build the
master branch (npm must be installed and bower needs to be configured to
run with the root user), and using spark 2.1 instead of spark 2.0.
Can you reproduce the error ? Is there something I'm missing when I build
Zeppelin ?
Thanks
RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by David Howell <da...@zipmoney.com.au>.
Hi Jeff,
My apologies, I realise this was caused by an uber jar I had loaded as a spark dependency which contained a conflicting library. Removing the dependency fixed the issue.
I've shaded & rebuilt the jar and no longer seeing the issue.
Sorry for the confusion!
-----Original Message-----
From: David Howell
Sent: Tuesday, 27 June 2017 4:44 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster
Hi Jeff,
The ticket says it is fixed from Zeppelin 0.7.0
I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.
See my repro example.
-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
Sent: Tuesday, 27 June 2017 4:40 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster
It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
Best Regard,
Jeff Zhang
On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi,
>I know this issue is resolved for reading from json, and tested for
>that use case, but I'm seeing the exact same error message when writing
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
> .toDF()
> .write
> .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
It works for me in master branch.
Maybe the issue is due to ZEPPELIN-2375
Could you try 0.7.2 ?
Best Regard,
Jeff Zhang
On 6/27/17, 2:43 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi Jeff,
>The ticket says it is fixed from Zeppelin 0.7.0
>
>I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json,
>but it still throws an error for writing to json.
>
>See my repro example.
>
>
>-----Original Message-----
>From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
>Sent: Tuesday, 27 June 2017 4:40 PM
>To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
>Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1
>on a standalone spark cluster
>
>
>It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
>
>
>Best Regard,
>Jeff Zhang
>
>
>
>
>
>On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>
>>Hi,
>>I know this issue is resolved for reading from json, and tested for
>>that use case, but I'm seeing the exact same error message when writing
>>to json.
>>
>>java.io.InvalidClassException:
>>org.apache.commons.lang3.time.FastDateParser;
>>local class incompatible: stream classdesc serialVersionUID = 2, local
>>class serialVersionUID = 3
>>
>>Easy to reproduce on AWS (also happens writing to HDFS)
>>
>>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>> .toDF()
>> .write
>> .json(f"s3n://ReplaceWithBucketName/test/")
>>
>>
>>I know writing to json is not a common use case, so not an urgent issue.
>>
>>I see the git commit looks like it is mostly replacing the import of
>>-import org.apache.commons.lang3.StringUtils;
>>+import org.apache.commons.lang.StringUtils;
>>
>>But the issue is with org.apache.commons.lang3.time.FastDateParser
>>
>>I had a quick search in the codebase and couldn't find any imports of
>>that class directly or indirectly so I'm not really sure what would fix
>>it.
>>
>>
>>
>>
>>--
>>View this message in context:
>>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>>m/I
>>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>>e-s
>>park-cluster-tp4900p5854.html
>>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>>list archive at Nabble.com.
>>
>
RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by David Howell <da...@zipmoney.com.au>.
Hi Jeff,
The ticket says it is fixed from Zeppelin 0.7.0
I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.
See my repro example.
-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
Sent: Tuesday, 27 June 2017 4:40 PM
To: users@zeppelin.apache.org; users@zeppelin.incubator.apache.org
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster
It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
Best Regard,
Jeff Zhang
On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi,
>I know this issue is resolved for reading from json, and tested for
>that use case, but I'm seeing the exact same error message when writing
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
> .toDF()
> .write
> .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
Best Regard,
Jeff Zhang
On 6/27/17, 12:46 PM, "David Howell" <da...@zipmoney.com.au> wrote:
>Hi,
>I know this issue is resolved for reading from json, and tested for that
>use
>case, but I'm seeing the exact same error message when writing to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class
>serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
> .toDF()
> .write
> .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of that
>class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on
a standalone spark cluster
Posted by David Howell <da...@zipmoney.com.au>.
Hi,
I know this issue is resolved for reading from json, and tested for that use
case, but I'm seeing the exact same error message when writing to json.
java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser;
local class incompatible: stream classdesc serialVersionUID = 2, local class
serialVersionUID = 3
Easy to reproduce on AWS (also happens writing to HDFS)
val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
.toDF()
.write
.json(f"s3n://ReplaceWithBucketName/test/")
I know writing to json is not a common use case, so not an urgent issue.
I see the git commit looks like it is mostly replacing the import of
-import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang.StringUtils;
But the issue is with org.apache.commons.lang3.time.FastDateParser
I had a quick search in the codebase and couldn't find any imports of that
class directly or indirectly so I'm not really sure what would fix it.
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/InvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-spark-cluster-tp4900p5854.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by Antoine <a....@gmail.com>.
Thanks Jonathan, I rebuilt the image and it now works perfectly. My image
was built a few hours before the fix. Sorry for the noise.
2017-01-20 23:32 GMT+02:00 Jonathan Kelly <jo...@gmail.com>:
> Hi, Antoine, this issue was being tracked in https://issues.apache.org/
> jira/browse/ZEPPELIN-1977, but it is now resolved as of yesterday (looks
> like about 18 hours ago). Maybe you need to pull from master again and
> rebuild?
>
> ~ Jonathan
>
> On Fri, Jan 20, 2017 at 1:19 PM Antoine <a....@gmail.com> wrote:
>
>> Hei,
>>
>> I'm trying to use Zeppelin from the master git branch with spark-2.1 and
>> I get an invalid class exeption when I use a standalone spark cluster.
>>
>> java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser;
>> local class incompatible: stream classdesc serialVersionUID = 2, local
>> class serialVersionUID = 3
>>
>> To reproduce the error, I configure the spark interpreter to connect to a
>> standalone cluster, it works with a local spark, and to load a file. For
>> example, spark.read.json("/data/file.json").
>>
>> I'm building and running Zeppelin with docker, based on the
>> dylanmei/docker-zeppelin docker image, but with a few changes to build the
>> master branch (npm must be installed and bower needs to be configured to
>> run with the root user), and using spark 2.1 instead of spark 2.0.
>>
>> Can you reproduce the error ? Is there something I'm missing when I build
>> Zeppelin ?
>>
>> Thanks
>>
>
Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a
standalone spark cluster
Posted by Jonathan Kelly <jo...@gmail.com>.
Hi, Antoine, this issue was being tracked in
https://issues.apache.org/jira/browse/ZEPPELIN-1977, but it is now resolved
as of yesterday (looks like about 18 hours ago). Maybe you need to pull
from master again and rebuild?
~ Jonathan
On Fri, Jan 20, 2017 at 1:19 PM Antoine <a....@gmail.com> wrote:
> Hei,
>
> I'm trying to use Zeppelin from the master git branch with spark-2.1 and I
> get an invalid class exeption when I use a standalone spark cluster.
>
> java.io.InvalidClassException:
> org.apache.commons.lang3.time.FastDateParser; local class incompatible:
> stream classdesc serialVersionUID = 2, local class serialVersionUID = 3
>
> To reproduce the error, I configure the spark interpreter to connect to a
> standalone cluster, it works with a local spark, and to load a file. For
> example, spark.read.json("/data/file.json").
>
> I'm building and running Zeppelin with docker, based on the
> dylanmei/docker-zeppelin docker image, but with a few changes to build the
> master branch (npm must be installed and bower needs to be configured to
> run with the root user), and using spark 2.1 instead of spark 2.0.
>
> Can you reproduce the error ? Is there something I'm missing when I build
> Zeppelin ?
>
> Thanks
>