You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Ronen Gross <ro...@gmail.com> on 2015/05/06 12:45:26 UTC
Spark1.2 Exception with Zeppelin
Hi,
I have an Exception when I try to run the Zeppelin Tutorial example.
Zeppelin in spark local mode worked OK but when I'm using spark in cluster
mode I got an error when I run the %sql
The Error is:
java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot be
cast to org.apache.spark.rdd.RDD at
org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
at scala.Option.map(Option.scala:145)
...
...
...
I used this command to build a distribution of Zeppelin:
mvn clean package -Pspark-1.2 -Phadoop-2.4 -DskipTests
-Dhadoop.version=2.5.0-cdh5.3.1 -Dspark.version=1.2.1 -P build-distr
Is it an error of using spark 1.2.0 ?
Thanks,
Ronen
Re: Spark1.2 Exception with Zeppelin
Posted by moon soo Lee <mo...@apache.org>.
Hi,
Yes, toDF() exists from spark 1.3. If you just remove .toDF(), example will
work for spark 1.2.
About the exception, we have related discussion in our old github
repository. https://github.com/NFLabs/zeppelin/issues/368
Thanks,
moon
On Tue, May 26, 2015 at 11:29 PM Ronen Gross <ro...@gmail.com> wrote:
> Hi,
>
> I added the -Pyarn" flag, modified Master property and updated zeppelin-env.sh
> file but still I get the same error when running the %sql
>
> java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot be
> cast to org.apache.spark.rdd.RDD at
> org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
> at
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
> at
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
> at scala.Option.map(Option.scala:145)
>
> At the ui I get this error: java.reflect.InvocationTargetException.
>
> I think that the example does not support the spark version in my
> environment which is 1.2.0.
>
> One step back in the tutorial I get an error when running .toDF().. I
> guess this is also because of spark version?
>
> Any suggestion? Maybe there is an older example I can use?
>
> Thanks,
> Ronen
>
>
>
> On Wed, May 6, 2015 at 4:37 PM, MrAsanjar . <af...@gmail.com> wrote:
>
>> Stas,
>> As far as I know you also have to update zeppelin-env.sh file also.
>> This is how my cluster configuration ( I used Juju to build the cluster
>> of LXC nodes on my laptop):
>> hadoop cluster nodes
>> 1 namenodes node
>> 1 resourcemanager node
>> 3+ compute nodes
>> Spark 1.3 node
>> 1 Spark+hadoop-plugin+zeppelin
>>
>> If you are using Ubuntu or testing on EWS, I could share my Zeppelin Juju
>> bundles. It builds above cluster in less then 15 minutes.
>>
>> On Wed, May 6, 2015 at 8:18 AM, Stas Zubarev <sz...@gmail.com> wrote:
>>
>>> I built Zeppelin with "-Pyarn" flag - does it mean that Zeppelin will be
>>> started in cluster mode on HDP 2.2 or I need additional configuration for
>>> it?
>>>
>>> On Wed, May 6, 2015 at 9:14 AM, MrAsanjar . <af...@gmail.com> wrote:
>>>
>>>> i assume your hadoop cluster is configured for yarn. make sure you are
>>>> build with "-Pyarn" flag also. Also verify you have "expost
>>>> HADOOP_CONF_DIR=" in your zeppelin-env.sh file
>>>>
>>>> On Wed, May 6, 2015 at 5:45 AM, Ronen Gross <ro...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have an Exception when I try to run the Zeppelin Tutorial example.
>>>>> Zeppelin in spark local mode worked OK but when I'm using spark in
>>>>> cluster mode I got an error when I run the %sql
>>>>>
>>>>> The Error is:
>>>>> java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot
>>>>> be cast to org.apache.spark.rdd.RDD at
>>>>> org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
>>>>> at
>>>>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>>>>> at
>>>>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>>>>> at scala.Option.map(Option.scala:145)
>>>>> ...
>>>>> ...
>>>>> ...
>>>>>
>>>>>
>>>>> I used this command to build a distribution of Zeppelin:
>>>>> mvn clean package -Pspark-1.2 -Phadoop-2.4 -DskipTests
>>>>> -Dhadoop.version=2.5.0-cdh5.3.1 -Dspark.version=1.2.1 -P build-distr
>>>>>
>>>>>
>>>>> Is it an error of using spark 1.2.0 ?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Ronen
>>>>>
>>>>
>>>>
>>>
>>
>
Zeppelin with highcharts on AWS.
Posted by "Wood, Dean Jr (GE Oil & Gas)" <De...@ge.com>.
Hi,
I’m trying to use AWS with zeppelin and am having some issues with using highcharts with zeppelin.
I’m not sure if this is a zeppelin or highcharts issue but maybe you’ll at least be able to clarify that. So I have a spark cluster on AWS and I have zeppelin running on another VM. When I try and plot with highcharts, it fails to open another window in my browser and the url it gives in the response is the private ip of the vm zeppelin is running on. I need some way to either change the url it is using to respond with or allow it to print within the existing zeppelin notebook.
Any ideas?
Dean
Re: Spark1.2 Exception with Zeppelin
Posted by Ronen Gross <ro...@gmail.com>.
Hi,
I added the -Pyarn" flag, modified Master property and updated zeppelin-env.sh
file but still I get the same error when running the %sql
java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot be
cast to org.apache.spark.rdd.RDD at
org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
at scala.Option.map(Option.scala:145)
At the ui I get this error: java.reflect.InvocationTargetException.
I think that the example does not support the spark version in my
environment which is 1.2.0.
One step back in the tutorial I get an error when running .toDF().. I guess
this is also because of spark version?
Any suggestion? Maybe there is an older example I can use?
Thanks,
Ronen
On Wed, May 6, 2015 at 4:37 PM, MrAsanjar . <af...@gmail.com> wrote:
> Stas,
> As far as I know you also have to update zeppelin-env.sh file also.
> This is how my cluster configuration ( I used Juju to build the cluster of
> LXC nodes on my laptop):
> hadoop cluster nodes
> 1 namenodes node
> 1 resourcemanager node
> 3+ compute nodes
> Spark 1.3 node
> 1 Spark+hadoop-plugin+zeppelin
>
> If you are using Ubuntu or testing on EWS, I could share my Zeppelin Juju
> bundles. It builds above cluster in less then 15 minutes.
>
> On Wed, May 6, 2015 at 8:18 AM, Stas Zubarev <sz...@gmail.com> wrote:
>
>> I built Zeppelin with "-Pyarn" flag - does it mean that Zeppelin will be
>> started in cluster mode on HDP 2.2 or I need additional configuration for
>> it?
>>
>> On Wed, May 6, 2015 at 9:14 AM, MrAsanjar . <af...@gmail.com> wrote:
>>
>>> i assume your hadoop cluster is configured for yarn. make sure you are
>>> build with "-Pyarn" flag also. Also verify you have "expost
>>> HADOOP_CONF_DIR=" in your zeppelin-env.sh file
>>>
>>> On Wed, May 6, 2015 at 5:45 AM, Ronen Gross <ro...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have an Exception when I try to run the Zeppelin Tutorial example.
>>>> Zeppelin in spark local mode worked OK but when I'm using spark in
>>>> cluster mode I got an error when I run the %sql
>>>>
>>>> The Error is:
>>>> java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot
>>>> be cast to org.apache.spark.rdd.RDD at
>>>> org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
>>>> at
>>>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>>>> at
>>>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>>>> at scala.Option.map(Option.scala:145)
>>>> ...
>>>> ...
>>>> ...
>>>>
>>>>
>>>> I used this command to build a distribution of Zeppelin:
>>>> mvn clean package -Pspark-1.2 -Phadoop-2.4 -DskipTests
>>>> -Dhadoop.version=2.5.0-cdh5.3.1 -Dspark.version=1.2.1 -P build-distr
>>>>
>>>>
>>>> Is it an error of using spark 1.2.0 ?
>>>>
>>>>
>>>> Thanks,
>>>> Ronen
>>>>
>>>
>>>
>>
>
Re: Spark1.2 Exception with Zeppelin
Posted by "MrAsanjar ." <af...@gmail.com>.
Stas,
As far as I know you also have to update zeppelin-env.sh file also.
This is how my cluster configuration ( I used Juju to build the cluster of
LXC nodes on my laptop):
hadoop cluster nodes
1 namenodes node
1 resourcemanager node
3+ compute nodes
Spark 1.3 node
1 Spark+hadoop-plugin+zeppelin
If you are using Ubuntu or testing on EWS, I could share my Zeppelin Juju
bundles. It builds above cluster in less then 15 minutes.
On Wed, May 6, 2015 at 8:18 AM, Stas Zubarev <sz...@gmail.com> wrote:
> I built Zeppelin with "-Pyarn" flag - does it mean that Zeppelin will be
> started in cluster mode on HDP 2.2 or I need additional configuration for
> it?
>
> On Wed, May 6, 2015 at 9:14 AM, MrAsanjar . <af...@gmail.com> wrote:
>
>> i assume your hadoop cluster is configured for yarn. make sure you are
>> build with "-Pyarn" flag also. Also verify you have "expost
>> HADOOP_CONF_DIR=" in your zeppelin-env.sh file
>>
>> On Wed, May 6, 2015 at 5:45 AM, Ronen Gross <ro...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have an Exception when I try to run the Zeppelin Tutorial example.
>>> Zeppelin in spark local mode worked OK but when I'm using spark in
>>> cluster mode I got an error when I run the %sql
>>>
>>> The Error is:
>>> java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot be
>>> cast to org.apache.spark.rdd.RDD at
>>> org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
>>> at
>>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>>> at
>>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>>> at scala.Option.map(Option.scala:145)
>>> ...
>>> ...
>>> ...
>>>
>>>
>>> I used this command to build a distribution of Zeppelin:
>>> mvn clean package -Pspark-1.2 -Phadoop-2.4 -DskipTests
>>> -Dhadoop.version=2.5.0-cdh5.3.1 -Dspark.version=1.2.1 -P build-distr
>>>
>>>
>>> Is it an error of using spark 1.2.0 ?
>>>
>>>
>>> Thanks,
>>> Ronen
>>>
>>
>>
>
Re: Spark1.2 Exception with Zeppelin
Posted by Stas Zubarev <sz...@gmail.com>.
I built Zeppelin with "-Pyarn" flag - does it mean that Zeppelin will be
started in cluster mode on HDP 2.2 or I need additional configuration for
it?
On Wed, May 6, 2015 at 9:14 AM, MrAsanjar . <af...@gmail.com> wrote:
> i assume your hadoop cluster is configured for yarn. make sure you are
> build with "-Pyarn" flag also. Also verify you have "expost
> HADOOP_CONF_DIR=" in your zeppelin-env.sh file
>
> On Wed, May 6, 2015 at 5:45 AM, Ronen Gross <ro...@gmail.com> wrote:
>
>> Hi,
>>
>> I have an Exception when I try to run the Zeppelin Tutorial example.
>> Zeppelin in spark local mode worked OK but when I'm using spark in
>> cluster mode I got an error when I run the %sql
>>
>> The Error is:
>> java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot be
>> cast to org.apache.spark.rdd.RDD at
>> org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
>> at
>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>> at
>> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
>> at scala.Option.map(Option.scala:145)
>> ...
>> ...
>> ...
>>
>>
>> I used this command to build a distribution of Zeppelin:
>> mvn clean package -Pspark-1.2 -Phadoop-2.4 -DskipTests
>> -Dhadoop.version=2.5.0-cdh5.3.1 -Dspark.version=1.2.1 -P build-distr
>>
>>
>> Is it an error of using spark 1.2.0 ?
>>
>>
>> Thanks,
>> Ronen
>>
>
>
Re: Spark1.2 Exception with Zeppelin
Posted by "MrAsanjar ." <af...@gmail.com>.
i assume your hadoop cluster is configured for yarn. make sure you are
build with "-Pyarn" flag also. Also verify you have "expost
HADOOP_CONF_DIR=" in your zeppelin-env.sh file
On Wed, May 6, 2015 at 5:45 AM, Ronen Gross <ro...@gmail.com> wrote:
> Hi,
>
> I have an Exception when I try to run the Zeppelin Tutorial example.
> Zeppelin in spark local mode worked OK but when I'm using spark in cluster
> mode I got an error when I run the %sql
>
> The Error is:
> java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf cannot be
> cast to org.apache.spark.rdd.RDD at
> org.apache.spark.SpaekContext$$anonfun$27.apply(SparkContext.scala:1045)
> at
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
> at
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:170)
> at scala.Option.map(Option.scala:145)
> ...
> ...
> ...
>
>
> I used this command to build a distribution of Zeppelin:
> mvn clean package -Pspark-1.2 -Phadoop-2.4 -DskipTests
> -Dhadoop.version=2.5.0-cdh5.3.1 -Dspark.version=1.2.1 -P build-distr
>
>
> Is it an error of using spark 1.2.0 ?
>
>
> Thanks,
> Ronen
>