You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by "Wood, Dean Jr (GE Oil & Gas)" <De...@ge.com> on 2015/06/23 20:25:09 UTC
Commands working in Spark shell failing in Zeppelin
Hi,
I’ve tried to send this before but it didn’t come through.
I have a fairly simple set of commands which work in the spark shell. However, they do not work on zeppelin. The commands are:
val wbIPR=Array(8.40, 8.09, 7.56, 6.83, 5.88, 4.72, 3.36, 1.78, 0.0)
val pressureIPR=Array(0.00,10000000.00,20000000.00,30000000.00,40000000.00,50000000.00,60000000.00,70000000.00,80000000.00)
case class IPR (w:Array[Double],p:Array[Double])
val iprInit=IPR(wbIPR,pressureIPR)
val iprList=sc.parallelize((1.0 to 0.9 by -0.001).map(k => IPR(iprInit.w.map(_ * k), iprInit.p)).zipWithIndex,5)
iprList.count()
The error message is:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 54, ip-172-31-7-206.ec2.internal): ExecutorLostFailure (executor 14 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
Any ideas why this could be failing? I don’t think it’s a compile issue, because if I run
val iprList=sc.parallelize(pressureIPR)
iprList.count()
everything works.
Thanks
Dean
Re: Commands working in Spark shell failing in Zeppelin
Posted by moon soo Lee <mo...@apache.org>.
Hi,
Thanks for sharing the problem.
I tried your code and got the same exception.
I've found it is coming from one of our recently merged code.
Here's issue for this problem
https://issues.apache.org/jira/browse/ZEPPELIN-127
Here's quick fix for it.
https://github.com/apache/incubator-zeppelin/pull/117
Let me know if it helps.
Best,
moon
On Tue, Jun 23, 2015 at 12:50 PM Wood, Dean Jr (GE Oil & Gas) <
Dean1.Wood@ge.com> wrote:
> P.S. This is working on a remote spark cluster in AWS. Spark 1.3.0 or
> 1.3.1. Fails in both.
>
> > On 23 Jun 2015, at 19:25, Wood, Dean Jr (GE Oil & Gas) <
> Dean1.Wood@ge.com> wrote:
> >
> > Hi,
> >
> > I’ve tried to send this before but it didn’t come through.
> >
> > I have a fairly simple set of commands which work in the spark shell.
> However, they do not work on zeppelin. The commands are:
> >
> > val wbIPR=Array(8.40, 8.09, 7.56, 6.83, 5.88,
> 4.72, 3.36, 1.78, 0.0)
> > val
> pressureIPR=Array(0.00,10000000.00,20000000.00,30000000.00,40000000.00,50000000.00,60000000.00,70000000.00,80000000.00)
> > case class IPR (w:Array[Double],p:Array[Double])
> > val iprInit=IPR(wbIPR,pressureIPR)
> > val iprList=sc.parallelize((1.0 to 0.9 by -0.001).map(k =>
> IPR(iprInit.w.map(_ * k), iprInit.p)).zipWithIndex,5)
> > iprList.count()
> >
> > The error message is:
> > org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage
> 3.0 (TID 54, ip-172-31-7-206.ec2.internal): ExecutorLostFailure (executor
> 14 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
> >
> > Any ideas why this could be failing? I don’t think it’s a compile issue,
> because if I run
> >
> > val iprList=sc.parallelize(pressureIPR)
> > iprList.count()
> >
> > everything works.
> >
> > Thanks
> >
> > Dean
>
>
Re: Commands working in Spark shell failing in Zeppelin
Posted by "Wood, Dean Jr (GE Oil & Gas)" <De...@ge.com>.
P.S. This is working on a remote spark cluster in AWS. Spark 1.3.0 or 1.3.1. Fails in both.
> On 23 Jun 2015, at 19:25, Wood, Dean Jr (GE Oil & Gas) <De...@ge.com> wrote:
>
> Hi,
>
> I’ve tried to send this before but it didn’t come through.
>
> I have a fairly simple set of commands which work in the spark shell. However, they do not work on zeppelin. The commands are:
>
> val wbIPR=Array(8.40, 8.09, 7.56, 6.83, 5.88, 4.72, 3.36, 1.78, 0.0)
> val pressureIPR=Array(0.00,10000000.00,20000000.00,30000000.00,40000000.00,50000000.00,60000000.00,70000000.00,80000000.00)
> case class IPR (w:Array[Double],p:Array[Double])
> val iprInit=IPR(wbIPR,pressureIPR)
> val iprList=sc.parallelize((1.0 to 0.9 by -0.001).map(k => IPR(iprInit.w.map(_ * k), iprInit.p)).zipWithIndex,5)
> iprList.count()
>
> The error message is:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 54, ip-172-31-7-206.ec2.internal): ExecutorLostFailure (executor 14 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>
> Any ideas why this could be failing? I don’t think it’s a compile issue, because if I run
>
> val iprList=sc.parallelize(pressureIPR)
> iprList.count()
>
> everything works.
>
> Thanks
>
> Dean