You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Alan Roche (JIRA)" <ji...@apache.org> on 2016/11/06 19:07:58 UTC
[jira] [Resolved] (SPARK-18293) Spark Kinesis Streaming exits on
NoSuchMethodError
[ https://issues.apache.org/jira/browse/SPARK-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Roche resolved SPARK-18293.
--------------------------------
Resolution: Not A Problem
Thanks Sean. You are right, - a run-time problem indeed, - which as mentioned, I had suspected but the culprit was difficult to track down - but located now..
Sorry for consuming your time.
> Spark Kinesis Streaming exits on NoSuchMethodError
> --------------------------------------------------
>
> Key: SPARK-18293
> URL: https://issues.apache.org/jira/browse/SPARK-18293
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 1.6.2
> Environment: Release label:emr-4.8.0
> Hadoop distribution:Amazon 2.7.2
> Applications:Spark 1.6.2
> Java JRE 8
> Yarn-Cluster mode
> Reporter: Alan Roche
> Priority: Critical
> Attachments: spark-kinesis-app-deps.txt
>
>
> Spark Driver Application exits with the below stack trace and info.
> My first intuition was that we had a 2.11 on our classpath giving rise to the _LongRef.create_ error , - but a "mvn dependency:tree" indicates that this is not the case and we are using spark-streaming-kinesis-asl_2.10-1.6.2.jar
> (Attaching output pf mvn dependency:tree to show all dependencies).
> NOTE: Our application jar is called integrity-test-spark:jar:1.0-SNAPSHOT, which is an integrity tester that runs on our clusters.
> java.lang.NoSuchMethodError: scala.runtime.LongRef.create(J)Lscala/runtime/LongRef;
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.retryOrTimeout(KinesisBackedBlockRDD.scala:245)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getKinesisIterator(KinesisBackedBlockRDD.scala:235)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getRecords(KinesisBackedBlockRDD.scala:197)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getNext(KinesisBackedBlockRDD.scala:158)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getNext(KinesisBackedBlockRDD.scala:131)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
> at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> 18:48:57
> 16/11/05 17:23:59 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-14,5,main]
> 16/11/05 17:23:59 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-14,5,main]
>
> 18:48:57
> java.lang.NoSuchMethodError: scala.runtime.LongRef.create(J)Lscala/runtime/LongRef; at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.retryOrTimeout(KinesisBackedBlockRDD.scala:245) at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getKinesisIterator(KinesisBackedBlockRDD.scala:235) at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getRecords(K
> java.lang.NoSuchMethodError: scala.runtime.LongRef.create(J)Lscala/runtime/LongRef;
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.retryOrTimeout(KinesisBackedBlockRDD.scala:245)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getKinesisIterator(KinesisBackedBlockRDD.scala:235)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getRecords(KinesisBackedBlockRDD.scala:197)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getNext(KinesisBackedBlockRDD.scala:158)
> at org.apache.spark.streaming.kinesis.KinesisSequenceRangeIterator.getNext(KinesisBackedBlockRDD.scala:131)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
> at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> 18:48:57
> 16/11/05 17:23:59 INFO storage.DiskBlockManager: Shutdown hook called
> 16/11/05 17:23:59 INFO storage.DiskBlockManager: Shutdown hook called
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org