Spark Streaming Error Help -> ERROR actor.OneForOneStrategy: key not found:

[jschindler <jo...@utexas.edu>]

I am getting this "ERROR actor.OneForOneStrategy: key not found:" exception
when I run my code and I'm not sure where it is looking for a key.  My set
up is I send packets to a third party service which then uses a webhook to
hit one of our servers, which then logs it using kafka.  I am just trying to
total up the number of elements/packets that come in using
".countByWindow(Seconds(30), Seconds(15)).print()".  I don't see anything in
the docs about countByWindow demanding a key value pair.  

I still see counts as an output so I'm thinking the exception is hurting
anything...

Also I am confused about some of the counts.  I start Spark up and it has
been producing counts greater than zero when I haven't sent any data packets
since the start of that run.  Are there packets saved that didn't get pushed
through from the last session?  This seems odd, I would think the count
would produce 0 if I haven't sent any data...

I'll post the code and exception error trace below, thanks for any help!

import org.apache.spark.streaming.kafka._
import org.apache.spark.SparkConf


object RunningPageTotal {
  def main(args: Array[String]) {

    val ssc = new StreamingContext("local[2]", "PageHitTotals", Seconds(5))
    ssc.checkpoint("checkpoint") 
    val topicMap = Map("User.Profile.Page.Hits" -> 1)

    val pageHitStream = KafkaUtils.createStream(ssc,
"Zookeeper_1,Zookeeper_1,Zookeeper_3", "Cons1", topicMap).map(_._2)
    val totalHits = pageHitStream.countByWindow(Seconds(30),
Seconds(15)).print()

    ssc.start()
    ssc.awaitTermination()
  }
}


14/07/03 12:42:25 INFO dstream.MappedDStream: Time 1404409345000 ms is
invalid as zeroTime is 1404409335000 ms and slideDuration is 15000 ms and
difference is 10000 ms
14/07/03 12:42:25 ERROR actor.OneForOneStrategy: key not found:
1404409345000 ms
java.util.NoSuchElementException: key not found: 1404409345000 ms
	at scala.collection.MapLike$class.default(MapLike.scala:228)
	at scala.collection.AbstractMap.default(Map.scala:58)
	at scala.collection.mutable.HashMap.apply(HashMap.scala:64)
	at
org.apache.spark.streaming.dstream.ReceiverInputDStream.getReceivedBlockInfo(ReceiverInputDStream.scala:77)
	at
org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:225)
	at
org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:223)
	at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
	at
org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:223)
	at
org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:165)
	at
org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1$$anonfun$receive$1.applyOrElse(JobGenerator.scala:76)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
	at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Error-Help-ERROR-actor-OneForOneStrategy-key-not-found-tp8746.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark Streaming Error Help -> ERROR actor.OneForOneStrategy: key not found:

[jschindler <jo...@utexas.edu>]

I think I have found my answers but if anyone has thoughts please share.

After testing for a while I think the error doesn't have any effect on the
process.

I think it is the case that there must be elements left in the window from
last run otherwise my system is completely whack.

Please let me know if any of this looks incorrect, thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Error-Help-ERROR-actor-OneForOneStrategy-key-not-found-tp8746p8750.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.