You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Rado Buransky (JIRA)" <ji...@apache.org> on 2016/01/07 14:49:39 UTC

[jira] [Created] (SPARK-12693) OffsetOutOfRangeException cause by retention

Rado Buransky created SPARK-12693:
-------------------------------------

             Summary: OffsetOutOfRangeException cause by retention
                 Key: SPARK-12693
                 URL: https://issues.apache.org/jira/browse/SPARK-12693
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.6.0
         Environment: Ubuntu 64bit, Intel i7
            Reporter: Rado Buransky
            Priority: Minor


I am running Kafka server locally with extremely low retention of 3 seconds and with 1 second segmentation. I create direct Kafka stream with auto.offset.reset = smallest. 

In case of bad luck (happens actually quite often in my case) the smallest offset retrieved druing stream initialization doesn't already exists when streaming actually starts.

Complete source code of the Spark Streaming application is here:
https://github.com/pygmalios/spark-checkpoint-experience/blob/cb27ab83b7a29e619386b56e68a755d7bd73fc46/src/main/scala/com/pygmalios/sparkCheckpointExperience/spark/SparkApp.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org