You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Taukir (JIRA)" <ji...@apache.org> on 2017/03/16 12:21:41 UTC
[jira] [Updated] (SPARK-19976) DirectStream API throws
OffsetOutOfRange Exception
[ https://issues.apache.org/jira/browse/SPARK-19976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Taukir updated SPARK-19976:
---------------------------
Affects Version/s: (was: 1.6.2)
2.0.0
> DirectStream API throws OffsetOutOfRange Exception
> --------------------------------------------------
>
> Key: SPARK-19976
> URL: https://issues.apache.org/jira/browse/SPARK-19976
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 2.0.0
> Reporter: Taukir
>
> I am using following code
> def functionToCreateContext(sc:SparkContext):StreamingContext = {
> val kafkaParams = new mutable.HashMap[String, Object]()
> kafkaParams.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBrokers)
> kafkaParams.put(ConsumerConfig.GROUP_ID_CONFIG, KafkaConsumerGrp)
> kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, classOf[StringDeserializer])
> kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, classOf[StringDeserializer])
> kafkaParams.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,"true")
> kafkaParams.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
> val consumerStrategy = ConsumerStrategies.Subscribe[String, String](threeparEventLogTopic.split(",").map(_.trim).filter(!_.isEmpty).toSet, kafkaParams)
> val kafkaStream = KafkaUtils.createDirectStream(ssc,LocationStrategies.PreferConsistent,consumerStrategy)
> }
> spark throws error and crash once OffsetOutOf RangeException
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org