You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gabor Somogyi (Jira)" <ji...@apache.org> on 2019/11/04 13:29:00 UTC
[jira] [Comment Edited] (SPARK-29710) Seeing offsets not resetting
even when reset policy is configured explicitly
[ https://issues.apache.org/jira/browse/SPARK-29710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966655#comment-16966655 ]
Gabor Somogyi edited comment on SPARK-29710 at 11/4/19 1:28 PM:
----------------------------------------------------------------
Just seen this issue by luck because the personal tag part is not set properly.
Well, what I don't fully understand how did you set "auto.offset.reset" on a Structured Streaming query? It's just throws exception and "startingOffsets" can be used.
* Please share the application which doesn't throw exception.
* It would be good to attach driver + executor logs.
BTW, are you using compacted topic?
was (Author: gsomogyi):
Just seen this issue by luck because the personal tag part is not set properly.
Well, what I don't fully understand how did you set "auto.offset.reset" on a Structured Streaming query? It's just throws exception and "startingOffsets" can be used.
* Please share the application which doesn't throw exception.
* It would be good to attach driver + executor logs.
BTW, are you using compacted topic?
> Seeing offsets not resetting even when reset policy is configured explicitly
> ----------------------------------------------------------------------------
>
> Key: SPARK-29710
> URL: https://issues.apache.org/jira/browse/SPARK-29710
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.1
> Environment: Window10 , eclipse neos
> Reporter: Shyam
> Priority: Major
>
>
> even after setting *"auto.offset.reset" to "latest"* I am getting below error
>
> org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: \{COMPANY_TRANSACTIONS_INBOUND-16=168}org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: \{COMPANY_TRANSACTIONS_INBOUND-16=168} at org.apache.kafka.clients.consumer.internals.Fetcher.throwIfOffsetOutOfRange(Fetcher.java:348) at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:396) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:999) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937) at org.apache.spark.sql.kafka010.InternalKafkaConsumer.fetchData(KafkaDataConsumer.scala:470) at org.apache.spark.sql.kafka010.InternalKafkaConsumer.org$apache$spark$sql$kafka010$InternalKafkaConsumer$$fetchRecord(KafkaDataConsumer.scala:361) at org.apache.spark.sql.kafka010.InternalKafkaConsumer$$anonfun$get$1.apply(KafkaDataConsumer.scala:251) at org.apache.spark.sql.kafka010.InternalKafkaConsumer$$anonfun$get$1.apply(KafkaDataConsumer.scala:234) at org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77) at org.apache.spark.sql.kafka010.InternalKafkaConsumer.runUninterruptiblyIfPossible(KafkaDataConsumer.scala:209) at org.apache.spark.sql.kafka010.InternalKafkaConsumer.get(KafkaDataConsumer.scala:234)
>
> [https://stackoverflow.com/questions/58653885/even-after-setting-auto-offset-reset-to-latest-getting-error-offsetoutofrang]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org