You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/07/12 05:20:00 UTC
[jira] [Updated] (SPARK-36084) spark kafka offset missed some
partition offset describle
[ https://issues.apache.org/jira/browse/SPARK-36084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-36084:
---------------------------------
Priority: Major (was: Critical)
> spark kafka offset missed some partition offset describle
> ---------------------------------------------------------
>
> Key: SPARK-36084
> URL: https://issues.apache.org/jira/browse/SPARK-36084
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: geekyouth
> Priority: Major
>
> Caused by: org.apache.spark.sql.streaming.StreamingQueryException: Set(rcf_adidas_wow_room_event-9, rcf_adidas_wow_room_event-8, rcf_adidas_wow_room_event-7, rcf_adidas_wow_room_event-6, rcf_adidas_wow_room_event-5) are gone. Some data may have been missed.
> Some data may have been lost because they are not available in Kafka any more; either the
> data was aged out by Kafka or the topic may have been deleted before all the data in the
> topic was processed. If you don't want your streaming query to fail on such cases, set the
> source option "failOnDataLoss" to "false".
> === Streaming Query ===
> Identifier: [id = b0e0f54e-e638-430e-9912-751234a6ce13, runId = 570e950b-edcb-40c4-99ea-fd4db958b225]
> Current Committed Offsets: {KafkaV2[Subscribe[rcf_adidas_wow_room_event]]: {"rcf_adidas_wow_room_event":
> {"8":3572,"2":3571,"5":3571,"4":3571,"7":3571,"1":3571,"9":3571,"3":3571,"6":3571,"0":3571}
> }}
> Current Available Offsets: {KafkaV2[Subscribe[rcf_adidas_wow_room_event]]: {"rcf_adidas_wow_room_event":
> {"2":3571,"4":3571,"1":3571,"3":3571,"0":3571}
> }}
>
> —
> My topic has 10 partitions , but the last offset dir records only 5 partitions.
> After i delete the last error offset dir , the query runs success.
>
> But i do not understand what cause this error.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org