You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2022/11/15 06:44:00 UTC
[jira] [Updated] (SPARK-41146) Assert the offset range for Kafka source in Trigger.AvailableNow
[ https://issues.apache.org/jira/browse/SPARK-41146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim updated SPARK-41146:
---------------------------------
Description:
We encountered the issue where the data source did not properly implement the offset with Trigger.AvailableNow, and the query ran with processing same data continuously without stopping.
We would like to proactively avoid such case for most used data sources.
was:
We encountered the issue where the data source did not properly implement the offset with Trigger.AvailableNow, and the query ran with processing same data continuously without stopping.
We would like to proactively avoid such case for most used data sources. I'll create a new JIRA ticket for Kafka data source as well.
> Assert the offset range for Kafka source in Trigger.AvailableNow
> ----------------------------------------------------------------
>
> Key: SPARK-41146
> URL: https://issues.apache.org/jira/browse/SPARK-41146
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> We encountered the issue where the data source did not properly implement the offset with Trigger.AvailableNow, and the query ran with processing same data continuously without stopping.
> We would like to proactively avoid such case for most used data sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org