You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2022/11/15 06:43:00 UTC

[jira] [Created] (SPARK-41145) Assert the offset range for file stream source in Trigger.AvailableNow

Jungtaek Lim created SPARK-41145:
------------------------------------

             Summary: Assert the offset range for file stream source in Trigger.AvailableNow
                 Key: SPARK-41145
                 URL: https://issues.apache.org/jira/browse/SPARK-41145
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.4.0
            Reporter: Jungtaek Lim


We encountered the issue where the data source did not properly implement the offset with Trigger.AvailableNow, and the query ran with processing same data continuously without stopping.

We would like to proactively avoid such case for most used data sources. I'll create a new JIRA ticket for Kafka data source as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org