[jira] [Updated] (SPARK-53687) Introduce WATERMARK clause in SQL statement

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-53687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-53687:
-----------------------------------
    Labels: pull-request-available  (was: )

> Introduce WATERMARK clause in SQL statement
> -------------------------------------------
>
>                 Key: SPARK-53687
>                 URL: https://issues.apache.org/jira/browse/SPARK-53687
>             Project: Spark
>          Issue Type: Task
>          Components: Declarative Pipelines, SQL, Structured Streaming
>    Affects Versions: 4.1.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> With Spark Declarative Pipeline, Apache Spark supports to define the streaming query with SQL statement (Flow or Streaming Table).
> The SQL statement is expected to be composed with streaming semantic (STREAM relation), and this converts the query part to be streaming. With this, the query can be stateful, e.g. aggregation will be a streaming aggregation, deduplication and stream-stream join, etc. And in majority of cases, we require watermark to be defined per source for the stateful query to work properly.
> Databricks introduced WATERMARK clause to cover this functionality in Lakeflow Declarative Pipeline, but when open sourcing this, we missed to introduce this clause. This ticket is to open source WATERMARK clause.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org