[jira] [Assigned] (SPARK-39564) Expose the information of catalog table to the logical plan in streaming query

["Apache Spark (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-39564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-39564:
------------------------------------

    Assignee: Apache Spark

> Expose the information of catalog table to the logical plan in streaming query
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-39564
>                 URL: https://issues.apache.org/jira/browse/SPARK-39564
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.4.0
>            Reporter: Jungtaek Lim
>            Assignee: Apache Spark
>            Priority: Major
>
> In Spark 3.1.0, we added ability to read and write from/to catalog table for streaming query. (Read: [SPARK-32885|https://issues.apache.org/jira/browse/SPARK-32885] / Write: [SPARK-32896|https://issues.apache.org/jira/browse/SPARK-32896])
> When the streaming query is being executed with such API, the table identifier for source and sink is "conditionally" exposed to the logical plan, depending on the DSv1 vs DSv2 and source vs sink.
> Since this is the only way to get the information of table, we would like to make some changes on relevant logical nodes to carry out and expose the information of table, specifically, table identifier.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org