(spark) branch master updated: [SPARK-52852][SDP] Remove unused spark_conf in create_streaming_table

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new a2a9d4092e00 [SPARK-52852][SDP] Remove unused spark_conf in create_streaming_table
a2a9d4092e00 is described below

commit a2a9d4092e00b6733da1bda2b90822ba3a7771c6
Author: Jacky Wang <ja...@databricks.com>
AuthorDate: Tue Jul 22 15:50:16 2025 +0900

    [SPARK-52852][SDP] Remove unused spark_conf in create_streaming_table
    
    ### What changes were proposed in this pull request?
    
    Remove the `spark_conf` from `create_streaming_table`
    
    ### Why are the changes needed?
    
    The `spark_conf` argument on `create_streaming_table` is unused. Configuration can be provided on the individual flows that target a streaming table created this way.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, an API arg is removed but on an unreleased API.
    
    ### How was this patch tested?
    
    Existing tests
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No
    
    Closes #51607 from JiaqiWang18/SPARK-52852-remove-spark_conf.
    
    Authored-by: Jacky Wang <ja...@databricks.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 python/pyspark/pipelines/api.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/python/pyspark/pipelines/api.py b/python/pyspark/pipelines/api.py
index 64fb279b8d6c..35beff19c981 100644
--- a/python/pyspark/pipelines/api.py
+++ b/python/pyspark/pipelines/api.py
@@ -400,7 +400,6 @@ def create_streaming_table(
     name: str,
     *,
     comment: Optional[str] = None,
-    spark_conf: Optional[Dict[str, str]] = None,
     table_properties: Optional[Dict[str, str]] = None,
     partition_cols: Optional[List[str]] = None,
     schema: Optional[Union[StructType, str]] = None,
@@ -414,9 +413,6 @@ def create_streaming_table(
 
     :param name: The name of the table.
     :param comment: Description of the table.
-    :param spark_conf: A dict whose keys are the conf names and values are the conf values. \
-        These confs will be set when the query for the dataset is executed and they can override \
-        confs set for the pipeline or on the cluster.
     :param table_properties: A dict where the keys are the property names and the values are the \
         property values. These properties will be set on the table.
     :param partition_cols: A list containing the column names of the partition columns.


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org