[spark] branch branch-3.2 updated: [SPARK-40117][PYTHON][SQL] Convert condition to java in DataFrameWriterV2.overwrite

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
     new 6ec86e5d5d9 [SPARK-40117][PYTHON][SQL] Convert condition to java in DataFrameWriterV2.overwrite
6ec86e5d5d9 is described below

commit 6ec86e5d5d9df0b0d111decd0ea16e5a3e4cd3fe
Author: Wenli Looi <wl...@ucalgary.ca>
AuthorDate: Wed Aug 17 15:28:55 2022 +0900

    [SPARK-40117][PYTHON][SQL] Convert condition to java in DataFrameWriterV2.overwrite
    
    ### What changes were proposed in this pull request?
    
    Fix DataFrameWriterV2.overwrite() fails to convert the condition parameter to java. This prevents the function from being called.
    
    It is caused by the following commit that deleted the `_to_java_column` call instead of fixing it: https://github.com/apache/spark/commit/a1e459ed9f6777fb8d5a2d09fda666402f9230b9
    
    ### Why are the changes needed?
    
    DataFrameWriterV2.overwrite() cannot be called.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No
    
    ### How was this patch tested?
    
    Manually checked whether the arguments are sent to JVM or not.
    
    Closes #37547 from looi/fix-overwrite.
    
    Authored-by: Wenli Looi <wl...@ucalgary.ca>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
    (cherry picked from commit 46379863ab0dd2ee8fcf1e31e76476ff18397f60)
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 python/pyspark/sql/readwriter.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 65753543cb5..892b2500097 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -1163,6 +1163,7 @@ class DataFrameWriterV2(object):
         Overwrite rows matching the given filter condition with the contents of the data frame in
         the output table.
         """
+        condition = _to_java_column(condition)
         self._jwriter.overwrite(condition)
 
     @since(3.1)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org