You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2023/07/10 05:29:03 UTC

[spark] branch master updated: [SPARK-43710][PS][FOLLOWUP] Fix `date_part` invocations

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 81ef113d068 [SPARK-43710][PS][FOLLOWUP] Fix `date_part` invocations
81ef113d068 is described below

commit 81ef113d068b325664630c1669b817800972829b
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Jul 10 13:28:45 2023 +0800

    [SPARK-43710][PS][FOLLOWUP] Fix `date_part` invocations
    
    ### What changes were proposed in this pull request?
    fix https://github.com/apache/spark/commit/bc20e85b0e1da510cc091dbd03f210ef9fc56b25:
    1, should use `F.lit("HOUR")` as the parameter;
    2, should also change `def microseconds` since the old function was removed;
    
    ### Why are the changes needed?
    to fix wrong usage
    
    ### Does this PR introduce _any_ user-facing change?
    no
    
    ### How was this patch tested?
    existing UT
    
    Closes #41911 from zhengruifeng/spark_43710_followup.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
 python/pyspark/pandas/indexes/timedelta.py | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/python/pyspark/pandas/indexes/timedelta.py b/python/pyspark/pandas/indexes/timedelta.py
index e46d602e985..36cbb7cf4e6 100644
--- a/python/pyspark/pandas/indexes/timedelta.py
+++ b/python/pyspark/pandas/indexes/timedelta.py
@@ -25,7 +25,6 @@ from pyspark._globals import _NoValue
 from pyspark.pandas.indexes.base import Index
 from pyspark.pandas.missing.indexes import MissingPandasLikeTimedeltaIndex
 from pyspark.pandas.series import Series
-from pyspark.pandas.spark import functions as SF
 from pyspark.sql import functions as F
 
 
@@ -150,9 +149,9 @@ class TimedeltaIndex(Index):
 
         @no_type_check
         def get_seconds(scol):
-            hour_scol = F.date_part("HOUR", scol)
-            minute_scol = F.date_part("MINUTE", scol)
-            second_scol = F.date_part("SECOND", scol)
+            hour_scol = F.date_part(F.lit("HOUR"), scol)
+            minute_scol = F.date_part(F.lit("MINUTE"), scol)
+            second_scol = F.date_part(F.lit("SECOND"), scol)
             return (
                 F.when(
                     hour_scol < 0,
@@ -178,7 +177,7 @@ class TimedeltaIndex(Index):
 
         @no_type_check
         def get_microseconds(scol):
-            second_scol = SF.date_part("SECOND", scol)
+            second_scol = F.date_part(F.lit("SECOND"), scol)
             return (
                 (
                     F.when(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org