[GitHub] [spark] zhengruifeng commented on a diff in pull request #39041: [SPARK-41528][CONNECT] Merge namespace of Spark Connect and PySpark API

[GitBox <gi...@apache.org>]

zhengruifeng commented on code in PR #39041:
URL: https://github.com/apache/spark/pull/39041#discussion_r1051742291


##########
python/pyspark/sql/functions.py:
##########
@@ -2341,6 +2415,7 @@ def approx_count_distinct(col: "ColumnOrName", rsd: Optional[float] = None) -> C
         return _invoke_function("approx_count_distinct", _to_java_column(col), rsd)
 
 
+@try_remote_functions
 def broadcast(df: DataFrame) -> DataFrame:

Review Comment:
   ditto: PySpark DF -> Connect DF conversion



##########
python/pyspark/sql/functions.py:
##########
@@ -208,6 +209,7 @@ def col(col: str) -> Column:
 column = col
 
 
+@try_remote_functions
 def asc(col: "ColumnOrName") -> Column:

Review Comment:
   It seems that `try_remote_functions` needs to support this conversion `pyspark.sql.column.Column` -> `pyspark.sql.connect.column.Column`



##########
python/pyspark/sql/utils.py:
##########
@@ -307,3 +311,52 @@ def is_timestamp_ntz_preferred() -> bool:
     """
     jvm = SparkContext._jvm
     return jvm is not None and jvm.PythonSQLUtils.isTimestampNTZPreferred()
+
+
+def is_remote() -> bool:
+    """
+    Returns if the current running environment is for Spark Connect.
+    """
+    return "SPARK_REMOTE" in os.environ
+
+
+def try_remote_functions(f: FuncT) -> FuncT:

Review Comment:
   I don't know it well, is it possible to do monkey patching instead of add this annotation to each function?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org