You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/15 15:47:03 UTC

[GitHub] [spark] grundprinzip opened a new pull request, #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

grundprinzip opened a new pull request, #38270:
URL: https://github.com/apache/spark/pull/38270

   ### What changes were proposed in this pull request?
   This patch changes the way simple scalar built-in functions are resolved in the Python Spark Connect client. Previously, it was trying to manually load specific functions. With the changes in this patch, the trivial binary operators like `<`, `+`, ... are mapped to their name equivalents in Spark so that the dynamic function lookup works.
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   UT, E2E


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996448216


##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -19,9 +19,12 @@
 import unittest
 import tempfile
 
+import pandas

Review Comment:
   Hm .. we gonna fix this or do something. pandas isn't a required library for SQL package. Should probably skip this tests when pandas is not installed for now until we have a clear way to handle this. (see `pyspark.testing.sqlutils.have_pandas` and `pyspark.sql.tests.test_arrow_map`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996476173


##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -19,9 +19,12 @@
 import unittest
 import tempfile
 
+import pandas

Review Comment:
   Interestingly, nothing in Spark Connect will work atm without pandas because we always call `toPandas` in the collection of the result. Let me know what you want to do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996476719


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   this is just in the DSL package that is used for testing. I'll check what Catalyst is doing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] amaliujia commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996356050


##########
python/pyspark/sql/connect/column.py:
##########
@@ -161,7 +161,7 @@ def to_plan(self, session: Optional["RemoteSparkSession"]) -> proto.Expression:
         return self.ref.to_plan(session)
 
 
-class ScalarFunctionExpression(Expression):
+class ScalarFunctionExpression(ColumnRef):

Review Comment:
   Why a function expression inherit from ColumnRef now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996427097


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -193,20 +189,10 @@ class SparkConnectPlanner(plan: proto.Relation, session: SparkSession) {
    */
   private def transformScalarFunction(fun: proto.Expression.UnresolvedFunction): Expression = {
     val funName = fun.getPartsList.asScala.mkString(".")
-    funName match {
-      case "gt" =>
-        assert(fun.getArgumentsCount == 2, "`gt` function must have two arguments.")
-        expressions.GreaterThan(
-          transformExpression(fun.getArguments(0)),
-          transformExpression(fun.getArguments(1)))
-      case "eq" =>
-        assert(fun.getArgumentsCount == 2, "`eq` function must have two arguments.")
-        expressions.EqualTo(
-          transformExpression(fun.getArguments(0)),
-          transformExpression(fun.getArguments(1)))
-      case _ =>
-        lookupFunction(funName, fun.getArgumentsList.asScala.map(transformExpression).toSeq)
-    }
+    UnresolvedFunction(
+      Seq(funName),

Review Comment:
   There is another thing that we probably have to do similar to what you've done for the unresolved identifier, just passing the full string to the backend and then figure out there when do we split accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on PR #38270:
URL: https://github.com/apache/spark/pull/38270#issuecomment-1279952083

   @amaliujia it would be great if you could have another look. I added additional tests on the planner side to iron out the issue with the name parts. 
   
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996537113


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   For a bit of more context, one thing is that we use `snake_case` to match w/ SQL function names (see `Column` or `functions.scala`). This kind of naming rule is already mixed in our existing SQL DSL (see also `org.apache.spark.sql.catalyst.package`). Should probably pick one and stick to that.
   
   In the past, we followed `camelCase` in both DSL, `Column` and `functions.scala`. After that, we renamed them all to `snake_case` for SQL compatibility in `Column` and `functions.scala` (so the new DSL added follows  `snake_case` at `org.apache.spark.sql.catalyst.package`)
   
   Therefore, I tend to use `snake_case` in this DSL case too but I am fine if others (or you) feel strongly this is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r998350192


##########
connector/connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectProtoSuite.scala:
##########
@@ -50,6 +50,34 @@ class SparkConnectProtoSuite extends PlanTest with SparkConnectPlanTest {
     comparePlans(connectPlan.analyze, sparkPlan.analyze, false)
   }
 
+  test("UnresolvedFunction resolution.") {
+    {
+      import org.apache.spark.sql.connect.dsl.expressions._
+      import org.apache.spark.sql.connect.dsl.plans._
+      assertThrows[IllegalArgumentException] {
+        transform(connectTestRelation.select(callFunction("default.hex", Seq("id".protoAttr))))
+      }
+    }
+
+    val connectPlan = {
+      import org.apache.spark.sql.connect.dsl.expressions._
+      import org.apache.spark.sql.connect.dsl.plans._
+      transform(
+        connectTestRelation.select(callFunction(Seq("default", "hex"), Seq("id".protoAttr))))
+    }
+
+    assertThrows[UnsupportedOperationException] {
+      connectPlan.analyze
+    }
+
+    val validPlan = {
+      import org.apache.spark.sql.connect.dsl.expressions._
+      import org.apache.spark.sql.connect.dsl.plans._
+      transform(connectTestRelation.select(callFunction(Seq("hex"), Seq("id".protoAttr))))
+    }
+    assert(validPlan.analyze != null)

Review Comment:
   This is not to validate that the catalyst plan exists, but really just that existing functions are actually resolved. The `!=null` is mostly to have any assertion and not throw.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r998312826


##########
connector/connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectProtoSuite.scala:
##########
@@ -50,6 +50,34 @@ class SparkConnectProtoSuite extends PlanTest with SparkConnectPlanTest {
     comparePlans(connectPlan.analyze, sparkPlan.analyze, false)
   }
 
+  test("UnresolvedFunction resolution.") {
+    {
+      import org.apache.spark.sql.connect.dsl.expressions._
+      import org.apache.spark.sql.connect.dsl.plans._
+      assertThrows[IllegalArgumentException] {
+        transform(connectTestRelation.select(callFunction("default.hex", Seq("id".protoAttr))))
+      }
+    }
+
+    val connectPlan = {
+      import org.apache.spark.sql.connect.dsl.expressions._
+      import org.apache.spark.sql.connect.dsl.plans._
+      transform(
+        connectTestRelation.select(callFunction(Seq("default", "hex"), Seq("id".protoAttr))))
+    }
+
+    assertThrows[UnsupportedOperationException] {
+      connectPlan.analyze
+    }
+
+    val validPlan = {
+      import org.apache.spark.sql.connect.dsl.expressions._
+      import org.apache.spark.sql.connect.dsl.plans._
+      transform(connectTestRelation.select(callFunction(Seq("hex"), Seq("id".protoAttr))))
+    }
+    assert(validPlan.analyze != null)

Review Comment:
   it's better to compare it with the catalyst plan



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996448785


##########
python/pyspark/sql/connect/column.py:
##########
@@ -19,18 +19,59 @@
 
 
 import pyspark.sql.connect.proto as proto
+
 from pyspark.sql.connect._typing import PrimitiveType
 
 if TYPE_CHECKING:
     from pyspark.sql.connect.client import RemoteSparkSession
     import pyspark.sql.connect.proto as proto
 
 
+def _bin_op(
+    name: str, doc: str = "binary function", reverse: bool = False
+) -> Callable[["ColumnRef", Any], "Expression"]:
+    def _(self: "ColumnRef", other: Any) -> "Expression":
+        if isinstance(other, get_args(PrimitiveType)):
+            other = LiteralExpression(other)
+        if not reverse:
+            return ScalarFunctionExpression(name, self, other)
+        else:
+            return ScalarFunctionExpression(name, other, self)
+
+    return _
+
+
 class Expression(object):
     """
     Expression base class.
     """
 
+    __gt__ = _bin_op(">")

Review Comment:
   Why was this moved from Column to Expression?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996789322


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   maybe name the two methods `function`?
    https://github.com/apache/spark/blob/ff7ab34a6957965d74504ed1d36de21d1ad7319c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L368-L369
   
   don't feel strong about the naming



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r998894775


##########
python/pyspark/sql/connect/column.py:
##########
@@ -26,11 +26,51 @@
     import pyspark.sql.connect.proto as proto
 
 
+def _bin_op(
+    name: str, doc: str = "binary function", reverse: bool = False
+) -> Callable[["ColumnRef", Any], "Expression"]:
+    def _(self: "ColumnRef", other: Any) -> "Expression":
+        if isinstance(other, get_args(PrimitiveType)):
+            other = LiteralExpression(other)
+        if not reverse:
+            return ScalarFunctionExpression(name, self, other)
+        else:
+            return ScalarFunctionExpression(name, other, self)
+
+    return _
+
+
 class Expression(object):
     """
     Expression base class.
     """
 
+    __gt__ = _bin_op(">")
+    __lt__ = _bin_op(">")

Review Comment:
   https://github.com/apache/spark/pull/38303



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] amaliujia commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996574296


##########
python/pyspark/sql/connect/column.py:
##########
@@ -73,20 +114,6 @@ def __str__(self) -> str:
         return f"Literal({self._value})"
 
 
-def _bin_op(
-    name: str, doc: str = "binary function", reverse: bool = False
-) -> Callable[["ColumnRef", Any], Expression]:
-    def _(self: "ColumnRef", other: Any) -> Expression:
-        if isinstance(other, get_args(PrimitiveType)):
-            other = LiteralExpression(other)
-        if not reverse:
-            return ScalarFunctionExpression(name, self, other)
-        else:
-            return ScalarFunctionExpression(name, other, self)
-
-    return _
-
-
 class ColumnRef(Expression):

Review Comment:
   +1 I have been thinking this `ColumnRef` thing. Let's revisit it on the naming, etc. 



##########
python/pyspark/sql/connect/column.py:
##########
@@ -73,20 +114,6 @@ def __str__(self) -> str:
         return f"Literal({self._value})"
 
 
-def _bin_op(
-    name: str, doc: str = "binary function", reverse: bool = False
-) -> Callable[["ColumnRef", Any], Expression]:
-    def _(self: "ColumnRef", other: Any) -> Expression:
-        if isinstance(other, get_args(PrimitiveType)):
-            other = LiteralExpression(other)
-        if not reverse:
-            return ScalarFunctionExpression(name, self, other)
-        else:
-            return ScalarFunctionExpression(name, other, self)
-
-    return _
-
-
 class ColumnRef(Expression):

Review Comment:
   +1 I have been thinking this `ColumnRef` thing. Let's revisit it on the naming, etc. in the future. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] amaliujia commented on pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
amaliujia commented on PR #38270:
URL: https://github.com/apache/spark/pull/38270#issuecomment-1279835063

   Can you also improve the Scala side testing coverage as you have update the SparkConnectPlanner?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #38270:
URL: https://github.com/apache/spark/pull/38270#issuecomment-1282490390

   late LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996416503


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -193,20 +189,10 @@ class SparkConnectPlanner(plan: proto.Relation, session: SparkSession) {
    */
   private def transformScalarFunction(fun: proto.Expression.UnresolvedFunction): Expression = {
     val funName = fun.getPartsList.asScala.mkString(".")
-    funName match {
-      case "gt" =>
-        assert(fun.getArgumentsCount == 2, "`gt` function must have two arguments.")
-        expressions.GreaterThan(
-          transformExpression(fun.getArguments(0)),
-          transformExpression(fun.getArguments(1)))
-      case "eq" =>
-        assert(fun.getArgumentsCount == 2, "`eq` function must have two arguments.")
-        expressions.EqualTo(
-          transformExpression(fun.getArguments(0)),
-          transformExpression(fun.getArguments(1)))
-      case _ =>
-        lookupFunction(funName, fun.getArgumentsList.asScala.map(transformExpression).toSeq)
-    }
+    UnresolvedFunction(
+      Seq(funName),

Review Comment:
   Will do!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r997387895


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   Oh nice, I definitely missed renaming the other overall `callFunction` as well. TBH I'm not sure what the right approach is here because the catalyst DSL calls it `callFunction` instead of `call_function` 🤷 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] amaliujia commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996416342


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -193,20 +189,10 @@ class SparkConnectPlanner(plan: proto.Relation, session: SparkSession) {
    */
   private def transformScalarFunction(fun: proto.Expression.UnresolvedFunction): Expression = {
     val funName = fun.getPartsList.asScala.mkString(".")
-    funName match {
-      case "gt" =>
-        assert(fun.getArgumentsCount == 2, "`gt` function must have two arguments.")
-        expressions.GreaterThan(
-          transformExpression(fun.getArguments(0)),
-          transformExpression(fun.getArguments(1)))
-      case "eq" =>
-        assert(fun.getArgumentsCount == 2, "`eq` function must have two arguments.")
-        expressions.EqualTo(
-          transformExpression(fun.getArguments(0)),
-          transformExpression(fun.getArguments(1)))
-      case _ =>
-        lookupFunction(funName, fun.getArgumentsList.asScala.map(transformExpression).toSeq)
-    }
+    UnresolvedFunction(
+      Seq(funName),

Review Comment:
   Testing a negative case is great. 
   
   I was also thinking a test here:
   ```
   val funName = fun.getPartsList.asScala.mkString(".")
   
   UnresolvedFunction(
         Seq(funName),
         ...
   ```
   
   Basically it first concatenates the function name by `.` then passing that to a Seq which become Seq with one element, but why not just don't do the concatenation, or we have to do the  concatenation?
   
   Having a test case here will better demonstrate the intention of this implementation. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.
URL: https://github.com/apache/spark/pull/38270


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r998885276


##########
python/pyspark/sql/connect/column.py:
##########
@@ -26,11 +26,51 @@
     import pyspark.sql.connect.proto as proto
 
 
+def _bin_op(
+    name: str, doc: str = "binary function", reverse: bool = False
+) -> Callable[["ColumnRef", Any], "Expression"]:
+    def _(self: "ColumnRef", other: Any) -> "Expression":
+        if isinstance(other, get_args(PrimitiveType)):
+            other = LiteralExpression(other)
+        if not reverse:
+            return ScalarFunctionExpression(name, self, other)
+        else:
+            return ScalarFunctionExpression(name, other, self)
+
+    return _
+
+
 class Expression(object):
     """
     Expression base class.
     """
 
+    __gt__ = _bin_op(">")
+    __lt__ = _bin_op(">")

Review Comment:
   `_bin_op("<")`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r997430729


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L250-L256 Please let me know what you want to do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on PR #38270:
URL: https://github.com/apache/spark/pull/38270#issuecomment-1279771663

   R: @hvanhovell @cloud-fan @HyukjinKwon @amaliujia 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996425341


##########
python/pyspark/sql/connect/column.py:
##########
@@ -161,7 +161,7 @@ def to_plan(self, session: Optional["RemoteSparkSession"]) -> proto.Expression:
         return self.ref.to_plan(session)
 
 
-class ScalarFunctionExpression(Expression):
+class ScalarFunctionExpression(ColumnRef):

Review Comment:
   I'll refactor this to make the difference between Expression and ColumnRef clear. Essentially, I'll move all of the binary operators up to the Expression class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996476241


##########
python/pyspark/sql/connect/column.py:
##########
@@ -19,18 +19,59 @@
 
 
 import pyspark.sql.connect.proto as proto
+

Review Comment:
   yes, will clean it up



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996782339


##########
connector/connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectProtoSuite.scala:
##########
@@ -49,6 +49,33 @@ class SparkConnectProtoSuite extends PlanTest with SparkConnectPlanTest {
     comparePlans(connectPlan.analyze, sparkPlan.analyze, false)
   }
 
+  test("UnresolvedFunction resolution.") {
+    {
+      import org.apache.spark.sql.connect.dsl.expressions._
+      import org.apache.spark.sql.connect.dsl.plans._

Review Comment:
   nit: only import them once in `test("UnresolvedFunction resolution.")`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on PR #38270:
URL: https://github.com/apache/spark/pull/38270#issuecomment-1279912865

   > Can you also improve the Scala side testing coverage as you have updated the SparkConnectPlanner?
   
   I'll figure out to add more. Interestingly the work here is already covered by your previous patch on the where clause. Because we rely on the unresolved function call. But I'll add a negative test case. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996478189


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   the catalyst DSL has a similar `callFunction`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996448929


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   I would name it `func` as that's (much) more common in the codebase as far as I can tell.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996537113


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   For a bit of more context, one thing is that we use `snake_case` to match w/ SQL function names (see `Column` or `functions.scala`). This kind of naming rule is already mixed in our existing SQL DSL (see also `org.apache.spark.sql.catalyst.package`). Should probably pick one and stick to that.
   
   In the past, we followed `camelCase` in both DSL, `Column` and `functions.scala`. After that, we renamed them all to `snake_case` for SQL compatibility in `Column` and `functions.scala` (so the new DSL added follows  `snake_case` at `org.apache.spark.sql.catalyst.package`)
   
   Therefore, I tend to use `snake_case` in this DSL case too but I don't object if others (or you) feel this is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996448319


##########
python/pyspark/sql/connect/column.py:
##########
@@ -19,18 +19,59 @@
 
 
 import pyspark.sql.connect.proto as proto
+

Review Comment:
   Should probably remove this since `pyspark` should be within the same import block.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996476445


##########
python/pyspark/sql/connect/column.py:
##########
@@ -73,20 +114,6 @@ def __str__(self) -> str:
         return f"Literal({self._value})"
 
 
-def _bin_op(
-    name: str, doc: str = "binary function", reverse: bool = False
-) -> Callable[["ColumnRef", Any], Expression]:
-    def _(self: "ColumnRef", other: Any) -> Expression:
-        if isinstance(other, get_args(PrimitiveType)):
-            other = LiteralExpression(other)
-        if not reverse:
-            return ScalarFunctionExpression(name, self, other)
-        else:
-            return ScalarFunctionExpression(name, other, self)
-
-    return _
-
-
 class ColumnRef(Expression):

Review Comment:
   Let's have a discussion about this, but this is an unrelated change to this one. I think we should probably call `Expression` -> `Column` and `ColumnRef` -> `AttributeReference` but it will require some more digging what the right name should be. However, as said, that's independent of this change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r998886997


##########
python/pyspark/sql/connect/column.py:
##########
@@ -26,11 +26,51 @@
     import pyspark.sql.connect.proto as proto
 
 
+def _bin_op(
+    name: str, doc: str = "binary function", reverse: bool = False
+) -> Callable[["ColumnRef", Any], "Expression"]:
+    def _(self: "ColumnRef", other: Any) -> "Expression":
+        if isinstance(other, get_args(PrimitiveType)):
+            other = LiteralExpression(other)
+        if not reverse:
+            return ScalarFunctionExpression(name, self, other)
+        else:
+            return ScalarFunctionExpression(name, other, self)
+
+    return _
+
+
 class Expression(object):
     """
     Expression base class.
     """
 
+    __gt__ = _bin_op(">")
+    __lt__ = _bin_op(">")

Review Comment:
   Yeah, I think this was a mistake.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38270:
URL: https://github.com/apache/spark/pull/38270#issuecomment-1279898457

   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] grundprinzip commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
grundprinzip commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996449640


##########
python/pyspark/sql/connect/column.py:
##########
@@ -19,18 +19,59 @@
 
 
 import pyspark.sql.connect.proto as proto
+
 from pyspark.sql.connect._typing import PrimitiveType
 
 if TYPE_CHECKING:
     from pyspark.sql.connect.client import RemoteSparkSession
     import pyspark.sql.connect.proto as proto
 
 
+def _bin_op(
+    name: str, doc: str = "binary function", reverse: bool = False
+) -> Callable[["ColumnRef", Any], "Expression"]:
+    def _(self: "ColumnRef", other: Any) -> "Expression":
+        if isinstance(other, get_args(PrimitiveType)):
+            other = LiteralExpression(other)
+        if not reverse:
+            return ScalarFunctionExpression(name, self, other)
+        else:
+            return ScalarFunctionExpression(name, other, self)
+
+    return _
+
+
 class Expression(object):
     """
     Expression base class.
     """
 
+    __gt__ = _bin_op(">")

Review Comment:
   Without this change the different expressions are not combinable. Until now only column references were combinable but actually all column expressions should be.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996450969


##########
python/pyspark/sql/connect/column.py:
##########
@@ -73,20 +114,6 @@ def __str__(self) -> str:
         return f"Literal({self._value})"
 
 
-def _bin_op(
-    name: str, doc: str = "binary function", reverse: bool = False
-) -> Callable[["ColumnRef", Any], Expression]:
-    def _(self: "ColumnRef", other: Any) -> Expression:
-        if isinstance(other, get_args(PrimitiveType)):
-            other = LiteralExpression(other)
-        if not reverse:
-            return ScalarFunctionExpression(name, self, other)
-        else:
-            return ScalarFunctionExpression(name, other, self)
-
-    return _
-
-
 class ColumnRef(Expression):

Review Comment:
   I think we should rename this to `AttributeReference` to avoid confusion e.g., I think it's a bit mixed with `Column` interface that is the user-facing interface. @amaliujia and @cloud-fan 
   
   Should better to keep it matched with either Catalyst internal types or user-facing Spark SQL interface classes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996449102


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   or `call_function` (ref https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/_autosummary/snowflake.snowpark.functions.html#snowflake.snowpark.functions.call_function)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996449102


##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -64,6 +64,36 @@ package object dsl {
           .build()
     }
 
+    /**
+     * Create an unresolved function from name parts.
+     *
+     * @param nameParts
+     * @param args
+     * @return Expression wrapping the unresolved function.
+     */
+    def fun(nameParts: Seq[String], args: Seq[proto.Expression]): proto.Expression = {

Review Comment:
   or `function` (ref https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/_autosummary/snowflake.snowpark.functions.html#snowflake.snowpark.functions.function)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38270:
URL: https://github.com/apache/spark/pull/38270#discussion_r996448216


##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -19,9 +19,12 @@
 import unittest
 import tempfile
 
+import pandas

Review Comment:
   Hm .. we gotta fix this or do something. pandas isn't a required library for SQL package. Should probably skip this tests when pandas is not installed for now until we have a clear way to handle this. (see `pyspark.testing.sqlutils.have_pandas` and `pyspark.sql.tests.test_arrow_map`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #38270: [SPARK-40538] [CONNECT] Improve built-in function support for Python client.

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38270:
URL: https://github.com/apache/spark/pull/38270#issuecomment-1282213175

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org