You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "nija-at (via GitHub)" <gi...@apache.org> on 2023/08/15 13:54:25 UTC

[GitHub] [spark] nija-at opened a new pull request, #42500: [SPARK-44816][CONNECT] Improve error message when UDF class is not found

nija-at opened a new pull request, #42500:
URL: https://github.com/apache/spark/pull/42500

   ### What changes were proposed in this pull request?
   
   Improve the error messaging on the connect client when using
   a UDF whose corresponding class has not been sync'ed with the
   spark connect service.
   
   Prior to this change, the client receives a cryptic error:
   
   ```
   Exception in thread "main" org.apache.spark.SparkException: Main$
   ```
   
   With this change, the message is improved to be:
   
   ```
   Exception in thread "main" org.apache.spark.SparkException: Failed to load class: Main$. Make sure the artifact where the class is defined is installed by calling session.addArtifact.
   ```
   
   ### Why are the changes needed?
   
   This change makes it clear to the user on what the error is.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. The error message is improved. See details above.
   
   ### How was this patch tested?
   
   Manually by running a connect server and client.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42500: [SPARK-44816][CONNECT] Improve error message when UDF class is not found

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42500:
URL: https://github.com/apache/spark/pull/42500#discussion_r1295240142


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1532,6 +1532,11 @@ class SparkConnectPlanner(val sessionHolder: SessionHolder) extends Logging {
               s"Failed to load class correctly due to $nsm. " +
                 "Make sure the artifact where the class is defined is installed by calling" +
                 " session.addArtifact.")
+          case cnf: ClassNotFoundException =>

Review Comment:
   This is great!
   
   One thing to think about: If there is no serialization proxy involved it should be relatively easy to figure out which class to add. However if a proxy involved you'll get a ClassCastException that is very cryptic. See 
    https://issues.apache.org/jira/browse/SPARK-36917. We could also try to make that a bit more actionable.
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] nija-at commented on a diff in pull request #42500: [SPARK-44816][CONNECT] Improve error message when UDF class is not found

Posted by "nija-at (via GitHub)" <gi...@apache.org>.
nija-at commented on code in PR #42500:
URL: https://github.com/apache/spark/pull/42500#discussion_r1295820716


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -1532,6 +1532,11 @@ class SparkConnectPlanner(val sessionHolder: SessionHolder) extends Logging {
               s"Failed to load class correctly due to $nsm. " +
                 "Make sure the artifact where the class is defined is installed by calling" +
                 " session.addArtifact.")
+          case cnf: ClassNotFoundException =>

Review Comment:
   I couldn't reproduce this in the context of connect. Can you help with getting a repro of this issue in connect, and I'll be happy to look into this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on pull request #42500: [SPARK-44816][CONNECT] Improve error message when UDF class is not found

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #42500:
URL: https://github.com/apache/spark/pull/42500#issuecomment-1690291499

   Merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell closed pull request #42500: [SPARK-44816][CONNECT] Improve error message when UDF class is not found

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell closed pull request #42500: [SPARK-44816][CONNECT] Improve error message when UDF class is not found
URL: https://github.com/apache/spark/pull/42500


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org