[GitHub] [spark] cloud-fan commented on a diff in pull request #37823: [SPARK-40380][SQL] Fix constant-folding of InvokeLike to avoid non-serializable literal embedded in the plan

[GitBox <gi...@apache.org>]

cloud-fan commented on code in PR #37823:
URL: https://github.com/apache/spark/pull/37823#discussion_r965390658


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:
##########
@@ -62,6 +63,14 @@ trait InvokeLike extends Expression with NonSQLExpression with ImplicitCastInput
       .map(cls => v => cls.cast(v))
       .getOrElse(identity)
 
+  // Returns true if we can trust all values of the given DataType can be serialized.
+  private def trustedSerializable(dt: DataType): Boolean = {
+    // Right now we conservatively block all ObjectType (Java objects) regardless of
+    // serializability, because the type-level info with java.io.Serializable and
+    // java.io.Externalizable marker interfaces are not strong guarantees.
+    // This restriction can be relaxed in the future to expose more optimizations.
+    !dt.isInstanceOf[ObjectType]

Review Comment:
   to be safe, how about `!dt.existsRecursively(_.isInstanceOf[ObjectType])`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org